JP7181000B2 - BIOLOGICAL TISSUE IMAGE PROCESSING APPARATUS AND METHOD - Google Patents

Description

The present invention relates to a biological tissue image processing apparatus and method.

Three-dimensional microscopy is known as a technique for analyzing or imaging the three-dimensional structure of biological tissue. Electron or optical microscopes are generally used in three-dimensional microscopy. For example, as three-dimensional microscopy methods using a scanning electron microscope (Scanning Electron Microscope), Focused Ion Beam SEM (FIB-SEM) method, Serial Block-Face SEM (SBF-SEM) method, and serial section SEM (Serial Section SEM) method (see Non-Patent Document 1) has been proposed. The serial section SEM method is also called array-tomography method.

In the continuous section SEM method, a plurality of sample sections (ultrathin sections) are cut out from a biological tissue sample and arranged on a substrate. Each sample section on the substrate is sequentially observed by a scanning electron microscope, thereby acquiring a plurality of images. Based on a plurality of acquired images, the three-dimensional structure of specific organs (cells, mitochondria, nuclei, etc.) contained in biological tissue is analyzed or imaged. According to the serial section SEM method, it is possible to re-observe a sample section that has already been observed.

Koga et al., "Serial section SEM method and its application to 3D structure analysis of Golgi apparatus," Microscopy, Vol.49, No.3, 2014.

As described above, when analyzing or imaging the three-dimensional structure of a biological tissue, planes or layers located at multiple depths in the biological tissue are observed, thereby generating multiple images corresponding to the multiple depths. is obtained. Then, a process of detecting or identifying a target component is applied to each image. Here, the target element is a specific tissue component (or cell component) to be detected or identified during image processing. The final target of analysis or imaging may be the target element, or a tissue component different from the final target of analysis or imaging may be the target element. To give an example of the latter, when imaging a cell lumen (cytoplasm) surrounded by a cell membrane, the cell membrane is first detected or identified as an element of interest.

Manually performing the entire process of detecting or identifying elements of interest in a large number of images is very tedious. It is conceivable to have a machine learning type estimator perform the processing. However, biological tissue has a very complicated three-dimensional structure, and it also has individual differences. It is necessary to perform manual correction as necessary while visually observing a plurality of target element images as estimation results. Implementation of means or tools to support the work is desired. It is desired to implement a means or tool for assisting manual correction of a plurality of target element images arranged on the time axis.

SUMMARY OF THE INVENTION An object of the present invention is to support manual correction of a plurality of target element images that are continuous on the spatial axis or the time axis in biological tissue image processing. Another object of the present invention is to enable efficient or accurate manual correction of a plurality of target element images in biological tissue image processing.

A biological tissue image processing apparatus according to the present invention applies a process of estimating a target element contained in the biological tissue to a plurality of original images that are consecutively obtained on the spatial axis or the time axis obtained by observing the biological tissue. An image generator that includes a machine learning type estimator and generates a plurality of target element images from the plurality of original images, and a user-selected position on the spatial axis or the time axis from the plurality of target element images and a work support unit for retrieving a specific attention element image corresponding to and displaying it as a work target image, and for reflecting a user's modification of the work target image to the specific attention element image.

In the biological tissue image processing method according to the present invention, a plurality of original images corresponding to a plurality of depths obtained by microscopic observation of a biological tissue are input to a machine-learning-type membrane estimator. obtaining a plurality of membrane likelihood maps corresponding to the depths of the membrane, obtaining a plurality of membrane images corresponding to the plurality of depths based on the plurality of membrane likelihood maps, and the plurality of membrane images extracting a specific membrane image corresponding to the depth selected by the user from among the images, displaying it as an image to be worked, and reflecting the user's modification of the image to be worked on to the specific membrane image; includes.

The above method is implemented as a hardware function or a software function, and in the latter case, a program for executing the function is installed in the information processing apparatus via a network or a portable storage medium. The concept of information processing apparatus includes a personal computer, an electron microscope system, an optical microscope system, and the like.

Advantageous Effects of Invention According to the present invention, manual correction of a plurality of target element images can be supported in biological tissue image processing. Alternatively, according to the present invention, in biological tissue image processing, manual correction can be efficiently or accurately performed on a plurality of target element images.

1 is a conceptual diagram showing a biological tissue image processing system according to an embodiment; FIG. 1 is a block diagram showing a first configuration example of a main body of a biological tissue image processing apparatus; FIG. FIG. 4 is a block diagram showing a configuration example of a machine-learning film estimator; FIG. 11 illustrates generation of a temporary membrane image based on a membrane likelihood map; FIG. 4 is a diagram showing a plurality of functions possessed by the processing tool unit; FIG. 4 is a diagram showing a first example of a work window; It is a figure for demonstrating a display process. It is a figure for demonstrating another display process. It is a figure for demonstrating the 1st example of an interpolation process. It is a figure for demonstrating the 2nd example of an interpolation process. FIG. 10 is a diagram showing a film image before correction and a film image after correction; It is a figure which shows an example of labeling processing. FIG. 11 illustrates presentation of multiple labeling candidates; It is a figure which shows an example of a management table. FIG. 10 is a diagram showing a second example of a work window; FIG. 3 is a diagram showing a three-dimensional image; FIG. 10 is a block diagram showing a second configuration example of the main body of the biological tissue image processing apparatus; FIG. 11 is a block diagram showing a third configuration example of the main body of the biological tissue image processing apparatus; 13 is a flow chart showing processing contents in the third configuration example. It is a figure which shows the process result in a 3rd structural example.

(A) Outline of Embodiment A biological tissue image processing apparatus according to an embodiment includes an image generation unit including a machine learning estimator, and a work support unit. A machine-learning estimator extracts an element of interest (more precisely, an image of an element of interest) included in the biological tissue for a plurality of original images that are consecutive on the spatial axis or the time axis obtained by observing the biological tissue. is applied. Through this processing, a plurality of target element images are generated from a plurality of original images. The work support unit extracts a specific attention element image corresponding to a position selected by the user from among the plurality of attention element images, displays it as an image to be worked, and performs correction of the image to be worked by the user. This is reflected in the element image.

According to the above configuration, when a position on the spatial axis or the time axis is selected, a specific target element image corresponding to the selected position is displayed as the work target image. If the displayed image for work is modified, the modification is reflected in the image for work, that is, the specific focused element image. In the correction work process, for example, when it is desired to compare two target element images adjacent on the spatial axis or temporal axis, two adjacent positions on the spatial axis or temporal axis may be alternately selected. Specifically, for example, the depth may be selected sequentially from a certain depth on the spatial axis to the shallow side or the deep side, and necessary corrections may be made to the displayed individual work target images. good. Thus, according to the above configuration, when correcting a plurality of target element images corresponding to a plurality of positions, it is possible to improve work efficiency or improve work accuracy. In the embodiment, a plurality of original images aligned on the depth axis as the spatial axis are input to the estimator, but a plurality of images aligned on the time axis may be input to the estimator.

The element of interest is a particular tissue component or a particular cellular component, and in embodiments the element of interest is a cell membrane. A cell lumen (cytoplasm) surrounded by a cell membrane, an intracellular organelle (organelle), and the like may be the elements of interest. The machine-learning estimator is for example a CNN, but other types of machine-learning estimators may be used. In an embodiment, the work support unit can also be used when generating a correct image to be given to the estimator during the learning process of the estimator. In the embodiment, a plurality of original images are acquired by serial section SEM method (array tomography method), but even if a plurality of original images are acquired by FIB-SEM method, SBF-SEM method, or other methods, good. A user who modifies each element-of-interest image is usually an expert with some knowledge of biological tissue. Each attention element image may be modified by others.

In an embodiment, the estimator generates a plurality of element-of-interest likelihood maps based on a plurality of original images, and the image generator generates a plurality of element-of-interest images based on the plurality of element-of-interest likelihood maps. Contains a generator. That is, in the embodiment, the image generator is composed of an estimator and an image generator. All of them may be configured by a machine learning type estimator. The image generator is composed of, for example, a binarizer.

In the embodiment, the work support unit displays a work window including a work target image, and the work window includes position selection display elements that are operated when selecting a position. According to this configuration, the position can be easily selected by operating the display element for position selection. As the display element for position selection, it is desirable to employ a slider as a display element that slides along a predetermined axis. With such a configuration, it becomes easier to intuitively recognize the position to be selected, and operability can be improved. In embodiments, the position selection display element is a depth selection display element.

In the embodiment, the work window further includes an addition display element that is operated when adding the content of the work target image and a deletion display element that is operated when deleting the content of the work target image. included. For example, in a state in which a membrane image showing cell membranes is displayed as an image to be worked on, a plurality of membrane pixels are interpolated in membrane discontinuities in the membrane image by operating the display element for addition, while the display element for deletion is operated. removes erroneously estimated portions (for example, a plurality of membrane pixels corresponding to mitochondrial membranes) in the membrane image.

In the embodiment, the work window further includes labeling display elements that are operated when labeling the work target image in predetermined units. Labeling is attribution, that is, labeling, for managing three-dimensional connections. For example, if an intracellular region at one depth and an intracellular region at another depth are determined to belong to the same cell, the same label is given to those regions. The work is equivalent to annotation. In the embodiment, since the labeling can be performed while correcting the image to be worked on, work efficiency can be improved. Labeling may be done automatically. Alternatively, one or more candidates may be automatically selected and presented for labeling.

In the embodiment, the work support unit has a function of simultaneously displaying a specific original image corresponding to the selected position as a background image when displaying the work target image. According to this configuration, it is possible to correct or label the image to be worked while referring to the original image, so that workability or accuracy can be improved. Preferably, one of the images is displayed in black and white and the other is displayed in color so that the original image and the work support image to be synthesized and displayed can be distinguished from each other.

In the embodiment, in addition to the function of displaying the image to be worked on, the work support unit has a function of displaying a specific original image corresponding to the selected position, and a function of displaying a specific target element likelihood corresponding to the selected position. It has a function to display the degree map. When the original image is displayed, it is possible to correct the image to be worked on while referring to the original image. Also, if the target element likelihood map is displayed, it becomes possible to evaluate the validity of the estimation result of the estimator, the validity of the result of the binarization processing, and the like. In the embodiment, the work support unit generates a first image stack consisting of a plurality of original images, a second image stack consisting of a plurality of element-of-interest likelihood maps, and a third image consisting of a plurality of element-of-interest likelihood maps. Stacks are managed while being associated with each other.

In the embodiment, when displaying any one of the specific original image, the specific target element likelihood map, and the specific target element image corresponding to the selected position, the work support unit displays the image , another image that is of the same type as the image and that corresponds to a position different from the selected position is superimposed and displayed. From the continuity of tissue on the spatial axis or the time axis, if a plurality of images aligned on the axis are displayed in a superimposed manner, it is possible to accurately judge the necessity of correction from the mutual comparison. In addition, such a superimposed display enables accurate labeling. In overlapping display, it is desirable to configure so that the transparency of at least the upper image can be varied. Three or more consecutive images may be superimposed on the spatial axis or the temporal axis.

In the embodiment, the work support unit has a 3D image processing unit that creates a 3D image representing the labeling results based on the labeling results for a plurality of target element images and displays the 3D image. According to this configuration, it is possible to correct the image to be worked on and the result of labeling while observing the three-dimensional image. The three-dimensional image makes it possible to easily judge the adequacy of the corrections and labeling done at each depth. For example, incorrect labeling can result in unnatural distortion of the three-dimensional image, or unnatural defects or protrusions in the three-dimensional image. Labeling results and the like are modified based on such information.

In the embodiment, the work support unit selects one or It has a candidate selection unit that selects a plurality of labeling candidates. This configuration assumes tissue continuity on the spatial or temporal axis, and the labeling result at one position is useful for labeling at another position. This configuration reduces the user's burden in labeling.

In the embodiment, the candidate selection unit performs each labeling based on the degree of similarity between the labeled portion in the first target element image and each labeling candidate included in the second target element image. Change the display mode of candidates. According to this configuration, the degree of connection possibility can be intuitively recognized from the display mode of each labeling candidate, so it is possible to assist the user in labeling.

A biological tissue image processing apparatus according to an embodiment applies a process of estimating a specific film included in a biological tissue to a plurality of original images obtained by observing the biological tissue and continuing on the spatial axis or the time axis. , thereby generating a plurality of first membrane likelihood maps corresponding to a plurality of positions, and a machine learning membrane estimator of a different type from the first membrane estimator. a machine-learning second membrane estimator that applies a process of estimating a specific membrane to a plurality of original images and thereby generates a plurality of second membrane likelihood maps; a plurality of first membrane likelihood maps; An image generator for generating a plurality of film images corresponding to a plurality of positions based on a degree map; A work support unit for displaying as a work target image, the work support unit identifying and displaying correction candidates included in the work target image based on differences between the plurality of first membrane likelihood maps and the plurality of second membrane likelihood maps. and This configuration uses two types of machine-learning film estimators together. The user can be urged to correct the part where the estimation results of the two membrane estimators are dissociated. Note that an element of interest other than the film may be an estimation target. More than two types of film estimators may be used.

N target element estimators including the above estimators (where N is an integer of 3 or more) are provided, and based on the N target element likelihood maps output from the N target element estimators, a work target image may be identified and identified. In this case, based on the N target element likelihood maps, the target element likelihood variation information may be calculated for each pixel, and correction candidate pixels may be specified based on the variation information. Variation information is, for example, standard deviation.

A biological tissue image processing apparatus according to an embodiment applies a process of estimating a cell membrane contained in a biological tissue to a plurality of original images that are consecutive on the spatial axis or the time axis obtained by observing the biological tissue, A machine learning type first estimator that generates a plurality of membrane likelihood maps corresponding to a plurality of positions by and an estimator integrated or separate from the first estimator, wherein A machine learning type second estimator that generates a plurality of organelle likelihood maps by applying a process of estimating organelles contained in a biological tissue, and a plurality of membranes based on the plurality of organelle likelihood maps. A correction unit that corrects the likelihood map, an image generator that generates a plurality of membrane images corresponding to a plurality of positions based on the plurality of corrected membrane likelihood maps, and a user selects from among the plurality of membrane images. a work support unit for extracting a specific film image corresponding to the determined position and displaying it as a work target image. In this configuration, organelles (organelles in cells) as elements of interest other than cell membranes are separately estimated, and the estimation results are used in correcting the membrane likelihood map. When estimating cell membranes, organelles or their membranes are likely to be misidentified. The above configuration separately estimates the organelle or its membrane, and improves the membrane likelihood map based on the estimation result. A single estimator may function as the first estimator and the second estimator. The first estimator and the second estimator may be composed of separate estimators. It is also conceivable to apply the same configuration as described above to combinations other than the combination of cell membranes and organelles. Three or more estimation targets may be estimated in parallel.

In the embodiment, the estimator's learning process includes a primary learning process and a secondary learning process, and in the secondary learning process, the estimator is provided with a plurality of target element images that have been modified by the work support unit. That is, the work support unit functions both in the learning process of the estimator and in the subsequent image processing process.

(B) Details of Embodiment FIG. 1 shows a biological tissue image processing system according to an embodiment. The illustrated biological tissue image processing system 10 is a system for analyzing and imaging the three-dimensional structure of biological tissue. Using this biological tissue image processing system, for example, an image that expresses nerve cells in the brain of a human body or an animal three-dimensionally is generated. Any tissue, organ, etc. in an organism can be an analysis target or an imaging target.

In the configuration example shown in FIG. 1, the biological tissue image processing system 10 includes a sample pretreatment device 12, a serial section preparation device 14, a scanning electron microscope (SEM) 16, and a biological tissue image processing device 18. .

The sample pretreatment device 12 is a device that performs pretreatment on the tissue 20 extracted from the living body, or corresponds to various instruments for the pretreatment. Examples of pretreatment include fixing treatment, dyeing treatment, conductive treatment, resin embedding treatment, and shaping treatment. All or part of them are implemented as necessary. Osmium tetroxide, uranium acetate, lead citrate, etc. may be used in the dyeing treatment. A staining process, described below, may be performed on each sample section. Some or all of the steps included in the pretreatment may be performed manually.

The serial sectioning device 14 is provided outside the SEM 16 or inside the SEM 16 . A plurality of sample segments 24 arranged in the depth direction (Z direction) are cut out from the cubic sample after pretreatment by the continuous slice creating device 14 . In that case, a device such as an ultramicrotome may be used. The work may be done manually. A plurality of sample segments 24 constitute a sample segment group 22 . In practice, a plurality of cut sample pieces are arranged in a predetermined array on the substrate 28 . The substrate 28 is, for example, a glass substrate or a silicone substrate. In FIG. 1, the sample slice array 22A consisting of two rows of sample slices is formed on the substrate 28, but this is merely an example. A sample unit 26 is composed of the substrate 28 and the sample slice array 22A.

Incidentally, the vertical and horizontal sizes of each sample section 24 are, for example, nm order or μm order. A sample section 24 having a larger size (for example, a size on the order of mm) may be produced. The thickness (size in the Z direction) of each sample slice 24 is, for example, several nanometers to several hundreds of nanometers. All numerical values given in the specification of the present application are examples.

The SEM 16 has an electron gun, deflector (scanner), objective lens, sample chamber, detector 34 and the like. A stage for holding the sample unit 26 and a moving mechanism for moving the stage are provided in the sample chamber. A control unit in the SEM 16 controls the operation of the moving mechanism, that is, the movement of the stage. Specifically, the electron beam 30 is applied to a specific sample section 24 selected from the sample section array 22A. The backscattered electrons 32 emitted from each irradiation position are detected by the detector 34 while scanning the irradiation position (for example, raster scanning). This forms an SEM image. This is done for each sample section 24 . The result is a plurality of SEM images 38 that are observations of a plurality of sample sections 24 . A plurality of SEM images 38 constitute an SEM image stack 36 .

If the direction orthogonal to the Z direction is defined as the X direction, and the direction orthogonal to the Z direction and the X direction is defined as the Y direction, it is preferable that the observation range in the X direction and the observation range in the Y direction match each other. , an observation range (electron beam two-dimensional scanning range) in each sample section 24 is determined. Secondary electrons or the like may be detected instead of the backscattered electrons 32 . A control unit in the SEM 16 also has a function of varying the acceleration voltage for forming the electron beam 30 . The acceleration voltage during electron beam formation may be switched according to the biological tissue, the object to be observed in the biological tissue, the purpose of observation, and the like. In general, the same acceleration voltage is set in the learning process of the film estimator and the subsequent analysis process of the biological tissue, which will be described later.

The SEM image stack 36 is composed of multiple SEM images 38 corresponding to multiple depths in the Z direction (multiple depths on the depth axis) (in other words, aligned in the Z direction in the data storage space). be done. Each SEM image 38 is an original image or an input image viewed from the biological tissue image processing device 18 side. Each SEM image 38 is electronic data, and each SEM image is transmitted from the SEM 16 to the biological tissue image processing device 18 via a network or a portable storage medium.

The biological tissue image processing apparatus 18 is configured by a personal computer in the illustrated configuration example. Biological tissue imaging device 18 may be incorporated within SEM 16, or biological tissue imaging device 18 may be incorporated within a system computer that controls SEM 16 and the like. SEM 16 may be controlled by biological tissue imaging device 18 .

The biological tissue image processing device 18 has a main body 40 , a display 46 and an input device 48 . A plurality of functions of the main body 40 will be described later in detail with reference to FIG. 2 and subsequent drawings. In FIG. 1, two representative functions (a film estimation function and a processing function) exhibited by the main body 40 are expressed as blocks. Specifically, the main body 40 has a machine learning type film estimator 42 and a processing tool unit (work support unit) 44 . The display 46 is configured by an LCD, an organic EL display device, or the like. The input device 48 is configured by a keyboard, pointing device, etc. operated by the user.

FIG. 2 shows a first configuration example of the main body 40. As shown in FIG. The main body 40 has a machine learning type film estimator 42, a binarizer (image generator) 50, a processing tool unit (work support unit) 44, a volume data processing unit 56, and the like. The processing tool unit 44 has a correction section 52 and a labeling processing section 54 . The volume data processing section 56 has an analysis section 56A and a rendering section 56B.

The entity of each configuration shown in FIG. 2 is basically software, that is, a program executed by a general-purpose processor such as CPU, GPU, etc., except for the portion corresponding to user's work or action. However, part or all of those configurations may be configured by dedicated processors or other hardware. All or part of the functions of the biological tissue image processing apparatus may be executed by one or more information processing devices present on a network. Each configuration in the main body 40 will be described in detail below.

The membrane estimator 42 functions as a membrane estimator. The membrane estimator 42 applies membrane estimation processing to the input original image (SEM image) 58 and outputs a membrane likelihood map 60 by this. In the illustrated configuration example, the film estimator 42 is configured by a CNN (Convolutional Neural Network). A specific configuration example thereof will be described later with reference to FIG. A CNN learning process is pre-performed prior to the actual running of the CNN so that the membranes are correctly estimated by the CNN. The learning process includes a primary learning process (initial learning process) and a secondary learning process.

In the primary learning process, a plurality of image pairs forming training data are provided to the membrane estimator 42, thereby improving (optimizing) the CNN parameters in the membrane estimator 42. FIG. That is, machine learning results are accumulated in the membrane estimator 42 . Here, each image pair is composed of an original image (SEM image) 58 and a correct image 68 corresponding thereto. The correct image 68 is created manually for the original image 58, for example. The correct image 68 may be created from the original image 58 by an unsupervised machine learning device, a simple classifier (for example, SVM (Support Vector Machine)), or the like. In this case, the original image 58 may be input to such a discriminator, and the output may be used as the correct image 68 when the user can determine that the discriminator is working to some extent based on the output. Correct image 68 may be generated based on the output of membrane estimator 42 after undergoing a certain primary learning process.

In the subsequent secondary learning process, on the premise that the film estimator 42 works to some extent through the primary learning process, the film estimator 42 is provided with a plurality of image pairs as teacher data in the same manner as in the primary learning process. In the embodiment, the teacher data consists of a plurality of image pairs used in the primary learning process and a plurality of image pairs added in the secondary learning process. Each added image pair consists of an original image 58 and a corresponding correct image 64A. The correct image 64A is created by the biological image processing apparatus itself by the configuration from the membrane estimator 42 to the processing tool unit 44. FIG. Specifically, when the original image 58 is input to the membrane estimator 42, the membrane estimator 42 outputs a membrane likelihood map 60 as an estimation result image. A correct image 64A is created through generation of a provisional membrane image based on the membrane likelihood map 60 and user (expert) modification of the provisional membrane image 62 using the processing tool unit 44 . Individual processing will be detailed later. It is also conceivable to use the temporary membrane image 62 as the correct image 62A. A secondary learning process further refines the set of CNN parameters in film estimator 42 . That is, more machine learning results are accumulated in the membrane estimator 42 . The secondary learning process ends, for example, when it is determined that the result of the estimation process on the original image 58 is sufficiently similar to the correct images 68 and 64A corresponding to the original image 58. FIG. After that, if necessary, the film estimator 42 is re-learned by the same method as described above.

In the secondary learning process, a correct image can be created based on the film estimation result, so the burden on the user is greatly reduced compared to manually creating all correct images from the original image. The same applies to the necessary learning process after the secondary learning process. In the primary learning process and the secondary learning process, a plurality of image pairs may be collectively given to the film estimator 42 and batch-processed.

A database 57 stores a plurality of original images 58, a plurality of correct images 68 and 68A, and the like. Film estimator 42 and database 57 may be integrated. U-net may be used as the machine learning type film estimator, or SVM, random forest, etc. may be used.

The binarizer 50 functions as an image generator. Specifically, the binarizer 50 is a module that generates a temporary membrane image 62 by binarizing the membrane likelihood map 60, as illustrated later with reference to FIG. The membrane likelihood map 60 consists of a plurality of membrane likelihoods arranged two-dimensionally. Each membrane likelihood is a numerical value indicating the likelihood (probability) of being a membrane. The membrane likelihood takes a value between 0 and 1, for example. The membrane likelihood map 60 can also be regarded as a membrane likelihood image. In the embodiment, a threshold value is set in the binarizer 50, and membrane likelihoods equal to or greater than the threshold are converted to 1, and membrane likelihoods lower than the threshold are converted to 0. The resulting image is the temporary membrane image 62 . In the embodiment, the membrane image before correction is called a temporary membrane image 62 in order to distinguish the membrane image before correction from the membrane image after correction for convenience.

Both the film estimator 42 and the binarizer 50 constitute an image generator 61 . The entire image generator 61 may be configured by CNN or the like. Even then, stepwise generation of membrane likelihood maps and provisional membrane images can be envisioned. It is also possible to use the temporary membrane image 62 as the correct image 62A. Processing such as noise removal and edge enhancement may be applied to the membrane likelihood map 60 before the binarization processing.

The processing tool unit 44 functions as a work support section or work support means. From the viewpoint of information processing, the processing tool unit 44 has a display processing function and an image processing function. From the viewpoint of work, it has a correction function and a labeling function, and these functions are expressed as a correction section 52 and a labeling processing section 54 in FIG. A plurality of specific functions exhibited by the processing tool unit 44 will be organized later with reference to FIG. Here, its outline or its main functions will be described.

The correction unit 52 displays the temporary membrane image, which is the work target, to the user as the work target image via a work window illustrated later in FIG. It is. The content of the correction is reflected in the temporary film image that is the image to be worked on. If the image to be worked on includes, for example, a discontinuous portion of the film, a film pixel group is added to the discontinuous portion. For example, if a part other than the film is included in the image to be worked on and the part is misidentified as the film, the film pixel group that constitutes the part is deleted. The correction unit 52 is a module that supports the work or operation of the user and manages each temporary membrane image in such addition and deletion.

In the illustrated configuration example, in addition to the generated temporary membrane image 62, the input original image 58 and the generated membrane likelihood map 60 are also sent to the correction unit 52 (or the processing tool unit 44). are entered sequentially. Accordingly, it is possible to display the original image 58 or the membrane likelihood map 60 corresponding to the work target image together with or instead of the temporary membrane image as the work target image. This will be explained later with reference to FIGS. The interpolation processing executed in the correction unit 52 will be described later using FIGS. 9, 10 and the like.

The labeling processing unit 54 is a module for labeling (painting and labeling) individual regions (cell lumens) included in the corrected membrane image (or uncorrected membrane image). Labeling includes manual labeling by the user and automatic labeling. Labeling will be described later with reference to FIGS. A management table for managing labeling results will be exemplified in FIG. 14 later. Three-dimensional labeling data 66 in which the intracellular space and the rest are distinguished is constructed when the correction work and the labeling work are completed. It is sent to the volume data processing unit 56 .

In the illustrated configuration example, the volume data processing unit 56 receives an original image stack (SEM image stack) 36 composed of a plurality of original images. The original image stack 36 constitutes the volume data. The volume data processing unit 56 also receives the three-dimensional labeling data 66 as described above. It is also a kind of volume data.

56 A of analysis parts analyze a target organ (for example, nerve cell) based on the three-dimensional labeling data 66, for example. For example, morphology, volume, length, etc. may be analyzed. At that time, the original image stack 36 is analyzed with reference to the three-dimensional labeling data. The rendering section 56B forms a three-dimensional image (stereoscopic representation image) based on the three-dimensional labeling data 66. FIG. For example, based on the three-dimensional labeling data 66, an imaged portion may be extracted from the original image stack 36 and a rendering process applied thereto.

In the already explained secondary learning process, each film image 64 after correction is input to the film estimator 42 as a correct image 64A. Since the estimation result by the film estimator 42 can be used and the processing tool unit 44 can be used when producing each correct image 64A, each correct image 64A can be compared with the case where they are not used. The burden of manufacturing is greatly reduced. That is, according to the above configuration, it is possible to reduce the labor and time required for the learning process of the film estimator 42 . In the analysis process after the secondary learning process, the film estimator 42 and the processing tool unit 44 are used together to improve the quality of the image group to be analyzed or rendered, and to easily and quickly generate the image group. It is possible to As will be described later, the processing tool unit 44 also functions as a three-dimensional image processing section (three-dimensional image generating means) and a candidate selecting section (candidate selecting means).

FIG. 3 schematically shows a configuration example of the film estimator 42. As shown in FIG. The membrane estimator 42 has multiple layers, including an input layer 80, a convolutional layer 82, a pooling layer 84, an output layer 86, and so on. They act according to the CNN parameter set 88 . CNN parameters 88 include a number of weighting factors, a number of bias values, and so on. CNN parameter group 88 is initially configured by initial value group 94 . For example, the initial value group 94 is generated using random numbers or the like.

In the learning process, the evaluation section 90 and the update section 92 function. For example, the evaluation unit 90 calculates an evaluation value based on a plurality of image pairs (original image 58 and corresponding correct images 68 and 64A) that constitute teacher data. Specifically, the evaluation value is calculated by sequentially applying the result 60A of the estimation processing for the original image 58 and the correct images 68 and 64A corresponding to the original image 58 to the error function. The updating unit 92 updates the CNN parameter group 88 so that the evaluation value changes in a favorable direction. By repeating it, the CNN parameter set 88 is globally optimized. In practice, the end of the learning process is determined when the evaluation value reaches a certain value. The configuration shown in FIG. 3 is merely an example, and estimators having various structures can be used as film estimator 42 .

FIG. 4 shows the operation of the binarizer. A membrane likelihood map 60 is output from the membrane estimator. The membrane likelihood map 60 consists of a plurality of membrane likelihoods 60a corresponding to a plurality of pixels, and each membrane likelihood 60a is a numerical value indicating the likelihood of being a membrane. As indicated by reference numeral 50A, the binarizer binarizes the membrane likelihood map 60, thereby generating a temporary membrane image 62 as a binarized image. For binarization, each membrane likelihood 60a is compared with a threshold. For example, film likelihoods 60a greater than or equal to the threshold are converted to pixels 62a having a value of one, and film likelihoods 60a less than the threshold are converted to pixels 62b having a value of zero. The pixel 62a is treated as a pixel forming a film (film pixel). The threshold can be variably set by the user. The setting may be automated. For example, the threshold may be varied while observing the temporary membrane image 62 .

FIG. 5 organizes a plurality of functions possessed by the processing tool unit. The upper part (A) shows a plurality of functions of the correction unit, and the lower part (B) shows a plurality of functions of the labeling processing unit. Details or specific examples of these functions will be described with reference to FIG. 6 and subsequent drawings. This section outlines their functions.

First, the functions of the correction section will be described. Corrections to the temporary membrane image are roughly classified into manual correction (A1) and automatic correction (A2). Manual correction (A1) uses a virtual pen and a virtual eraser as correction tools. The pen is a correction tool for drawing membranes (converting non-membrane pixels into membrane pixels), and the eraser is a correction tool for erasing membranes (converting membrane pixels into non-membrane images). Other correction tools may be provided.

During the manual correction (A1), as indicated by (A11), the temporary membrane image corresponding to the depth selected by the user is displayed in color as the image to be worked on. At that time, the original image corresponding to the selected depth is displayed as a black and white image as a background image. As shown in (A12), it is also possible to superimpose and display multiple images of the same type (original image, membrane likelihood map, or temporary membrane image) arranged in the depth direction based on the selected depth. is. Further, as indicated by (A13), correction candidates may be displayed to assist manual correction work.

The automatic correction (A2) automatically interpolates the film discontinuous portion included in the temporary film image. Specifically, two points specified by the user are automatically connected. Several functions are prepared for this purpose, and one of them is selected as required. For example, as indicated by (A21), interpolation may be made between two points on the image for work based on the image for work (temporary membrane image). As indicated by (A22), interpolation may be performed between two points on the image for work based on the original image corresponding to the image for work. As indicated by (A23), interpolation may be performed between two points on the image for work based on the likelihood map corresponding to the image for work. As indicated by (A24), three-dimensional interpolation may be performed based on a plurality of images (original image, membrane likelihood map, or temporary membrane image) arranged in the depth direction.

Next, the function of the labeling processor will be described. As labeling methods, there are manual labeling (B1) and automatic labeling (B2).

In manual labeling (B1), a labeling tool is used, and a user designates a specific region (closed region) as a labeling target. The area is then painted and a unique label is given to the area. A label is an identifier or an attribute value for managing connection relationships. At that time, as shown in (B11), an image corresponding to another depth (for example, an adjacent provisional membrane image that has been labeled) may be overlaid on the work target image, if necessary. . Alternatively, as indicated by (B12), the connection relationship between regions in the depth direction may be automatically determined, and one or more labeling candidates may be displayed on the work target image. At that time, the degree of connection possibility may be automatically determined, and the display mode of each labeling candidate may be controlled based on the determination result.

In automatic labeling (B2), the connection relation of regions between layers is automatically determined, and painting and labeling are automatically performed by this. In that case, for example, as shown in (B21), for a specific labeled region, define a perpendicular (virtual line parallel to the Z direction) passing through the representative coordinates, and one or more , the same label as the label attached to the specific area may be attached (skewering method). In that case, a certain range may be set on both sides in the depth direction with the depth at which the specific region exists as the reference depth, and automatic labeling may be performed within that range. Further, as indicated by (B22), based on the degree of similarity between regions in the depth direction, etc., the connection relationship between layers is automatically determined, and labeling is automatically performed based on the determination result. can be When automatic labeling is performed, the labeling result is usually visually confirmed by the user.

The above-mentioned functions are examples, and the processing tool unit is used so that the correction work and labeling work by the user are efficient, the burden is reduced, and the quality of the three-dimensional labeling data is improved. preferably configured.

FIG. 6 exemplifies a working window displayed by the processing tool unit. In the illustrated example, task window 100 includes display image 102 . The displayed image 102 is a composite image consisting of the temporary membrane image (worked image) corresponding to the depth selected by the user and the original image. An original image as a grayscale image constitutes a background image, and a temporary membrane image (work target image) as a color image (for example, a blue image) is displayed superimposed thereon. It should be noted that the illustrated display image 102 shows brain tissue, in which cross sections of a plurality of cells appear. At the same time, cross-sections of intracellular organelles (mitochondrion, etc.) also appear.

When tab 104 is selected, display image 102 described above is displayed. When tab 105 is selected, only the original image as a grayscale image is displayed. When tab 106 is selected, only the membrane likelihood map (membrane likelihood image) as a grayscale image is displayed. By observing the likelihood map, it can be confirmed that the threshold is appropriately set. In addition, by displaying the original image alone, it becomes easy to observe the details of the film in detail. A tab for displaying only the temporary membrane image as the image to be worked on may be added. The membrane likelihood map may be used as a background image, and the temporary membrane image may be displayed superimposed thereon.

The depth selection tool 108 is a display element (manipulation element) for selecting a specific depth (display depth as a display position) in the Z direction (that is, on the depth axis as the spatial axis). It consists of a Z-axis symbol 108b representing the Z-axis and a marker 108a as a slider that slides along the Z-axis symbol 108b. A desired depth can be selected by moving the marker 108a. Such a depth selection tool 108 has the advantage that the depth to be selected and the amount of change in depth can be easily recognized intuitively. Note that the left end point of the Z-axis symbol 108b corresponds to zero depth, and the right end point of the Z-axis symbol 108b corresponds to the maximum depth. Depth selection tools having other forms may be employed. The depth input field 114 is a field for directly specifying the depth as a numerical value. The currently selected depth may be displayed as a numerical value in the depth input field 114 according to the position of the marker 108a.

The transparency adjustment tool 110 is a tool for adjusting the transparency (display weight) of the color temporary film image (work target image) synthesized and displayed while the display image 102 is being displayed. For example, when the marker 110a is moved to the left, the display weight of the color temporary film image is reduced, its transparency is increased, and the original image is predominantly displayed. Conversely, when the marker 110a is moved to the right, the display weight of the color temporary film image is increased, the transparency thereof is decreased, and the color temporary film image is expressed more clearly.

The overlapping display tool 112 displays an image (display image, original image or membrane likelihood map) adjacent to the shallow side in the depth direction with respect to the currently displayed image (display image, original image or membrane likelihood map). Alternatively, it is operated when an image (display image, original image, or membrane likelihood map) adjacent to the deeper side in the depth direction is displayed in an overlapping manner. Moving the marker 112a to the left increases the display weight of the image adjacent to the shallow side, and conversely, moving the marker 112a to the right increases the display weight of the image adjacent to the deep side. Three or more images may be superimposed and displayed. However, if too many images are superimposed, the image contents become too complicated, so it is desirable to superimpose a small number of images. Such overlapping display makes it easier to obtain depth information.

The button row 115 is composed of a plurality of virtual buttons 116, 118, 120, 121, 122, and 126. FIG. A button 116 is a display element operated when zooming (enlarging or reducing) an image. A button 118 is a display element operated when using the pen tool. When button 118 is turned on, the cursor changes to a pen shape, which can be used to add film pixels. It is also possible to change the size of the pen. A button 120 is a display element operated when using an eraser. When button 120 is turned on, the cursor changes to an eraser shape, which can be used to delete film pixels. It is also possible to change the size of the eraser.

A button 121 is a display element operated when painting. When the button 121 is turned on and any area is specified, that area is painted out. By operating the button 121, it is also possible to select an arbitrary function from a plurality of functions prepared for painting (or labeling). By operating the object numbering (labeling) button 122, a color palette 124 is displayed. For example, painted areas are given a color selected from a color palette. This causes the area to be tinted with the selected color. Each individual color is associated with an object number. If regions are given the same color or object number across layers, they define a three-dimensional lumen region within a particular cell.

A button 126 is a button for black-and-white inversion. When it is operated, in the displayed image, the part displayed in black is displayed in white, and conversely, the part displayed in white is displayed in black.

In addition to the above, it is desirable to provide a button for displaying a three-dimensional image. Display of a three-dimensional image will be described later with reference to FIGS. 15 and 16. FIG. The content of the work window 100 shown in FIG. 6 is merely an example, and it is desirable that the content be appropriately determined so as to improve the workability of the user's work.

Display processing in the processing tool unit will be described based on FIGS. 7 and 8. FIG. The processing shown in FIG. 7 corresponds to the display processing indicated by (A11) in FIG. The processing shown in FIG. 8 corresponds to the display processing shown by (A12) and (B11) in FIG.

In FIG. 7, the original image stack 130 is composed of a plurality of original images arranged in the Z direction (depth direction). The membrane-likelihood map stack 132 is composed of a plurality of membrane-likelihood maps arranged in the Z direction. The temporary membrane image stack 134 is composed of a plurality of temporary membrane image stacks aligned in the Z direction. Each temporary membrane image becomes a membrane image after being corrected, but this is a distinction for the sake of convenience, and here the temporary membrane image is referred to without distinguishing between before and after correction. These stacks 130, 132, 134 are managed by a processing tool unit or by a controller of the biological tissue processing apparatus.

If display of the original image is specified, the original image 130 i corresponding to the depth Z i selected by the user is retrieved from the original image stack 130 and displayed on the display 46 . If display of the membrane likelihood map is specified, the membrane likelihood map 132 i corresponding to the depth Z i selected by the user is retrieved from the membrane likelihood map stack 132 and displayed on the display 46 . . If display of a temporary membrane image is specified, the temporary membrane image 134 i corresponding to the depth Z i selected by the user is retrieved from the temporary membrane image stack 134 and displayed on the display 46 . Actually, the temporary membrane image 134 i corresponding to the depth Zi is synthesized on the original image 130 i corresponding to the depth Zi, and the synthesized image is displayed on the display 46 . Prior to synthesis, the temporary membrane image 134i is converted into, for example, a blue color image. Further, at the time of synthesis, the original image 130i is multiplied by the display weight w0, and the temporary membrane image 134i is multiplied by the display weight w1. Each display weight w0, w1 can be changed by the user.

The temporary membrane image 134i is an image to be worked on, and when the user modifies it, the contents of the correction are immediately reflected in the temporary membrane image 134i, as indicated by reference numeral 135. FIG. Correction processing and labeling processing are performed on each displayed work target image while changing the depth Zi. In the process, it is also possible to display the original image 130i or the membrane likelihood map 132i alone, if necessary. In the embodiment, the past m correction contents are managed as a correction history in the ring buffer, and an Undo operation can be performed as necessary.

FIG. 8 shows display processing when superimposed display is selected. Components similar to those shown in FIG. 7 are denoted by the same reference numerals, and descriptions thereof are omitted. In the illustrated display processing, it is possible to selectively apply superimposition display processing to three types of display images. The overlapping display process may be applied to only some images.

When the display of the original image is designated, and the overlapped display of the adjacent image shallower in the Z direction is instructed, the original image 130i corresponding to the depth Zi and the depth Zi The original image 130i-1 corresponding to -1 is retrieved. They are polymerized and displayed. When superimposing the original image 130i+1 corresponding to the depth Zi+1 on the original image 130i, the same process as described above is performed. The transparency of the additionally displayed original images 130i-1 and 130i+1 is adjusted by varying the display weight w3.

When the display of the membrane-likelihood map is specified, and when superimposed display of adjacent images on the shallow side (or deep side) in the Z direction is instructed, the membrane-likelihood map stack 132 is displayed at the depth Zi. The corresponding membrane-likelihood map 132i and membrane-likelihood map 132i-1 (or membrane-likelihood map 132i+1) corresponding to depth Zi-1 (or depth Zi+1) are retrieved. They are polymerized and displayed.

Similarly, when display of a temporary membrane image is specified and superimposition display of an adjacent image on the shallow side (or deep side) in the Z direction is instructed, from the temporary membrane image stack 134, a deep A temporary membrane image 134i corresponding to depth Zi and a temporary membrane image 134i-1 (or temporary membrane image 134i+1) corresponding to depth Zi-1 (or depth Zi+1) are retrieved. They are polymerized and displayed. In that case, two adjacent synthesized images are actually superimposed and displayed. A polymerization treatment may be performed after the synthesis treatment.

FIG. 9 shows a first example of interpolation processing. This corresponds to the interpolation processing indicated by (A21) in FIG. In FIG. 9, the temporary membrane image corresponding to the depth Z i is retrieved from the temporary membrane image stack 134 and displayed as the working image 140 . In the process of correcting the image 140 to be worked on, the user designates two points 142 and 144 at both ends of the film discontinuity portion 146 . Between the two points 142 and 144 is automatically interpolated based on the content of the working image 140 . Reference numeral 148 indicates an automatically added interpolation line (connection line). In that case, spline interpolation or other techniques may be used. In addition, when a cellular component other than the cell membrane is detected as a membrane in the work target image 140 , it is erased with an eraser, as indicated by reference numeral 150 .

FIG. 10 shows a second example of interpolation processing. This corresponds to the interpolation processing indicated by (A22) in FIG. In FIG. 10, the temporary membrane image corresponding to the depth Zi is retrieved from the temporary membrane image stack 134 and displayed as the working image 140 . On the image 140 to be worked on, the user designates two points 142 and 144 at both ends of the film discontinuity. Meanwhile, from the original image stack 130, an original image 152 corresponding to depth Zi is also retrieved. Edge enhancement processing is applied to the original image 152 as necessary. Then, two points corresponding to the two points 142 and 144 are identified on the original image 152 (see reference numeral 154), and interpolated between those points (see reference numeral 156). For example, a route connecting two points is searched. In that case, various known techniques can be used. In FIG. 10, an interpolation line 158 having the same shape as the searched path is added on the image 140 to be worked on. Various methods of solving the shortest path problem, for example, can be used when searching for paths. For example, the Dijkstra method may be used. A three-dimensional route may be searched by a technique similar to that described above.

FIG. 11 shows a temporary membrane image 180 before correction and a temporary membrane image 182 after correction. A temporary membrane image 180 before correction includes membrane discontinuities 184 and 188 . In the temporary film image 182 after correction, as indicated by reference numerals 186 and 190, the broken film portions are repaired. In addition, the temporary membrane image 180 before correction includes non-membrane portions 192 and 196 . In the corrected temporary membrane image 182 , those non-membrane portions have disappeared, as indicated by reference numerals 194 and 198 . According to the corrected temporary film image 182, it is possible to perform labeling accurately, which in turn makes it possible to improve the quality of the three-dimensional labeling data.

FIG. 12 exemplifies three-dimensional connection processing included in the labeling processing. The content corresponds to the process indicated by (B21) or the process indicated by (B22) in FIG. FIG. 12 shows a plurality of temporary film images D1 to D4 arranged in the Z direction. The temporary film image D1 corresponding to the depth Zi is the work target image. It is used as a reference image in the processing described below.

In the process shown in (B21) above, a representative point is specified for the region R1 included in the temporary membrane image D1, which is the reference image. For example, a central point, a barycentric point, or the like is specified as a representative point. A perpendicular line C is then defined that passes through the representative point. From the reference image, for example, N temporary membrane images are referenced on the deep side, and each region through which the perpendicular line C passes is specified in these images. The same label is assigned to the regions R1, R2, R3, R4, . . . through which the perpendicular C passes. Also, the above processing may be applied to the N temporary membrane images on the shallow side from the reference image. The result of automatic labeling is usually visually confirmed by the user.

In the process shown in (B22) above, (the outer edge of) the area R1 included in the temporary membrane image D1 is projected onto the temporary membrane image D2 to define the projection area R1a. A region R2 that most overlaps with the projection region R1a is specified on the temporary membrane image D2. Subsequently, the region R2 is projected onto the temporary membrane image D3 to define a projection region R2a. A region R3 that most overlaps with the projection region R2a is specified on the temporary membrane image D3. Similarly, a projection region R3a is defined on the temporary membrane image D4, and a region R4 is specified based thereon. The same label is assigned to the region R1 and all of the regions R2, R3, and R4 identified from it as a starting point.

In the above process, the projection source area is updated when the layer is changed, but it may be fixed. For example, the region R1 may be projected onto the temporary membrane images D2, D3, and D4. Also, in the above processing, the connection destination is searched for on one side in the Z direction from the reference image, but the connection destination may be searched for on both sides in the Z direction from the reference image. The search scope may be user-selected. In any case, the result of automatic labeling is usually visually confirmed by the user. In that case, the working window shown in FIG. 6 is used.

The three-dimensional connection processing described above is an example, and three-dimensional connection processing other than the above may be adopted. One or a plurality of feature amounts for each region may be used to identify the connection relationship between regions. Area, shape, perimeter, center of gravity, luminance histogram, texture, etc. may be used as the feature amount of the region. Alternatively, a superimposed area, a distance between representative points, or the like may be used as the feature amount between regions.

FIG. 13 shows an example of candidate display. This process corresponds to the process indicated by (A13) in FIG. A projection area 202 is defined on the temporary membrane image (work target image) 200 corresponding to the depth Zi. Projection region 202 is, for example, the region defined by the projection of the labeled original region on the tarsal image corresponding to depth Zi−1. In the temporary membrane image 200, a plurality of regions 204a, 204b, 204c overlapping the projection region 202 are identified. Further, similarities (for example, overlapping areas) are calculated between the projected area 202 (or the original area) and each of the areas 204a, 204b, and 204c. Based on the degrees of similarity calculated for the regions 204a, 204b and 204c, the regions 204a, 204b and 204c are identified and expressed in different colors. That is, a difference in similarity is identified and expressed as a difference in hue. This allows the user to perform labeling while referring to the degree of similarity of the individual regions 204a, 204b, and 204c. For example, in the illustrated example, the same label as the label given to the original region is given to the region 204a with the highest degree of similarity.

FIG. 14 exemplifies a management table. The illustrated management table 206 is a table for managing the results of labeling processing. The management table 206 is constructed, for example, in the labeling processing section. The management table 206 consists of a plurality of records 206a, each record 206a corresponding to one pixel. The record 206a manages pixel coordinates, classes, and labels. A class is data that identifies membrane, inside membrane, and others as attributes of a pixel. A label is numerical data specifying a connection relationship in a three-dimensional data space. The same label is assigned to a group of pixels (within a certain cell membrane) that are connected.

Next, another example of the work window will be described with reference to FIGS. 15 and 16. FIG. In FIG. 15, the same reference numerals are given to the same configurations as those shown in FIG. 6, and the description thereof will be omitted.

In FIG. 15, a button row 115 in the work window 100A includes a button 210 for displaying a three-dimensional image. When the button 210 is operated, a sub-display image 212 is displayed on the display image 102 or at another location. In the illustrated example, the sub-display image 212 includes a wire frame (a figure indicating a coordinate system) 213 representing a three-dimensional space, and a three-dimensional image 215 expressed therein. The three-dimensional image 215 is an assembly of three parts 214a, 214b, 214c in the illustrated example. Each portion 214a, 214b, 214c respectively shows the three-dimensional morphology of the cell lumen.

Specifically, a three-dimensional image 215 having a stereoscopic effect is generated by executing rendering processing based on the three-dimensional labeling data generated by the labeling processing. Parameters such as viewpoint, light source, etc. can be changed by the user. The sub display image 212 may be popped up in the form of another window. By referring to the three-dimensional image 215, it is possible to easily judge the validity of the correction result and the validity of the labeling processing result.

The sub-display image 212 includes a frame figure 216 indicating the currently selected depth. When the marker 108a is slid in the depth selection tool 108 included in the work window 100A, the content of the displayed work target image changes, and at the same time the frame figure 216 moves in the depth direction. By referring to the position of the frame graphic 216, it is possible to intuitively recognize the work target layer in the three-dimensional data space.

Three-dimensional images 226, 227 shown in FIG. 16 may be displayed. Three-dimensional images 226 and 227 represent a cell lumen as a stack of multiple discs 228 . With such three-dimensional images 226 and 227, it is possible to reflect the thickness of each sample section in the three-dimensional image. A plurality of flat discs may be displayed instead of the plurality of discs.

FIG. 17 shows a second configuration example of the main body of the biological tissue image processing apparatus. Components similar to those shown in FIG. 2 are denoted by the same reference numerals, and descriptions thereof are omitted.

In the main body 40A, the membrane estimator 250 has a function of simultaneously estimating cell membranes and other membranes (mitochondrial membranes in the configuration example shown). When the original image (SEM image) 58 is input to the membrane estimator 250, a membrane likelihood map A254 is output as the cell membrane estimation result, and at the same time, a membrane likelihood map B258 is output as the mitochondrial membrane estimation result.

The membrane likelihood map correction unit 262 is a module that corrects the membrane likelihood map A based on the membrane likelihood map B. FIG. Specifically, if the membrane likelihood b at the coordinates (x, y) forming the membrane likelihood map B258 is equal to or greater than the determination threshold α, the membrane likelihood a at the coordinates (x, y) in the membrane likelihood map A254 is replaced with 0. On the other hand, if the membrane likelihood b at the coordinates (x, y) in the membrane likelihood map B258 is smaller than the determination threshold α, the value of the membrane likelihood a at the coordinates (x, y) in the membrane likelihood map A254 is maintained. be. The corrected membrane likelihood map 264 is output from the membrane likelihood map correction unit 262 . A binarizer 50 binarizes the membrane likelihood map 264 to generate a temporary membrane image 62 .

A binarizer 268 and a modifier 270 provided in parallel with the binarizer 50 and the modifier 52 are for creating a correct image 272 based on the membrane likelihood map B258. A binarizer 268 has the same function as the binarizer 50 and generates a tentative membrane image from the membrane likelihood map B. FIG. Corrector 270 has the same function as corrector 52 . The correction unit 52 manually or automatically corrects the temporary membrane image, thereby generating the correct image 272 . The processing tool unit 266 is composed of a correction section 52 , a labeling processing section 54 and a correction section 270 .

For cell membrane learning in the membrane estimator 250, the original image 58 and the corresponding correct images 64A and 68 are used as teacher data. A correct image 64A is the film image 64 after correction by the correction unit 52, and a correct image 68 is an image separately created from the original image. For learning about the mitochondrial membrane in the membrane estimator 250, the original image 58 and the corresponding correct images 272 and 274 are used as teacher data. A correct image 272 is a film image corrected by the correction unit 270, and a correct image 274 is an image separately created from the original image. Note that the temporary membrane image 62 can be used as the training data instead of the correct image 64A.

According to the second configuration example, the estimation result of the cell membrane is corrected by another estimation result, so it is possible to improve the estimation accuracy of the cell membrane. Moreover, since the work window shown in FIG. 6 can be used when correcting the membrane image showing the mitochondrial membrane, the work efficiency can be improved in the correction work.

In the second configuration example, the correction is performed when the membrane likelihood in the membrane likelihood map B258 is equal to or greater than the threshold value α, but the degree of correction varies depending on the magnitude of the membrane likelihood in the membrane likelihood map B258. may be made. Instead of the single membrane estimator 250, a membrane estimator for estimating the cell membrane and a membrane estimator for estimating the mitochondrial membrane may be provided side by side. In that case, each of the two membrane estimators preferably consists of a machine learning type estimator. A multi-class estimator may be provided that separately and simultaneously estimates plasma membrane, mitochondria, and background.

FIG. 18 shows a third configuration example of the main body of the biological tissue image processing apparatus. The main body 40B has two machine-learning film estimators 280 and 282 that input the original image 58 and perform film estimation. The two machine-learning film estimators 280, 282 have different configurations, ie their types or schemes are different. For example, membrane estimator 280 corresponds to membrane estimator 42 shown in FIG. The membrane estimator 282 is constructed by, for example, a random forest. Both of the membrane estimators 280 and 282 are for estimating the cell membrane, but since their properties or characteristics are different, the two membrane estimators 280 and 282 do not always give the same estimation results. For example, one of the film estimators 280, 282 may determine that a portion of the original image is a film, and the other of them may also be determined to be a film. One may be determined to be membrane and the other of them may be determined to be non-membrane.

The processing unit 296 compares the membrane likelihood map A 284 output from the membrane estimator 280 and the membrane likelihood map B 286 output from the membrane estimator 282, and corrects pixels (or groups of pixels) having a large difference between the two. It is determined as a candidate. In the illustrated configuration example, information 302 specifying correction candidates is sent from the processing unit 296 to the correction unit 306 . A processing unit 296 may be provided within the correction unit 306 .

The binarizer 292 corresponds to the binarizer 50 shown in FIG. 2 and it produces the tentative membrane image 298 from the membrane likelihood map A 284 . The binarizer 294 performs the same function as the binarizer 292, in that it produces the tentative membrane image 300 from the membrane likelihood map B286.

The processing tool unit 304 has similar functions to the processing tool unit 44 shown in FIG. The correction unit 306 is for correcting the temporary membrane image 298 output from the binarizer 292 and the temporary membrane image 300 output from the binarizer 294 . The modified film image 308 is sent to the labeling processor 310 . The membrane image 308 is sent to the membrane estimator 280 as the correct image 312 . The modified film image 309 is sent to the film estimator 282 as a correct image 316 . The processing tool unit 304 has a function of identifying and displaying correction candidates when displaying the temporary membrane image corresponding to the selected depth or when displaying the input temporary membrane image 298 . The identification of correction candidates and their identification display will be described later with reference to FIGS. 19 and 20. FIG. Note that the modified film image 309 may be sent to the labeling processing unit 310 .

In the learning process of the membrane estimator 280, the original image 58 and the corresponding correct images 312 and 314 are input to the membrane estimator 280 as teacher data. Correct image 312 is film image 308 after correction. The correct image 314 is created separately from the original image. In the learning process of the membrane estimator 282, the original image 58 and the corresponding correct images 316 and 318 are input to the membrane estimator 282 as teacher data. Correct image 316 is film image 309 after correction. A correct image 318 is created separately from the original image.

Based on FIG.19 and FIG.20, the operation|movement (especially operation|movement of a process part and a correction part) of the said 3rd structural example is demonstrated.

In FIG. 19, in S30, the membrane likelihood map A and the membrane likelihood map B are compared on a pixel-by-pixel basis. The unit of comparison may be another unit. A difference map is generated by comparing the two membrane likelihood maps A and B. FIG. In S32, a plurality of difference values forming the difference map are sorted. That is, they are ranked. In S34, the top M difference values are identified in the sorted difference value sequence. That is, M difference values are identified, including the largest difference value. A plurality of pixels corresponding to them are candidates for correction. In S36, a plurality of correction candidates are identified and displayed on the work target image. For example, it is displayed with high brightness or with a specific hue.

A display example thereof is schematically shown in FIG. Local portions 322 and 324 on the work target image 320 are portions where a plurality of correction candidate images are concentrated. Normally, a plurality of correction candidate pixels are locally concentrated in this manner. In some cases, multiple correction candidate pixels line up along the edge. In any case, according to such candidate display, the user can specify a portion that should be corrected or a portion where dissociation has occurred in the two membrane estimation results, and perform the necessary correction can be applied. This will improve the accuracy of your repair work.

Furthermore, as a fourth configuration example, there is a mode in which variation information is calculated for each pixel based on N membrane likelihood maps calculated in parallel, correction candidates are specified from the variation information, and they are identified and displayed. (where N is an integer of 3 or more). Specifically, N membrane estimators of different types, a calculator for computing the standard deviation of the membrane likelihood on a pixel-by-pixel basis based on the N membrane-likelihood maps output from them, and a display processor for identifying and displaying pixels that are candidates for correction. Information other than the standard deviation (for example, variance or information corresponding thereto) may be calculated as the variation information. With this configuration, if the N estimation results are not aligned, it is highly likely that it is not a film, so it is displayed as a correction candidate.

According to the above embodiment, by operating the depth selection tool included in the work window, the temporary membrane image corresponding to the desired depth is displayed in the work window, and necessary corrections and labeling can be performed thereon. is possible. It is easy to repeat the correction of the temporary membrane image while changing the depth step by step, and it is possible to efficiently proceed with the correction work while considering the continuity of the tissue. In addition, when the temporary membrane image is displayed, the corresponding original image (of the same depth) is displayed as a background image. While you are at it, you can make corrections.

If the three-dimensional image of the labeled portion can be observed during the correction work, it is possible to judge the suitability of the correction and labeling of the temporary membrane image at each depth. In addition, superimposed display of a plurality of images with different depths makes it possible to easily identify, for example, an unnatural overlap (correction error or labeling error) between two regions.

A plurality of original images may be acquired with a transmission electron microscope or an optical microscope. Instead of cell membranes, intracellular cavities, intracellular organelles, and the like may be subject to estimation by the estimator. Although the continuous section SEM method was adopted in the above embodiment, FIB-SEM method, SBF-SEM method, and other methods may be adopted. An image sequence that is continuous on the time axis may be processed instead of the image sequence that is continuous on the spatial axis.

In the above embodiments, a biological tissue image processing device (or software equivalent thereto) having a film estimator that has undergone a secondary learning process may be provided or delivered to the user. A parameter set that has undergone the secondary learning process may be provided to the user's information processing device from a cloud (server) on the Internet.

10 biological tissue image processing system, 16 scanning electron microscope (SEM), 18 biological tissue image processing device, 42 membrane estimator, 44 processing tool unit (work support unit), 50 binarizer, 52 correction unit, 54 labeling section, 56 volume data processing section;

Claims (17)

By generating a plurality of target element likelihood maps based on a plurality of original images that are consecutive on the spatial axis or the time axis obtained by observing the biological tissue, the target element likelihood map is included in the biological tissue with respect to the plurality of original images. an image generation unit that includes a machine learning estimator that applies processing for estimating a target element, and generates a plurality of target element images from the plurality of original images based on the plurality of target element likelihood maps ;
Extracting a specific attention element image corresponding to a position on the space axis or the time axis selected by the user from the plurality of attention element images, displaying it as a work target image, and modifying the work target image by the user a work support unit for reflecting the above-mentioned specific target element image;
including
The image generator includes a plurality of different types of machine learning estimators as the estimators,
The plurality of original images are input to the plurality of estimators, respectively;
The work support unit identifies and displays correction candidates included in the work target image based on differences between the plurality of target element likelihood maps respectively generated by the plurality of estimators.
A biological tissue image processing apparatus characterized by: In the biological tissue image processing device according to claim 1,
the element of interest is a specific membrane;
the plurality of estimators include a first film estimator and a second film estimator of machine learning type different from the first film estimator;
The first membrane estimator applies a process of estimating the membrane to the plurality of original images, thereby generating a plurality of first membrane likelihood maps;
The second membrane estimator applies a process of estimating the membrane to the plurality of original images, thereby generating a plurality of second membrane likelihood maps;
The work support unit identifies and displays correction candidates included in the work target image based on differences between the plurality of first membrane likelihood maps and the plurality of second membrane likelihood maps.
A biological tissue image processing apparatus characterized by: By generating a plurality of target element likelihood maps based on a plurality of original images that are consecutive on the spatial axis or the time axis obtained by observing the biological tissue, the target element likelihood map is included in the biological tissue with respect to the plurality of original images. an image generation unit that includes a machine learning estimator that applies processing for estimating a target element, and generates a plurality of target element images from the plurality of original images based on the plurality of target element likelihood maps;
Extracting a specific attention element image corresponding to a position on the space axis or the time axis selected by the user from the plurality of attention element images, displaying it as a work target image, and modifying the work target image by the user a work support unit for reflecting the above-mentioned specific target element image;
including
The image generator includes a first estimator and a second estimator as the estimators,
The first estimator applies processing for estimating cell membranes contained in the biological tissue to the plurality of original images, thereby generating a plurality of membrane likelihood maps;
The second estimator is an estimator integrated with or separated from the first estimator, and applies a process of estimating organelles contained in the biological tissue to the plurality of original images, This generates multiple organelle likelihood maps,
a correction unit that corrects the plurality of membrane likelihood maps based on the plurality of organelle likelihood maps;
The image generation unit generates a plurality of membrane images as the plurality of target element images based on the plurality of membrane likelihood maps corrected by the correction unit.
A biological tissue image processing apparatus characterized by: In the biological tissue image processing apparatus according to any one of claims 1 to 3 ,
The work support unit displays a work window including the work target image,
The work window includes a position selection display element that is operated when selecting the position.
A biological tissue image processing apparatus characterized by: In the biological tissue image processing device according to claim 4 ,
The work window further includes an addition display element operated when adding the content of the work target image and a deletion display element operated when deleting the content of the work target image. to be
A biological tissue image processing apparatus characterized by: In the biological tissue image processing apparatus according to claim 5 ,
The work window further includes labeling display elements that are operated when labeling the work target image in predetermined units.
A biological tissue image processing apparatus characterized by: In the biological tissue image processing apparatus according to any one of claims 1 to 6 ,
The work support unit has a function of simultaneously displaying a specific original image corresponding to the selected position as a background image when displaying the work target image.
A biological tissue image processing apparatus characterized by: In the biological tissue image processing apparatus according to any one of claims 1 to 7 ,
The work support unit, in addition to the function of displaying the work target image, has a function of displaying a specific original image corresponding to the selected position, and a function of displaying a specific target element likelihood corresponding to the selected position. with the function of displaying a degree map,
A biological tissue image processing apparatus characterized by: In the biological tissue image processing apparatus according to any one of claims 1 to 8 ,
When displaying any one of the specific original image, the specific target element likelihood map, and the specific target element image corresponding to the selected position, the work support unit may display the image with On the other hand, it has a function of superimposing and displaying another image that is of the same type as the image and that corresponds to a position different from the selected position,
A biological tissue image processing apparatus characterized by: In the biological tissue image processing apparatus according to any one of claims 1 to 9 ,
The work support unit has a three-dimensional image processing unit that creates a three-dimensional image representing the labeling result based on the labeling result for the plurality of target element images and displays the three-dimensional image,
modifying and labeling the image to be worked on while observing the three-dimensional image;
A biological tissue image processing apparatus characterized by: In the biological tissue image processing apparatus according to any one of claims 1 to 10 ,
The work support unit selects one or more second target element images among the plurality of target element images based on the labeling result for the first target element image among the plurality of target element images. has a candidate selection unit that selects labeling candidates for
A biological tissue image processing apparatus characterized by: In the biological tissue image processing device according to claim 11 ,
The candidate selection unit performs each labeling based on the degree of similarity between the labeled portion in the first target element image and each labeling candidate included in the second target element image. change the display mode of candidates,
A biological tissue image processing apparatus characterized by: In the biological tissue image processing apparatus according to any one of claims 1 to 12 ,
the learning process of the estimator includes a primary learning process and a secondary learning process;
In the secondary learning process, a plurality of target element images that have been modified by the work support unit are provided to the estimator.
A biological tissue image processing apparatus characterized by: A plurality of original images corresponding to a plurality of depths obtained by microscopic observation of a biological tissue are input to a plurality of different types of machine learning type film estimators, thereby corresponding to the plurality of depths. obtaining a plurality of membrane likelihood maps;
obtaining a plurality of membrane images corresponding to the plurality of depths based on the plurality of membrane likelihood maps;
A specific membrane image corresponding to the depth selected by the user is extracted from the plurality of membrane images and displayed as an image to be worked, and the user's modification of the image to be worked is reflected in the specific membrane image. and
identifying and displaying correction candidates included in the image to be worked on based on differences between the plurality of membrane likelihood maps respectively generated by the plurality of membrane estimators;
A biological tissue image processing method comprising: In the biological tissue image processing method according to claim 14 ,
training the membrane estimator using a plurality of membrane images that have been modified by the user;
A biological tissue image processing method characterized by: A program for processing a biological tissue image in an information processing device,
A plurality of original images corresponding to a plurality of depths obtained by microscopic observation of a biological tissue are input to a plurality of different types of machine learning type film estimators, thereby corresponding to the plurality of depths. the ability to generate multiple membrane likelihood maps;
a function of generating a plurality of membrane images corresponding to the plurality of depths based on the plurality of membrane likelihood maps;
A specific membrane image corresponding to the depth selected by the user is extracted from the plurality of membrane images and displayed as an image to be worked, and the user's modification of the image to be worked is reflected in the specific membrane image. and
a function of identifying and displaying a correction candidate included in the work target image based on the difference between the plurality of membrane likelihood maps respectively generated by the plurality of membrane estimators;
A program characterized by comprising: 17. The program according to claim 16 ,
including a function to train the membrane estimator using a plurality of membrane images that have been modified by the user;
A program characterized by

Images