EP3403238A1 - Système et procédé de segmentation de minéralogie à multimodalité - Google Patents

Système et procédé de segmentation de minéralogie à multimodalité

Info

Publication number
EP3403238A1
EP3403238A1 EP16705331.3A EP16705331A EP3403238A1 EP 3403238 A1 EP3403238 A1 EP 3403238A1 EP 16705331 A EP16705331 A EP 16705331A EP 3403238 A1 EP3403238 A1 EP 3403238A1
Authority
EP
European Patent Office
Prior art keywords
sample
datasets
fib
sem
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP16705331.3A
Other languages
German (de)
English (en)
Inventor
Sreenivas Naga Bhattiprolu
Tom WAITE
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Carl Zeiss X Ray Microscopy Inc
Original Assignee
Carl Zeiss X Ray Microscopy Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Carl Zeiss X Ray Microscopy Inc filed Critical Carl Zeiss X Ray Microscopy Inc
Publication of EP3403238A1 publication Critical patent/EP3403238A1/fr
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
    • G01N33/24—Earth materials
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N23/00—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00
    • G01N23/22—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by measuring secondary emission from the material
    • G01N23/2206—Combination of two or more measurements, at least one measurement being that of secondary emission, e.g. combination of secondary electron [SE] measurement and back-scattered electron [BSE] measurement
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N23/00—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00
    • G01N23/22—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by measuring secondary emission from the material
    • G01N23/225—Investigating or analysing materials by the use of wave or particle radiation, e.g. X-rays or neutrons, not covered by groups G01N3/00 – G01N17/00, G01N21/00 or G01N22/00 by measuring secondary emission from the material using electron or ion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G06T2207/10061—Microscopic image from scanning electron microscope
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30132—Masonry; Concrete
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
    • H01J2237/00—Discharge tubes exposing object to beam, e.g. for analysis treatment, etching, imaging
    • H01J2237/22—Treatment of data
    • H01J2237/221—Image processing

Definitions

  • Imaging modalities have been used to identify and visualize mineral content of rocks, both in two dimensions (2D) and in three dimensions (3D).
  • these imaging modalities can analyze rock samples from oil and gas extraction operations to determine porosity and mineralogy to model flow and mechanical characteristics of the samples obtained during exploration and production operations.
  • these imaging modalities are characterized as destructive and non-destructive techniques.
  • some modalities only analyze surface features, whereas others can analyze three-dimensional structure.
  • these imaging modalities create image datasets such as 3D volumes or 2D images.
  • Image analysis techniques are then employed to infer mineral content from the volumes and the images created by the different imaging modalities.
  • Non-destructive imaging systems include x-ray computed tomography (CT) microscopy and Scanning Electron Microscopy (SEM) systems. These systems provide the ability to visualize features such as pores, organics and minerals in the samples.
  • CT computed tomography
  • SEM Scanning Electron Microscopy
  • the X-ray CT microscopy systems irradiate the sample with x-rays, typically in a range between 1 and several hundred keV. 2D projection images are collected at multiple angles and a 3D volume of the sample is reconstructed from the projections. While the CT intensity correlates with mineral density, there is no direct way of identifying mineralogy on an X-ray CT microscope system.
  • SEM systems instead irradiate the sample surface with a beam of high energy electrons, typically between 500eV and 30keV.
  • the signals derived from electron-sample interaction are used in constructing high resolution 2D images of the sample surface. This enables the simultaneous operation of SEM in multiple modes such as Back Scattered Electron (BSE), Secondary Electron (SE), Energy Dispersive X-ray (EDX), and
  • EDX Cathodoluminescence
  • Destructive imaging systems include Focused Ion Beam Scanning Electron Microscope (FIB-SEM) systems.
  • a FIB-SEM is a multiple beam system that integrates ion beam and electron beam systems.
  • FIB system irradiates the sample with a focused high- current beam of ions of a source material such as gallium to mill the sample surface with high precision.
  • the milled surface is then imaged at high resolution using the integrated SEM system.
  • the FIB milling and SEM imaging process is repeated until a desired volume is sampled.
  • the SEM images from each slice are stacked to construct a 3D volume of the milled region of the sample.
  • One current imaging analysis technique creates a 3D mineral map of the sample by analyzing volume image datasets of a sample created from x-ray imaging systems. A total mineral content of the sample is then defined, and x-ray attenuation coefficients are calculated for the defined minerals. The technique then segments the grey scale 3D images by identifying characteristic grey scale levels in the images corresponding to the calculated x-ray attenuation coefficients.
  • Another imaging analysis technique employs multi-phase segmentation of 3D x-ray tomography volume image datasets.
  • the 3D x-ray tomography volumes are processed to obtain a standardized intensity grey scale images, which are then segmented into at least 3 phases.
  • the segmentation steps include computing a median/mean-filtered- gradient image of the standardized intensity image, creating an intensity vs. gradient graph from the median/mean-filtered-gradient image and the standardized intensity image, partitioning the intensity vs. gradient graph into at least 3 regions, and using thresholds defining the regions to segment the standardized grey scale image to create the segmented image. Then, volumetric fractions and spatial distributions of the segmented phases are calculated and compared with target values.
  • the invention features a mineralogy segmentation method for a multimodality imaging system.
  • the method comprises generating one or more image datasets of a sample, creating one or more mineral maps covering at least part of the one or more image datasets, and applying the one or more mineral maps as a mask to the one or more image datasets to identify and label regions associated with minerals within the sample.
  • the one or more image datasets are generated using different imaging modalities of the multimodality imaging system.
  • feature vectors are extracted from the labeled regions via feature generators and machine learning training algorithms are executed upon the extracted feature vectors to learn behavior information of the feature vectors. This behavior information learned from the machine learning training algorithms is then applied to the one or more image dataset to construct a segmented image representation of the sample that classifies the minerals within the sample.
  • labeled images are converted to a set of feature vectors via feature generators.
  • One such feature generator is a Gabor filter that is used for texture and edge extraction. Gabor filters are primarily characterized by three parameters: wavelength, angle, and bandwidth. Multiple Gabor feature generators can be built by varying these parameters.
  • the application of feature generators on the labeled images result in images where each pixel is represented by a feature vector that is an array of points in vector space. Minerals with unique texture in the image datasets will cause unique feature vectors to be generated that represent corresponding minerals, in one example.
  • machine learning training algorithms are then executed upon the extracted feature vectors to associate certain values of feature vector combinations of features with the corresponding mineral phase.
  • One such machine learning training algorithm is Random Forest which is an ensemble learning method that constructs multiple decision trees during training. Each decision tree is constructed by using a random subset of training data. The decision tree information from the labeled images is then applied to the one or more unlabeled image datasets to construct a segmented image representation of the sample that classifies the minerals within the sample.
  • Another exemplary machine learning training algorithm is Multilayer
  • MLP Perceptron
  • the mineral maps are created using energy dispersive x-ray spectroscopy to identify the minerals within the sample.
  • generating the one or more image datasets of the sample can comprise generating one or more x-ray volume datasets of the sample as the one or more image datasets. These can include one or more x-ray volume datasets of the sample using different x-ray energies.
  • the generation of the one or more image datasets of the sample can comprise generating one or more FIB-SEM volume datasets of the sample as one or more image datasets using a focused ion beam scanning electron microscope (FIB-SEM) imaging and analysis system of the multi-modality imaging system. Backscattered electron and/or secondary electron modes of the FIB-SEM imaging and analysis system can be used.
  • FIB-SEM focused ion beam scanning electron microscope
  • At least one of the image datasets is a FIB-SEM volume dataset and the segmented image representation of the sample is a segmented FIB-SEM volume dataset.
  • the at least one of one or more image datasets of the sample includes one or more FIB-SEM datasets of the sample as one or more image datasets using a focused ion beam scanning electron microscope (FIB-SEM) imaging and analysis system of the multimodality imaging system.
  • FIB-SEM focused ion beam scanning electron microscope
  • the invention features a multimodality imaging system for segmenting minerals of a sample.
  • This system comprises an imaging system generating one or more image datasets of the sample and an energy dispersive x-ray spectroscopy system creating one or more mineral maps that identify the minerals within the sample.
  • a computer system applies the one or more mineral maps as a mask to the one or more image datasets to identify and label regions associated with the minerals within the sample.
  • a feature extraction engine is utilized that extracts feature vectors from the labeled regions.
  • a machine learning engine is also utilized that executes machine learning training algorithms upon the extracted feature vectors to learn behavior information of the feature vectors. Finally, the behavior information learned from the machine learning training algorithms are applied to the one or more image datasets to construct a segmented image representation of the sample that classifies the minerals within the sample.
  • Exemplary applications for the invention include the analysis of samples from materials science and geosciences.
  • the present invention can be used to construct segmented image representations of samples such as Solid Oxide Fuel Cells (SOFC) and meteorites, in examples.
  • SOFC Solid Oxide Fuel Cells
  • the segmented image representations classify phases and minerals within the samples with improved image resolution and with improved operational efficiency as compared to current systems and methods.
  • Fig. 1 is a system block diagram that shows the major components of a multimodality imaging system for determining mineralogy of a sample
  • Fig. 2 is a schematic block diagram that provides a high-level description of a method for constructing segmented image representations of a sample according to concepts of the present invention
  • Fig. 3A-3C are flow charts that describe preferred embodiments of the invention, where each of the embodiments describes a method executed by a computer system to construct segmented image representations of the sample, with Fig. 3 A describing a method for generating a segmented x-ray volume dataset of the sample, and with Figs. 3B and 3C describing methods for generating a segmented FIB-SEM volume dataset and a segmented 2D high pixel density image of the sample, respectively; [ 0025 ] Fig.
  • FIG. 4 is a schematic block diagram that shows relationships between different collaborative processes/applications executing on a computer system of the multimodality imaging system, where the processes include a feature extraction engine and a machine learning engine that operate upon the image datasets of the sample in the embodiments of Fig. 3A-3C to construct associated segmented image representations of the sample;
  • FIG. 5 shows images of different exemplary Gabor filter feature generators utilized by the feature extraction engine
  • Fig. 6 shows three different image dataset snippets for samples having different mineral textures, and power versus frequency plots of associated feature vectors created for each of the samples;
  • Fig. 7 A shows an exemplary scatter plot of feature vectors corresponding to two different classes of minerals within a sample
  • Fig. 7B and 7C show the same exemplary scatter plot of feature vectors as shown in Fig. 7A, with Fig. 7B additionally showing the results of applying a low- dimensional Multilayer Perceptron (MLP) machine learning training algorithm to the plot of Fig. 7 A and with Fig. 7C additionally showing the results of applying a high- dimensional Multilayer Perceptron (MLP) machine learning training algorithm to the plot of Fig. 7A for constructing segmented mineralogy.
  • MLP Multilayer Perceptron
  • the term “and/or” includes any and all combinations of one or more of the associated listed items. Further, the singular forms and the articles “a”, “an” and “the” are intended to include the plural forms as well, unless expressly stated otherwise. It will be further understood that the terms: includes, comprises, including and/or comprising, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements,
  • Fig. 1 shows an exemplary multi-modality imaging system 100.
  • the system 100 includes a datastore 90, a computer system 124, and a multi-modality sample imaging and analysis system 120.
  • the system 100 is utilized to classify (e.g. segment and label) regions of minerals within samples 114 and characterize microstructures within the samples 114.
  • the multi-modality sample imaging and analysis system 120 includes subsystems associated with different imaging modalities. These include a multi energy x- ray imaging and analysis system 110, a focused ion beam scanning electron microscope (FIB-SEM) imaging and analysis system 118, and an Energy Dispersive X-ray
  • EDX Spectroscopy
  • the computer system 124 includes one or more hardware-based central processing units (CPU) 102, one or more graphical processing units (GPUs) 137 and physical memory 133.
  • the computer system 124 also includes an operating system 104 that executes on the CPU(s) 102 and can optionally communicate with the GPU 137.
  • One or more applications or processes execute on the operating system 104 within one or more contexts as determined by the operating system 104.
  • the applications include a machine learning engine 172, a feature extraction engine 170, user interface applications 106 and an alignment and registration engine 138.
  • the applications are proprietary and/or standardized graphics applications that present application programming interfaces (APIs).
  • APIs presented by the applications enable cooperative communications and processing between the operating system 104, CPU 102, and GPU 137 via memory 133 and communications buses, in examples.
  • the computer system 124 also includes machine learning training algorithms 196 and feature generators 190.
  • the feature extraction engine 170 uses the feature generators 190 to extract feature vectors 192 from image datasets.
  • the machine learning engine 172 applies the machine leaning training algorithms 196 to the extracted feature vectors 192.
  • Operators utilize the user interface applications 136 to interact with the major components of the multi-modality imaging system 100 such as the computer system 124, the multi-modality sample imaging and analysis system 120, and the datastore 90.
  • a display device 60 connected to the computer system 124 allows the operator to execute tasks upon the components and display results.
  • the x-ray imaging and analysis system 110 uses x-rays and possibly x-rays at different energies to generate one or more x-ray volume datasets of the sample 114 as the image datasets.
  • the imaging typically operate across the 3D volume of the sample 114.
  • the imaging includes low energy (LE) dataset that typically uses x-rays below 70 keV, and high energy (HE) dataset that typically uses x-rays above 100 keV, in examples.
  • the LE datasets create LE x-ray volume datasets 130 and HE datasets create HE x-ray volume datasets 132.
  • the alignment and registration engine 138 of the computer system 124 then aligns and registers the LE and HE volume datasets 130/132 with each other to create a multi energy x-ray volume dataset 134 and saves it to the datastore 90.
  • the FIB-SEM imaging and analysis system 118 enables creation of Secondary Electron (SE) and/or Back Scattered Electron (BSE) image datasets of the sample 114.
  • SE Secondary Electron
  • BSE Back Scattered Electron
  • an operator can use the FIB-SEM imaging and analysis system 118 to scan across the entirety of the sample 114 to create BSE and SE FIB-SEM volume datasets 140/142 of the sample 114.
  • the alignment and registration engine 138 aligns and registers the BSE and SE FIB-SEM volume datasets 140/142 with each other to create a multi modal FIB-SEM volume dataset 144 and saves it to the datastore 90.
  • an operator can use the FIB-SEM imaging and analysis system 118 to create BSE and SE 2D high pixel density images 150/152 of the surface of the sample 114 as the image datasets.
  • the BSE and SE 2D high pixel density images 150/152 are scanned over a much larger region at high resolution.
  • the EDX system 116 also typically operates at slower rates compared to the BSE and SE imaging speeds of the FIB-SEM imaging and analysis system 118. For these reasons, the EDX mineral maps 80 are created by limiting the scan to a specific area or cross-section of the sample 114.
  • the alignment and registration engine 138 aligns and registers the BSE and SE 2D high pixel density images 150/152 with each other to create a multi modal 2D high pixel density image 154 and saves it to the datastore 90.
  • the system 100 uses the EDX system 116 of the Multi-modality sample imaging and analysis system 120 to create one or more EDX mineral maps 80 of the sample 114.
  • EDX mineral maps 80 are 2D images of selected slices of the sample 114.
  • the EDX mineral maps 80 are generated by utilizing the characteristic spectra of elements within the sample 114 and identifying mineralogy at every pixel for the selected slice.
  • the EDX mineral maps 80 establish a "ground truth" of the mineralogy of the analyzed surface within the sample 114 volume.
  • the slices of the sample 114 selected for creation of the EDX mineral maps 80 include one or more 2D slices or surfaces of the sample 114 that are at least also covered by the image datasets of the sample 114 associated with each of the embodiments/imaging modalities.
  • the EDX mineral maps 80 are preferably created from one or more 2D slices or sample surfaces covered by the multi energy x-ray volume dataset 134.
  • the EDX mineral maps 80 are created from one or more 2D slices or sample surfaces covered by the multi modal FIB-SEM volume dataset 144.
  • the at least one slice selected for creation of the EDX mineral maps 80 is from the surface of the sample 114.
  • the EDX mineral maps 80 are created from one or more small regions within the sample 114 that are also covered by the multi modal 2D high pixel density image 154.
  • the EDX mineral maps 80 are collected/generated from a surface slice of the sample 114.
  • the EDX mineral maps 80 can also be created from slices or surfaces associated with selected regions of interest exposed from within the sample 1 14.
  • BSE images taken of the sample 114 via the FIB-SEM imaging and analysis system 118 can assist in identifying regions of interest within the sample 114 from which the EDX mineral maps 80 can be created.
  • the alignment and registration engine 138 aligns and registers the EDX mineral maps 80 with the multi energy x-ray volume dataset 134, the multi modal FIB-SEM volume dataset 144, and the multi modal 2D high pixel density image 154 and saves these aligned versions to the datastore 90.
  • the machine learning engine 172 accepts the aligned and correlated datasets 132/130, 142/140, 152/150 along with EDX mineral maps 80. In conjunction with the feature extraction engine 170, the machine learning engine 172 applies the information associated with the corresponding pixels for each mineral identified by the EDX mineral maps 80 to each of the multi energy x-ray volume dataset 134, the multi modal FIB-SEM volume dataset 144, and the multi modal 2D high pixel density image 154 to create corresponding segmented image representations of the sample 114 and saves them to the datastore 90.
  • the corresponding segmented image representations are segmented x-ray volume datasets 136, segmented FIB-SEM volume datasets 146, and segmented 2D high pixel density images 156, respectively.
  • the disclosed embodiments of the mineralogy segmentation methods preferably accept as input either single or multi energy x-ray volume datasets 134, single or multi modal FIB-SEM volume datasets 144, or single or multi modal 2D high pixel density images 154, it can be appreciated that the segmentation methods also accepts individual datasets or multiple datasets of the same modality.
  • the datasets are then processed into their associated segmented image representations.
  • a single LE or HE x-ray volume dataset 130/132 can be processed to create a segmented x-ray volume dataset 136.
  • a single BSE 2D high pixel density image 150 can be processed to create a segmented 2D high pixel density image 156.
  • a BSE FIB- SEM volume dataset 142 of the sample 114 can be generated, and then processed to create a segmented FIB-SEM volume dataset 146.
  • Fig. 2 provides a high-level description of a method for constructing segmented image representations of a sample 114.
  • Image datasets 134/144/154 of a sample 114 are created according to one or more of the imaging modalities for the sample 114.
  • an EDX mineral map 80 (ground truth) is aligned with a surface of the image dataset.
  • the EDX mineral map 80 is applied to the image datasets 134/144/154 to create associated segmented image datasets 136/146/156 for mineralogy.
  • Fig. 3A-3C describe the preferred mineralogy segmentation methods executed by the computer system 124 for different imaging modalities.
  • Fig. 3A shows a method of the computer system 124 for generating a segmented x-ray volume dataset 136 of the sample 114 for an x-ray modality.
  • step 304 the computer system 124 instructs the multi energy x-ray imaging and analysis system 110 to execute x-ray CT imaging of the sample 114 over different energy ranges (one low energy (LE) and one high energy (HE) ) to create x-ray volume datasets at each of the energy ranges.
  • LE and HE x-ray volume datasets 130/132 are created as a result of this step.
  • step 306 the alignment and registration engine 138 generates an aligned and registered multi energy x-ray volume dataset 134 from the HE and LE x-ray volume datasets 132/130.
  • the computer system 124 instructs the EDX system 116 to create one or more EDX mineral maps 80 of the sample 114.
  • the alignment and registration engine 138 aligns and registers the multi energy x-ray volume dataset 134 with the EDX mineral map(s) 80 in step 310.
  • step 312 the computer system 124 applies the EDX mineral map(s) 80 as a mask to the multi energy x-ray volume dataset 134 to identify and label regions of pixels/voxels associated according to their mineralogy.
  • step 314 the computer system 124 determines if all regions of pixels associated with minerals in the multi energy x-ray volume dataset 134 are represented by corresponding labels in the current EDX mineral map(s) 80. If this statement is false, control passes back to the beginning of step 308 to create additional EDX mineral maps 80. Otherwise, control passes to step 316.
  • the feature extraction engine 170 extracts feature vectors 192 from labeled regions within the multi energy x-ray volume dataset 134 via feature generators 190 such as Gabor, Gaussian, and Histogram Oriented Gradient filters, in examples.
  • feature generators 190 such as Gabor, Gaussian, and Histogram Oriented Gradient filters, in examples.
  • the machine learning engine 172 executes machine learning training algorithms 196 upon the extracted feature vectors 192 to learn behavior of the feature vectors 192.
  • An exemplary machine learning training algorithm 196 is Random Forest 196- 1.
  • the machine learning engine 172 preferably applies the behavior information learned from the training algorithms 196 across the entirety of the multi energy x-ray volume dataset 134 and beyond the EDX mineral map(s) 80 to construct a segmented and labeled x-ray volume dataset 136 of the sample 114.
  • the computer system 124 verifies the segmentation by comparing selected slices of the segmented x-ray volume dataset 136 against one or more ground truth EDX mineral maps 80.
  • Fig. 3B shows a method for the computer system 124 generating a segmented FIB-SEM volume dataset 146 of the sample 114 for a FIB-SEM imaging modality.
  • step 342 the computer system 124 instructs the FIB-SEM imaging and analysis system 118 to execute a FIB-SEM Secondary Electron (SE) scan and/or a FIB- SEM Back Scattered Electron (BSE) scan of the of sample to create SE and/or BSE FIB- SEM volume datasets 142/140 of the sample 114, respectively.
  • SE FIB-SEM Secondary Electron
  • BSE FIB- SEM Back Scattered Electron
  • step 344 the alignment and registration engine 138 generates an aligned and registered multi modal FIB-SEM volume dataset 144 from the SE and/or BSE FIB-SEM volume datasets 142/140.
  • the computer system 124 instructs the EDX system 116 to create one or more EDX mineral maps 80 for regions/slices covered by the FIB-SEM volume dataset 144 to establish the "ground truth" of the sample 114.
  • the alignment and registration engine 138 aligns and registers the multi modal FIB-SEM volume dataset 144 with the EDX mineral map(s) in step 348.
  • step 350 the computer system 124 applies the EDX mineral map(s) 80 as a mask to the multi modal FIB-SEM volume dataset 144 to identify and label regions of pixels associated with minerals.
  • step 352 the computer system determines if all regions of pixels/voxels associated with minerals in the multi modal FIB-SEM volume dataset 144 are represented by corresponding labels in the current EDX mineral map(s) 80. If this statement is false, control passes back to the beginning of step 346 to create additional EDX mineral maps 80. Otherwise, control passes to step 354.
  • step 354 the feature extraction engine 170 extracts feature vectors 192 from labeled regions within the multi modal FIB-SEM volume dataset 144 via feature generators 190.
  • the machine learning engine 172 executes machine learning training algorithms 196 upon the extracted feature vectors 192 to learn behavior of the feature vectors 192.
  • step 358 the machine learning engine 172 preferably applies the behavior information learned from the training algorithms 196 across the entirety of the multi modal FIB-SEM volume dataset 144 to construct a segmented FIB-SEM volume dataset 146 of the sample 114.
  • step 360 the computer system 124 verifies the segmentation by comparing selected slices of the segmented FIB-SEM volume dataset 146 against one or more EDX mineral map(s) 80.
  • Fig. 3C shows a method for the computer system 124 to generate a segmented 2D high pixel density image 156 of the sample 114 for a FIB-SEM modality.
  • step 372 the computer system 124 instructs the FIB-SEM imaging and analysis system 118 to execute a FIB-SEM Secondary Electron (SE) scan and/or a FIB- SEM Back Scattered Electron (BSE) scan of the of sample to create SE and BSE 2D images 152/150 of the sample 114, respectively.
  • SE FIB-SEM Secondary Electron
  • BSE FIB- SEM Back Scattered Electron
  • step 374 the alignment and registration engine 138 generates an aligned and registered multi modal 2D image 154 from the SE and BSE 2D images 152/150.
  • step 376 the computer system 124 instructs the EDX system 116 to create one or more EDX mineral maps 80 from regions of the sample that are covered by the multi modal 2D image 154 to establish the "ground truth" of the sample 114.
  • registration engine 138 aligns and registers the multi modal 2D mage 154 with the EDX mineral map(s) in step 378.
  • step 380 the computer system 124 applies the EDX mineral map(s) 80 as a mask to the multi modal 2D high pixel density image 154 to identify and label regions of pixels associated with minerals.
  • step 382 the computer system determines if all regions of pixels/voxels associated with minerals in the multi modal 2D high pixel density image 154 are represented by corresponding labels in the current EDX mineral map(s) 80. If this statement is false, control passes back to the beginning of step 376 to create additional EDX mineral maps 80. Otherwise, control passes to step 384.
  • the feature extraction engine 170 extracts feature vectors 192 from labeled regions within the multi modal 2D high pixel density image 154 via feature generators 190.
  • the machine learning engine 172 executes machine learning training algorithms 196 upon the extracted feature vectors 192 to learn behavior of the feature vectors 192.
  • the machine learning engine 172 preferably applies the behavior information learned from the training algorithms 196 across the entirety of the multi modal 2D high pixel density image 154 to create a segmented 2D high pixel density image 156 of the sample 114.
  • the computer system 124 verifies the segmentation by comparing selected areas of the segmented 2D high pixel density image 156 against one or more EDX mineral map(s) 80.
  • Fig. 4 shows how the feature extraction engine 170 and the machine learning engine 172 of the computer system 124 operate in a collaborative fashion to construct the segmented image representations 136/146/156 of the sample 114.
  • the feature extraction engine 170 loads the feature generators 190 from the computer system 124.
  • the feature generators 190 include filters such as mean 190-1, variance 190-2, Sobel 190-3, Gabor 190-4, Histogram of Oriented Gradients (HOG) 190-5, Laplacian 190-6, and Hessian 190- 7 filters, in examples. These filters are typically digital filters.
  • the Gabor filter 190-4 is applied to the multi energy x-ray volume dataset 134, the multi modal FIB- SEM volume dataset 144, and the multi modal 2D high pixel density image 154 with varying angle, bandwidth, and frequency parameter values.
  • each pixel in the image dataset is assigned an array of vector values for each feature generator 190 (e.g. filter) applied. These vector values are also known as feature vectors 192.
  • the feature extraction engine 170 then provides the feature vectors 192 for processing by other applications that execute within the computer system 124.
  • the machine learning engine 172 loads the classifiers/machine learning training algorithms 196 from the computer system 124.
  • Exemplary machine learning training algorithms include Random Forest 196-1, Multilayer Perception neural network (MLP) 196-2, and Support Vector Machine (SVM) 196-3.
  • the machine learning engine 172 accepts the feature vectors 192 passed by the feature extraction engine 170.
  • the machine learning engine 172 executes the machine learning training algorithms 196 upon the feature vectors 192, and applies the information learned from this process to the entirety of the image datasets that were aligned with the EDX mineral maps 80 (e.g. the multi energy x-ray volume dataset 134, the multi modal FIB-SEM volume dataset 144, and the multi modal 2D high pixel density image 154).
  • the constructed segmented image representations that result are the segmented x-ray energy volume dataset 136, the segmented FIB-SEM volume dataset 146, and the segmented 2D high pixel density image 156, respectively.
  • FIG. 5 shows images representing different exemplary 2D Gabor feature generators 190-4-1 through 190-4-4, each having different (wavelength, angle, bandwidth) parameter values (64,0,4), (64,45,4), (64,90,4) and (16,90,4), respectfully.
  • a large set of feature generators 190 such as Gabor filters 190-4 are selected over a range of different wavelength, angle and bandwidth settings.
  • Gabor filters 190-4 are complex kernels and they generate a set of filtered images.
  • a Gabor filter 190-4 kernel has 18 frequencies and eleven angles, which creates a total of 198 images.
  • feature vectors 192 are created.
  • the created feature vectors 192 are arrays of Gabor values at each pixel location in the image dataset 134/144/154.
  • FIG. 6 shows power versus frequency plots of Gabor feature vectors 192- 1 through 192-3 created for three 64x64 images snippets of mineral samples 114-1 through 114-3 having different textures.
  • Each of the plots was generated by applying Gabor feature generators 190-4 to an image dataset 134/144/154 of samples 114-1 through 114-3.
  • the same Gabor feature generator 190-4 bandwidth setting was selected by the feature extraction engine 170 when applying the Gabor feature generator 190-4 to the image datasets 134/144/154.
  • Gabor feature generators 190-4 are windowed Fourier transforms, the vertical axis of each plot in the current example is power, and the x axis is a combination of angle and frequency. Moreover, because the feature vectors 192 were created by applying Gabor filter 190-4 feature generators, the feature vectors 192 are also referred to as Gabor feature vectors 192-1 through 192-3.
  • the machine learning engine 172 "learns" the texture of each sample 114 by executing machine learning training algorithms 196 upon the generated feature vectors 192-1 through 192-3.
  • the differences in the power versus frequency plots of feature vectors 192-1 through 192-3 correspond to the different textures of the samples 114-1 through 114-3.
  • the large spikes in the plots of the feature vectors 192-1 through 192-3 are narrow frequencies and angles where the power is high.
  • the broad flat area is the background noise.
  • Fig. 7 A shows an exemplary scatter plot of feature vectors 192-9 and 192-10 corresponding to two different classes of minerals within a sample 114.
  • the sample 114 includes two types or classes of minerals.
  • feature vectors 192-9 and 192-10 are associated with two distinct features of the sample 114.
  • Feature vectors 192-9 associated with a first class of minerals or "Class 1 " are represented as circles in the scatter plot, and feature vectors 192-10 associated with a second class of minerals or "Class 2" are represented as crosses ( + ) in the scatter plot.
  • Fig. 7B shows the results of applying a low-dimensional Multilayer Perceptron (MLP) machine learning training algorithm 196-2 to the plot of Fig. 7 A for constructing segmented image representations of the sample 136/146/156.
  • MLP Multilayer Perceptron
  • two MLP hidden units, or hidden layer neurons, are selected to accomplish the low-dimensional MLP 196-2 processing.
  • a decision surface or boundary 200-1 is defined among the feature vectors for constructing a segmented image representation of the sample 114.
  • feature vectors 192 that fall to the left of the decision boundary 200-1 are statistically determined to belong to the "Class 1 " class of minerals.
  • Feature vectors 192 located further away to the left of the decision boundary 200-1 have a higher probability of belonging to Class 1 than feature vectors located nearer the decision boundary 200-1.
  • features having feature vectors 192 that fall to the right of the decision boundary 200-1 are determined to belong to the "Class 2" class of minerals.
  • Feature vectors 192 located further away to the right of the decision boundary 200-1 have a higher probability of belonging to Class 2 than feature vectors located nearer the decision boundary 200-1.
  • a perfect image segmentation algorithm would optimally generate a decision boundary 200-1 such that all circle data points for feature vectors 192-9 would be located to right of the decision boundary 200-1, and all cross data points for feature vectors 192-10 would be located to the left of the decision boundary 200-1, in one example.
  • Features having feature vectors that fall on the decision boundary 200-1 belong to both classes with 50% probability.
  • Fig. 7C shows the results of applying a high-dimensional Multilayer Perceptron (MLP) machine learning training algorithm to the plot of Fig. 7 A.
  • MLP Multilayer Perceptron
  • six MLP hidden units are selected to accomplish the high-dimensional MLP 196-2 processing.
  • a decision surface or boundary 200-2 is defined among the feature vectors 192-9 and 192-10 for constructing a segmented image representation of the sample 114.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Remote Sensing (AREA)
  • Immunology (AREA)
  • Food Science & Technology (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • Geology (AREA)
  • Medicinal Chemistry (AREA)
  • Pathology (AREA)
  • Quality & Reliability (AREA)
  • General Life Sciences & Earth Sciences (AREA)
  • Environmental & Geological Engineering (AREA)
  • Analysing Materials By The Use Of Radiation (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un système et un procédé d'imagerie à multimodalité pour la segmentation de minéralogie. Des ensembles de données d'images de l'échantillon sont générés pour une ou plusieurs modalités, dont des modalités de microscope à rayons X et de microscope électronique à balayage de faisceau d'ions concentrés (MEB à FIB). Des cartes minérales sont alors créées par spectroscopie aux rayons X par dispersion d'énergie (EDX) à partir d'au moins une partie de l'échantillon couvert par les ensembles de données d'images. Les cartes minérales EDX sont appliquées en tant que masque aux ensembles de données d'images pour identifier et étiqueter des zones de minéraux dans l'échantillon. Des vecteurs d'attributs sont alors extraits des zones étiquetées par des générateurs d'attributs en tant que filtres de Gabor. Enfin, un entraînement par apprentissage par machine et des algorithmes de classement de type forêts d'arbres décisionnel sont appliqués aux vecteurs d'attributs extraits pour construire une représentation d'image segmentée de l'échantillon qui classe les minéraux dans l'échantillon.
EP16705331.3A 2016-01-11 2016-01-11 Système et procédé de segmentation de minéralogie à multimodalité Ceased EP3403238A1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2016/012864 WO2017123196A1 (fr) 2016-01-11 2016-01-11 Système et procédé de segmentation de minéralogie à multimodalité

Publications (1)

Publication Number Publication Date
EP3403238A1 true EP3403238A1 (fr) 2018-11-21

Family

ID=55398394

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16705331.3A Ceased EP3403238A1 (fr) 2016-01-11 2016-01-11 Système et procédé de segmentation de minéralogie à multimodalité

Country Status (4)

Country Link
EP (1) EP3403238A1 (fr)
JP (1) JP6704052B2 (fr)
CN (1) CN108431870A (fr)
WO (1) WO2017123196A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112529112A (zh) * 2020-12-29 2021-03-19 中国地质科学院地质力学研究所 一种矿物识别的方法和装置

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9905394B1 (en) * 2017-02-16 2018-02-27 Carl Zeiss Microscopy Gmbh Method for analyzing an object and a charged particle beam device for carrying out this method
CN109792651B (zh) * 2017-03-21 2021-02-05 华为技术有限公司 一种通信方法及设备
JP7052643B2 (ja) * 2018-08-29 2022-04-12 住友金属鉱山株式会社 試料分析方法
JP7111084B2 (ja) * 2019-03-25 2022-08-02 日新イオン機器株式会社 イオンビーム照射装置及びイオンビーム照射装置用プログラム
WO2021019361A1 (fr) * 2019-08-01 2021-02-04 Khalifa University of Science and Technology Procédé et système d'estimation rapide et précise de propriétés pétrophysiques d'échantillons de roche
CN111126281B (zh) * 2019-12-25 2021-03-09 重庆紫光华山智安科技有限公司 一种人群密度估计方法、装置、存储介质及电子设备
JP7417097B2 (ja) 2020-04-08 2024-01-18 日本製鉄株式会社 焼結鉱の組織構成学習装置、組織構成学習方法及び組織構成学習プログラム
JP7417095B2 (ja) 2020-04-08 2024-01-18 日本製鉄株式会社 焼結鉱の組織構成学習装置、組織構成学習方法及び組織構成学習プログラム
JP7499194B2 (ja) 2021-01-27 2024-06-13 大成建設株式会社 分析方法および分析装置
CN113989229A (zh) * 2021-10-28 2022-01-28 杭州图谱光电科技有限公司 一种基于随机森林和卷积神经网络的细胞计数方法
CN117783168B (zh) * 2024-02-05 2024-06-28 中国科学院南京地质古生物研究所 一种化石样品内部结构跨尺度综合分析方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2223842B (en) * 1988-09-06 1993-02-03 Shell Int Research Automated mineral identification and rock characterization process
CN101710424B (zh) * 2009-12-22 2012-04-25 中国矿业大学(北京) 矿石图像分割方法
WO2013055876A1 (fr) * 2011-10-14 2013-04-18 Ingrain, Inc. Procédé et système à image double pour générer une image multidimensionnelle d'un échantillon
WO2013148632A1 (fr) * 2012-03-29 2013-10-03 Ingrain, Inc. Procédé et système pour estimer des propriétés de milieux poreux, tels que des roches denses ou à pores fins
CA2871781C (fr) * 2012-05-18 2017-02-07 Ingrain, Inc. Procede et systeme d'estimation de proprietes de roches a partir d'echantillons de roche en utilisant une imagerie numerique de la physique des roches
US9128584B2 (en) * 2013-02-15 2015-09-08 Carl Zeiss X-ray Microscopy, Inc. Multi energy X-ray microscope data acquisition and image reconstruction system and method
CN103714536B (zh) * 2013-12-17 2017-06-16 深圳先进技术研究院 基于稀疏表示的多模态磁共振图像的分割方法及装置
US9183656B2 (en) * 2014-03-11 2015-11-10 Fei Company Blend modes for mineralogy images
CN104778452B (zh) * 2015-04-02 2018-04-17 浙江大学 一种基于机器学习的可行区域检测方法
CN104729971B (zh) * 2015-04-08 2017-02-22 中国石油大学(华东) 一种岩石纳米ct的孔隙标定方法
CN105006018B (zh) * 2015-06-30 2017-11-14 四川大学 三维ct岩心图像超分辨率重建方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112529112A (zh) * 2020-12-29 2021-03-19 中国地质科学院地质力学研究所 一种矿物识别的方法和装置

Also Published As

Publication number Publication date
JP6704052B2 (ja) 2020-06-03
JP2019504452A (ja) 2019-02-14
WO2017123196A1 (fr) 2017-07-20
CN108431870A (zh) 2018-08-21

Similar Documents

Publication Publication Date Title
US10107769B2 (en) Multimodality mineralogy segmentation system and method
JP6704052B2 (ja) マルチモダリティ鉱物質セグメント化システムおよび方法
Andrew A quantified study of segmentation techniques on synthetic geological XRM and FIB-SEM images
Basavanhally et al. Multi-field-of-view framework for distinguishing tumor grade in ER+ breast cancer from entire histopathology slides
EP2391887B1 (fr) Évaluation des risques à base d'images - indicateur pronostique de survie et résultats d'une histopathologie numérique
US20160307312A1 (en) Method For Determining Fabric And Upscaled Properties Of Geological Sample
Chekkoury et al. Automated malignancy detection in breast histopathological images
US10223782B2 (en) Digital rock physics-based trend determination and usage for upscaling
JP2023539830A (ja) 空間分解したデータセットからクロスモーダル特徴を特定するための方法
WO2008157843A1 (fr) Système et procédé pour la détection, la caractérisation, la visualisation et la classification d'objets dans des données d'image
Kazak et al. Machine-learning-assisted segmentation of focused ion beam-scanning electron microscopy images with artifacts for improved void-space characterization of tight reservoir rocks
Ghongade et al. Detection and classification of breast cancer from digital mammograms using RF and RF-ELM algorithm
Barisin et al. Methods for segmenting cracks in 3d images of concrete: A comparison based on semi-synthetic images
Helwan et al. Shape and texture features for the identification of breast cancer
Sharma et al. A comparative study of cell nuclei attributed relational graphs for knowledge description and categorization in histopathological gastric cancer whole slide images
US20230144724A1 (en) Method for Finding Image Regions that Significantly Influence Classification in a Tool for Pathology Classification in a Medical Image
WO2017039475A1 (fr) Procédé mis en œuvre par ordinateur et système pour créer un modèle minéral tridimensionnel d'un échantillon d'un milieu hétérogène
Rathore et al. A novel approach for ensemble clustering of colon biopsy images
US20220397697A1 (en) Grain-based minerology segmentation system and method
Krishnammal et al. Automated brain image classification using neural network approach and abnormality analysis
Ibeh et al. PLATYMATCH–A particle-matching algorithm for the analysis of platy particle kinematics using X-ray Computed Tomography
Tan et al. A review of image processing and quantification analysis for solid oxide fuel cell
Roullier et al. Mitosis extraction in breast-cancer histopathological whole slide images
Varfolomeev et al. 3D multiclass digital core models via microCT, SEM-EDS and deep learning
Igor et al. Segmentation of 3D image of a rock sample supervised by 2D mineralogical image

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20180806

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20200428

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

REG Reference to a national code

Ref country code: DE

Ref legal event code: R003

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20231005