WO2020033566A1 - Neural networks for volumetric segmentation and parcellated surface representations - Google Patents

Abstract

A method for quantification of brain regions comprises obtaining an MRI image of a brain, the MRI image comprising multiple slabs, each slab comprising multiple slices, each slice comprising a gray-scale image of the brain taken at a different cross-section. The method further comprises performing a series of convolutions, pooling, and upsampling on each slab to obtain a segmented output, the segmented output comprising groups of segmented slabs segmented according to a class. The method further comprises combining the groups of segmented slabs together to obtain a segmented MRI image, the segmented MRI image comprising a voxel matrix. The method further comprises calculating volumes, surface areas, and thicknesses of the brain regions based on the voxel matrix.

Description

NEURAL NETWORKS FOR VOLUMETRIC SEGMENTATION AND

PARCELLATED SURFACE REPRESENTATIONS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of and priority under 35 U.S.C. §119 from U.S. Provisional Patent Application Serial No. 62/715,730 entitled“NEURAL NETWORKS FOR VOLUMETRIC SEGMENTATION AND PARCELLATED SURFACE

REPRESENTATIONS,” filed on August 7, 2018, U.S. Provisional Patent Application Serial No. 62/731,005 entitled“NEURAL NETWORKS FOR VOLUMETRIC SEGMENTATION AND PARCELLATED SURFACE REPRESENTATIONS,” filed on September 13, 2018, U.S. Provisional Patent Application Serial No. 62/757,121 entitled“NEURAL NETWORKS FOR VOLUMETRIC SEGMENTATION AND PARCELLATED SURFACE

REPRESENTATIONS,” filed on November 7, 2018, and U.S. Provisional Patent Application Serial No. 62/824,218 entitled “NEURAL NETWORKS FOR VOLUMETRIC SEGMENTATION AND PARCELLATED SURFACE REPRESENTATIONS,” filed on March 26, 2019, the disclosures of all of which are hereby incorporated by reference in their entirety for all purposes.

FIELD

[0002] The embodiments disclosed herein are generally directed towards systems and methods for predicting and managing risk of brain function, disorders, and diseases for individuals. More specifically, there is a need for systems and methods for quantifying aspects of the morphological features in various regions of the brain to develop disease risk models, identify biomarkers for disease progression, cognitive decline, impairment, etc.

BACKGROUND

[0003] Conventional imaging software packages for volumetric measurements of regions of the brain are widely used by the neuro research community for disease detection, prediction, basic neuroscience research, and understanding of neurological diseases. However, atlas and statistical based approaches require iterative steps, which can be time intensive and expensive to process at scale.

[0004] Recently, deep learning-based techniques have shown superior performance both in accuracy and processing time and have outperformed traditional approaches for segmentation of images. Specifically, Fully Convolutional Networks (FCNs) for semantic segmentation have found success in many medical imaging applications. For applications where three- dimensional (3D) information is necessary for proper segmentation, recent work on Volumetric Fully Convolutional Neural Networks (V-NETs) has had a great deal of success. For example, lung nodules can only be segmented in 3D because they cannot be differentiated from vascularity in 2-dimensions. Similarly, structures of the brain may require 3D information to be properly segmented.

[0005] In addition to volume-based measurements, geometrical measurements from a surface representation including cortical thickness and surface areas have been shown to increase the predictive power for neurological disease. However, a U-NET or V-NET style architecture will not natively produce a surface representation, only the volumetric representation. Recently, geometric deep learning has emerged to address non-grid 3D images like point clouds and meshes.

[0006] As such, there is a need for neural network-based approaches for disease detection, prediction, basic neuroscience research, and understanding of neurological diseases. It can potentially reduce post-processing time from hours to minutes or even seconds, which reduces computational costs and is feasible for quick analysis and same day delivery of results. Also, it would not require any iterative steps and allows for further processing time optimization via GPUs and Spark, if needed.

SUMMARY

[0007] The present disclosure provides for systems and methods for predicting and managing risk of brain function, disorders, and diseases for individuals. According to one embodiment of the present disclosure, a method for quantification of brain regions is provided. The method comprises obtaining an MRI image of a brain, the MRI image comprising multiple slabs, each slab comprising multiple slices, each slice comprising a gray-scale image of the brain taken at a different cross-section. The method further comprises performing a series of convolutions, pooling, and upsampling on each slab to obtain a segmented output, the segmented output comprising groups of segmented slabs, segmented according to a class. The method further comprises combining the groups of segmented slabs together to obtain a segmented MRI image, the segmented MRI image comprising a voxel matrix. The method further comprises calculating volumes, surface areas, and thicknesses of the brain regions based on the voxel matrix.

[0008] According to one embodiment of the present disclosure, a system is provided for quantification of brain regions. The system comprises a magnetic resonance imaging device configured to obtain an MRI image of a brain, the MRI image comprising multiple slabs, each slab comprising multiple slices, each slice comprising a gray-scale image of the brain taken at a different cross-section. The system also comprises a computing device communicatively connected to the magnetic resonance imaging device. The computing device comprises a mask generator configured to perform a series of convolutions, pooling, and upsampling on each slab to obtain a segmented output. The segmented output comprises groups of segmented slabs segmented according to a class. The mask generator is configured to combine the groups of segmented slabs together to obtain a segmented MRI image. The segmented MRI image comprises a voxel matrix. The system also comprises a voxel calculator configured to calculate volumes, surface areas, and thicknesses of the brain regions based on the voxel matrix.

[0009] It is understood that other configurations of the subject technology will become readily apparent to those skilled in the art from the following detailed description, wherein various configurations of the subject technology are shown and described by way of illustration. As will be realized, the subject technology is capable of other and different configurations and its several details are capable of modification in various other respects, all without departing from the scope of the subject technology. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are included to provide further understanding and are incorporated in and constitute a part of this specification, illustrate aspects of the subject technology, and together with the description serve to explain the principles of the subject technology. In the drawings:

[0011] FIG. 1 illustrates volumetric and surface measurements derived from a Tl-weighted image, according to certain aspects of the disclosure.

[0012] FIGs. 2A-2C illustrate an exemplary system for training a neural network, according to certain aspects of the disclosure.

[0013] FIG. 3 illustrates an exemplary system for training a neural network, according to certain aspects of the disclosure.

[0014] FIG. 4 illustrates a voxel matrix showing axial, coronal, and sagittal views, according to certain aspects of the disclosure.

[0015] FIG. 5 illustrates a 2D slice of a voxel matrix M, according to certain aspects of the disclosure. [0016] FIG. 6 illustrates a voxel diagram for calculation of a total surface area, according to certain aspects of the disclosure.

[0017] FIG. 7 illustrates three layers utilized in calculating an average cortical thickness, according to certain aspects of the disclosure.

[0018] FIG. 8 illustrates overlaid masks for identifying brain lesions, according to certain aspects of the disclosure.

[0019] FIG. 9 illustrates an example flow diagram for quantification of brain regions, according to certain aspects of the disclosure.

[0020] FIG. 10 is a block diagram illustrating an example computer system with which aspects of the subject technology can be implemented.

[0021] In one or more implementations, not all of the depicted components in each figure may be required, and one or more implementations may include additional components not shown in a figure. Variations in the arrangement and type of the components may be made without departing from the scope of the subject disclosure. Additional components, different components, or fewer components may be utilized within the scope of the subject disclosure.

DETAILED DESCRIPTION

[0022] The detailed description set forth below is intended as a description of various implementations and is not intended to represent the only implementations in which the subject technology may be practiced. As those skilled in the art would realize, the described implementations may be modified in various different ways, all without departing from the scope of the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.

[0023] Systems and methods for enhancing the ability of clinicians to identify features of interest in an individual using MRI images and a deep learning network are disclosed herein. These procedures enhance a clinician’ s ability to quantify brain regions for improved accuracy and consistent diagnosis of diseases, including Alzheimer’s Disease, cognitive decline, and early and late mild cognitive impairment. The disclosed systems and methods also apply to other neurological applications including, but not limited to, brain aging, sleep, depression, anxiety, OCD, plasticity, fluid intelligence, memory, Autism, and ischemic stroke. The systems and methods described herein leverage the performance of either a V-Net or U-Net style architecture for medical imaging segmentation.

[0024] According to an aspect of the present disclosure, a method for quantification of brain regions comprises obtaining an MRI image of a brain, the MRI image comprising multiple slabs, each slab comprising multiple slices, each slice comprising a gray-scale (e.g., binary) image of the brain taken at a different cross-section. The method further comprises performing a series of convolutions, pooling, and upsampling on each slab to obtain a segmented output, the segmented output comprising groups of segmented slabs segmented according to a class. The method further comprises combining the groups of segmented slabs together to obtain a segmented MRI image, the segmented MRI image comprising a voxel matrix. The method further comprises calculating volumes, surface areas, and thicknesses of the brain regions based on the voxel matrix.

[0025] According to an aspect of the present disclosure, a system for quantification of brain regions includes a magnetic resonance imaging device configured to obtain an MRI image of a brain, the MRI image comprising multiple slabs, each slab comprising multiple slices, each slice comprising a gray-scale (e.g., binary) image of the brain taken at a different cross-section. The system further comprises a computing device communicatively connected to the magnetic resonance imaging device. The computing device comprises a mask generator configured to perform a series of convolutions, pooling, and upsampling on each slab to obtain a segmented output, the segmented output comprising groups of segmented slabs segmented according to a class. The mask generator is further configured to combine the groups of segmented slabs together to obtain a segmented MRI image, the segmented MRI image comprising a voxel matrix. The computing device further comprises a voxel calculator configured to calculate volumes, surface areas, and thicknesses of the brain regions based on the voxel matrix.

[0026] This specification describes various exemplary embodiments of systems and methods for predicting and managing risk of brain function, disorders, and diseases for individuals. The disclosure, however, is not limited to these exemplary embodiments and applications or to the manner in which the exemplary embodiments and applications operate or are described herein. Moreover, the figures may show simplified or partial views, and the dimensions of elements in the figures may be exaggerated or otherwise not in proportion. In addition, as the terms“on,” “attached to,”“connected to,”“coupled to,” or similar words are used herein, one element (e.g., a material, a layer, a substrate, etc.) can be“on,”“attached to,”“connected to,” or“coupled to” another element regardless of whether the one element is directly on, attached to, connected to, or coupled to the other element or there are one or more intervening elements between the one element and the other element. In addition, where reference is made to a list of elements (e.g., elements a, b, c), such reference is intended to include any one of the listed elements by itself, any combination of less than all of the listed elements, and/or a combination of all of the listed elements. Section divisions in the specification are for ease of review only and do not limit any combination of elements discussed.

[0027] Unless otherwise defined, scientific and technical terms used in connection with the present teachings described herein shall have the meanings that are commonly understood by those of ordinary skill in the art. Further, unless otherwise required by context, singular terms shall include pluralities and plural terms shall include the singular.

[0028] As used herein, the phrase “features of interest,” “pathological features,” or “metastatic features” can refer to tissue structures that have clinical significance, for example, benign tumors, metastatic cancers, damaged or diseased tissue, etc.

[0029] As used herein, the phrase “medical imaging techniques,” “medical imaging methods,” or“medical imaging systems” can denote techniques or processes for obtaining visual representations of the interior of an individual’s body for clinical analysis and medical intervention, as well as visual representation of the function of some organs or tissues. Within these visual representations, various imaging features can be identified and characterized to provide a structural basis for diagnosing and treating various types of diseases (e.g., dementia, cancer, cardiovascular disease, cerebrovascular disease, liver disease, etc.). Examples of medical imaging techniques can include, but are not limited to, x-ray radiography, magnetic resonance imaging, ultrasound, positron emission tomography (PET), computed tomography (CT), etc.

[0030] As used herein, Magnetic Resonance Imaging (MRI) denotes a radiology imaging technique that uses an MRI scanner (that produces magnet fields and radio waves) and a computing device to produce images of body structures. In various embodiments, the MRI scanner can be a“closed MRI” consisting of a giant circular magnet where the patient subject is placed on a moveable bed that is inserted into the magnetic tube, an“open MRI” consisting of two horizontal magnetic disks connected to a pillar between them where the patient subject sits or stands between the disks, and a“portable MRI” consisting of a hand portable scanner containing magnet(s) that are optimized to general ultra- low frequency magnetic fields coupled to highly sensitive superconducting quantum interface detectors (SQUID). MRI works by employing the MRI scanner magnet(s) to create a strong magnetic field that aligns the protons of nuclei of interest, typically ¹ H, which are then exposed to radiofrequency waves. This net magnetic moment created by spins of the nuclei of interest within tissues in the body produce a faint signal that are detected by receiver coil(s). The receiver information is processed by a computer, and an image is produced. [0031] The image and resolution produced by the MRI can be quite detailed and can detect tiny changes of structures and function of tissues within the body. For some procedures, contrast agents, such as gadolinium, can be used to increase the accuracy of the images.

[0032] As used herein, an“MRI pulsed sequence” or“MRI sequence” denotes a programmed set of changing radiofrequency pulses and magnetic gradients that are designed to result in images that emphasize one or more desired tissue image features (or appearance). Each sequence will have a number of parameters, and multiple sequences are grouped together into an MRI protocol. Examples of the types of MRI pulsed sequences that are available include, but are not limited to: spin echo sequences (e.g., Tl-weighted, T2-weighted, etc.), inversion recovery sequences, gradient echo sequences, diffusion weighted sequences, saturation- recovery sequences, echo planar pulse sequences, spiral pulse sequences, etc.

[0033] As used herein, a“structural MRI” denotes MRI techniques that are focused on providing detailed images of anatomical structures, most commonly neurological or other soft (e.g., tendons, ligaments, fascia, skin, fibrous, fat, synovial membranes, muscles, nerves, blood vessels, etc.) tissue structures. Examples of MRI pulsed sequences that can be used for structural MRI include, but are not limited to: spin echo sequences (e.g., Tl-weighted, T2- weighted, etc.), inversion recovery sequences, gradient echo sequences, diffusion weighted sequences, etc.

[0034] As used herein, functional MRI denotes MRI techniques that are focused on providing images that can emphasize non-structural, anatomical information within tissues such as metabolic activity and the diffusive properties of water in tissue. Examples of MRI pulsed sequences that can be used for functional MRI include, but are not limited to: diffusion weighted sequences, arterial spin labeling, etc.

Neural Networks for Characterizing and/or Quantifying Brain Regions

[0035] Various aspects and embodiments are disclosed herein for applying neural network techniques to quantify aspects of the morphological features in various regions of the brain to develop disease risk models, identify biomarkers for disease progression, cognitive decline, and impairment. For example, in one aspect, a convolutional neural network is trained using a training database that contains patient cohort training data that pairs each patient’ s MRI images (e.g., 3D Tl-weighted MRI images, etc.) with their corresponding cortical surface and volumetric measurements. The trained convolutional neural network can then be used to segment regions of a subject’s brain using their structural MRI images (i.e., 3D Tl-weighted MRI images) and quantify certain aspects (e.g., volume, surface area, thickness, etc.) of the morphological features (e.g., hippocampus, inferior lateral ventricles, amygdala, etc.) of the subject’s brain.

[0036] In another aspect, the trained convolutional neural network can identify structural abnormalities (e.g., lesions, etc.) in one or more segmented regions of the brain and identify them visually using overlaid masks.

[0037] It is a common misconception that deep learning is time consuming. Although training may take a while, once the deep learning network is trained, inference is relatively fast. For some real-time processing (e.g., autonomous vehicles), inference times of lOOms or greater could be a problem, but that is not necessary for our biomedical imaging applications, which do not need to be instantaneous.

[0038] Advantages of the disclosed aspects includes a reduction in post-processing time from hours (e.g., conventional techniques require eight to twelve or more hours) to seconds, which reduces computational costs and is feasible for quick analysis and same day delivery of results. In contrast, atlas-based approaches and statistical approaches require iterative steps, which can be time consuming to converge and have a non-deterministic time for completion. By utilizing neural networks, these iterative steps are avoided. Furthermore, processing may be sped up by utilizing GPUs and Spark, if needed. These two factors greatly reduce the processing time.

[0039] FIG. 1 illustrates volumetric measurements 110 and surface measurements 120 derived from a Tl-weighted image 100, according to certain aspects of the disclosure. As illustrated by FIG. 1, the approach disclosed herein derives surface areas, thicknesses, and volume measurements of various regions of the brain, which are color-coded as different colors. These measurements are compared to diseased and healthy populations to determine risk profiles and diagnose diseases for a patient. As illustrated, the disclosed aspects utilize a Tl-weighted MRI image 100 to generate volumetric measurements 110 and surface measurements 120 from an automatically generated segmented brain image. This approach takes less than 10 minutes, whereas conventional (e.g., atlas-based) approaches take three to twelve or more hours.

[0040] FIG. 2A-2C illustrate an exemplary system 200 for training a neural network 230, according to certain aspects of the disclosure. As shown in FIG. 2, manually annotated and/or derived masks 210 for volumetric segmentation and 3D mesh data 220 are used as ground truths for training the model (e.g., the neural network 230). Masks 210 consist of a series of slices of binary images replicated for each class (e.g., hippocampus, ventricles). Meshes 220 consist of (x,y,z) coordinates/vertices and classes for each of the vertices. For example, the training data may include thousands of diverse DXs. A training database 222 for storing training images generated by an MRI image of a brain may be created that contains triplicates of original images (e.g., 3D Tl -weighted images) paired with their corresponding cortical surfaces and volumetric measurements. A neural network 230, such as a V-NET or U-NET, may be utilized for segmentation of subcortical regions. Although a 2D model is shown, a 3D model may also be utilized. A geometric neural network may also be implemented.

[0041] FIG. 3 illustrates an exemplary system 300 for training a neural network 320, according to certain aspects of the disclosure. As shown in FIG. 3, the neural network 320 may be trained using slabs of a full 3D volume 310 to reduce memory and reduce the number of parameters to improve training of the neural network 320. For example, the neural network 320 may include convolution layers 322 and deconvolution layers 324 for processing the slabs 310. The neural network 320 may then output 3D segmentation results 330, which may be utilized for risk stratification 340. For example, the risk stratification 340 may make a determination that a patient is at risk for dementia.

[0042] FIG. 4 illustrates a voxel matrix 400 showing axial 410, coronal 420, and sagittal 430 views, according to certain aspects of the disclosure. According to an aspect of the present disclosure, quantitative biomarkers may be determined from the automatically segmented MRI image. For example, data from the multiple MRI image slices are represented in a 3D voxel matrix 400, whose volume represents the total volume arising from the sum total of the surface area of each image slice. A slice of this matrix, such as a two-dimensional matrix, represents data for a corresponding image slice in one of the three spatial orientations (e.g., axial 410, coronal 420, and sagittal 430).

[0043] Each element of matrix M, m_{X y Z}, is denoted as a voxel, and its dimensions represent the fundamental units of measurement used to calculate volumes and surface areas of brain regions identified from image slice data and machine leaning algorithms. Each voxel contains an integer value that is a predefined identifier to a specific brain region.

[0044] Region volumes of the brain may be calculated by using the voxels. For example, the width, height, and depth (e.g., the spatial dimensions) of each voxel may be denoted as dr, dy, and dz, respectively. The dimensions for each voxel are calculated by dividing the sum width, sum height, and sum depth of the image slices by the number of matrix elements rq in the voxel matrix M in the x, y, and z directions, respectively. Thus,

[0047] dz =

nz

[0048] Where /, is the number of image slices in the x, y, or z spatial direction and z, are the image slice thickness in the corresponding dimension.

[0049] Thus, the voxel volume, v, is: V— dx · dy · dz

[0050] Generally, t_x = t_y = t_z, thus dx = dy = dz, and v reduces to: df where I is any one of the three spatial dimensions.

[0051] From machine learning algorithms, units of each brain sub-region are identified along a grid corresponding to the voxel locations described above and each are assigned a predefined integer value and as its corresponding voxel location.

[0052] FIG. 5 illustrates a 2D slice 500 of a voxel matrix M, according to certain aspects of the disclosure. For example, a 2D slice 500 of the voxel matrix M, may be filled with integer values pertaining to various sub-cortical brain structures, which are mapped from ML algorithms from an image slice (e.g., slice z = k).

[0053] According to an aspect, the total volume, V, of region p, that pertains to a specific brain substructure 510 may be calculated. For example, the voxel volume, v_{x y z}, is summed for each voxel matrix element

_z in M that has a value p, the value of the region in question, which yields:

[0054] V (p) = å x,y,z, v_{x y z}, where indices x, y, z are selected such that elements

-X y z

mx,y,z P-

[0055] In general, the voxels are cubic and voxel volume v_{x y z} is equal to constant v. Thus, by denoting the number of elements n_p in the set of all voxels S_m in M that have value p, then the above equation above reduces to:

[0056] F(p) = n_p v

Surface Area Calculation

[0057] Two kinds of surface area may be calculated from the voxels: (1) the total surface area of a brain region carved out by its volume in the voxel matrix, and (2) the surface area brought about by voxels touching a specific region not having the same value as the voxel itself. One example would be the cortical surface area, which is the part of the cortex touching the super-cortical regions, but excluding white matter and adjacent cortical structures.

[0058] In general, both calculations involve summing the surface area for each face of a given voxel type touching the face of a different voxel type and not its own like faces. As mentioned, the voxels are in general cubic and so the surface area, s_v, of each of the six voxel faces is v ²/³ or the square of any two of the dimensions of a voxel (e.g., dx² ).

[0059] In the case where the voxels are not cubic, the s_vi for each face i, (x y, x z, y z) is calculated and indexed accordingly based on their direction in space.

[0060] To find the total surface area A p) of a given brain region p, the set of voxels S(p) is gathered from M of the region in question. Then for each voxel, each of the six faces is examined, f_pi of voxel i, S_pi, in S(p) and sum the surface area of those faces touching region q ¹ p.

[0061] This amounts to looking at the six adjacent voxels of S_pi, S_pi(x ± l, y ± 1, z ± 1). For any adjacent voxel having value q ¹ p, the surface area of the face is added to the total surface area.

[0062] FIG. 6 illustrates a voxel diagram 600 for calculation of a total surface area, according to certain aspects of the disclosure. For example, voxel S_pi may be the center cube 610 in bold with p = 3, and five of its neighboring voxels. It is noted that a front-most neighbor has been omitted for clarity.

[0063] Here the surface area A(p) for the central cube of p = 3 would be the sum of areas of the faces f_3i touching voxels q = 2 (top and right), q = 1 (back), and the bottom, q = 4, with the face touching the left voxel q = 3, being omitted because it is the same type.

[0064] The procedure for calculating the partial surface area is nearly identical to the total surface area calculation except that only the sum faces fq p q=_k, where k is the only external voxel value is considered.

[0065] For example, this may be a value pertaining to a region outer to the cortical surface. If the voxel value for this is 2, then, looking at the diagram above, only voxel faces touching q = 2, the top and right faces, would be considered. The surface areas of only these two faces are added, and then the procedure is repeated for all voxels in the set S(p).

[0066] According to an aspect, the surface areas may be calculated from a parcellated surface representation derived from the voxel matrix.

Average Cortical Thickness

[0067] FIG. 7 illustrates three layers utilized in calculating an average cortical thickness, according to certain aspects of the disclosure. A central layer 720 is the cerebral cortex or gray matter (pial ) layer (e.g., cortical region), the average thickness of which will be calculated. Immediately below is a white matter layer 730 and above is a super cortical area 710. [0068] The calculations are performed for each of several cortical sections, which are predefined segments of the cortex, such as entorhinal, fusiform, parahippocampal, etc., for both the left and right hemispheres.

[0069] To perform the calculation, first the set of voxels S(p) is obtained from M pertaining to cortical region p, (e.g., left fusiform) depicted in the central layer 720.

[0070] From this a subset, S' (p ) is obtained of only the voxels that border a voxel from the super cortical region 710, that is, the surface voxels.

[0071] A second set of voxels W (p), is obtained that is made up of white matter voxels 730, that border cortical region 720.

[0072] The Cartesian coordinates, in mm, which is the fundamental voxel unit, are then found for each set S' (p) and W (p).

[0073] For each voxel in S'(p), the white matter voxel in W (p) is determined that is closest in distance, and this value is added to a total sum by repeating this procedure over all voxels in S'(p). Finally, the result is divided by the total number of calculations performed, that is, size n_s of S'(p).

[0074] To clarify, functional f_min(S'_pi— W (p)) is denoted that finds the minimum distance of voxel S,_pi in S'(p) to all the voxels in W (p).

[0075] With this, the average cortical thickness, T_ave is:

[0077] This calculation is then performed for each of the cortical subsections considered in the model, thereby obtaining the average thickness for each.

Evaluation of Neural Networks

[0078] The performance of the neural network can be assessed using the Dice Coefficient. The Dice Coefficient measures how much of an overlap between the target and the resultant mask according to:

DSC -

[0079]

[0080] The performance can be also be assessed using a geodesic error (e.g., geodesic distance) rather than a Euclidean distance and shape correspondences, as well as classification accuracy for region assignment of the nodes. Shape correspondence refers to the task of labeling each node of a given shape to the corresponding node of a reference shape. [0081] As described herein, the aspects may be applied to various neurological applications including but not limited to ML risk models and statistical risk prediction as well as biomarkers of dementia and Alzheimer’s Disease, cognitive decline, and early and late mild cognitive impairment, brain aging, sleep, depression, anxiety, OCD, plasticity, fluid intelligence, memory, Autism, and ischemic stroke. Moreover, aspects of the present disclosure can be used to enable patients to leam and discover information about the brain in a personalized, visually appealing, and interactive way. Finally, aspects can be applied to other MR derived images such as Fluid-attenuated inversion recovery (FLAIR) for the quantification and assessment (e.g., lesion location in the brain, lesion volume, number of lesion) for white matter hyperintensities, which are associated with risk of stroke, cognitive decline, dementia, and mortality.

[0082] According to an aspect of the present disclosure, brain region segmentation via a convolutional neural network using anatomical imaging, Tl -weighted imaging is performed.

[0083] In some embodiments, segmentation and localization of abnormalities, including but not limited to white matter lesion hyperintensities, using overlaid masks from brain region segmentation is performed, as illustrated in FIG. 8.

[0084] In some embodiments, the cortical surface is determined using geometric deep learning. According to additional aspects, quantification of derived volumes, morphology, and shapes is determined.

[0085] The techniques described herein may be implemented as method(s) that are performed by physical computing device(s), as one or more non-transitory computer-readable storage media storing instructions (e.g., stored sequences of instructions) which, when executed by computing device(s), cause performance of the method(s), or, as physical computing device(s) that are specially configured with a combination of hardware and software that causes performance of the method(s).

[0086] FIG. 9 illustrates an example flow diagram 900 for quantification of brain regions, according to certain aspects of the disclosure. For explanatory purposes, the blocks of the example process 900 are described herein as occurring in serial, or linearly. However, multiple blocks of the example process 900 may occur in parallel. In addition, the blocks of the example process 900 need not be performed in the order shown and/or one or more of the blocks of the example process 900 need not be performed.

[0087] At block 902, an MRI image of a brain is obtained. The MRI image may include multiple slabs, each slab comprising multiple slices, each slice comprising a gray-scale image of the brain taken at a different cross-section. [0088] At block 904, a series of convolutions, pooling, and upsampling is performed on each slab to obtain a segmented output. The segmented output may include groups of segmented slabs segmented according to a class.

[0089] At block 906, the groups of segmented slabs are combined together to obtain a segmented MRI image. The segmented MRI image may include a voxel matrix. At block 908, volumes, surface areas, and thicknesses of the brain regions are calculated based on the voxel matrix.

[0090] According to an aspect of the present disclosure, the process 900 further includes generating risk models and detecting diseases based on the calculated volumes and surface areas.

[0091] According to aspects, obtaining the MRI image further comprises deriving mesh data from the MRI image, the mesh data comprising coordinates, vertices, and a category for each of the vertices.

[0092] According to an aspect of the present disclosure, the process 900 further includes generating a database comprising labeled data for training purposes, the labeled data comprising triplicates of the MRI image paired with corresponding labeled cortical surface images and volumetric measurements of the brain.

[0093] According to aspects, calculating the volumes comprises summing voxel elements of the voxel matrix corresponding to specific brain regions, the specific brain regions determined by the segmented MRI image.

[0094] According to aspects, calculating the surface areas comprises calculating a total surface area of the brain, and calculating a partial surface area of the brain.

[0095] According to an aspect of the present disclosure, the process 900 further includes calculating a cortical thickness of the brain based on the voxel matrix.

[0096] According to an aspect of the present disclosure, the process 900 further includes segmenting and localizing abnormalities in the brain by overlaying masks of the abnormalities over the segmented MRI image, the abnormalities comprising lesion hyperintensities.

[0097] According to an aspect of the present disclosure, the process 900 further includes validating the segmented MRI image when an overlap between the segmented image and a manually labeled MRI image is within a defined threshold, the defined threshold based on a geodesic error.

[0098] According to aspects, performing the series of convolutions, pooling, and upsampling comprises contracting features of the MRI image to capture context through repeated convolutions, each convolution followed by a rectified linear unit (ReLU) and a max pooling operation for downsampling, and expanding features of the MRI image to localize the context through upsampling of a feature map followed by convolutions. The expanding may include concatenation with a correspondingly cropped feature map from the contracting.

Computer-Implemented System

[0099] FIG. 10 is a block diagram that illustrates a computer system 1000, upon which embodiments of the present teachings may be implemented. In various embodiments of the present teachings, computer system 1000 can include a bus 1002 or other communication mechanism for communicating information, and a processor 1004 coupled with bus 1002 for processing information. In various embodiments, computer system 1000 can also include a memory, which can be a random access memory (RAM) 1006 or other dynamic storage device, coupled to bus 1002 for determining instructions to be executed by processor 1004. Memory also can be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1004. In various embodiments, computer system 1000 can further include a read-only memory (ROM) 1008 or other static storage device coupled to bus 1002 for storing static information and instructions for processor 1004. A storage device 1010, such as a magnetic disk or optical disk, can be provided and coupled to bus 1002 for storing information and instructions.

[0100] In various embodiments, computer system 1000 can be coupled via bus 1002 to a display 1012, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. An input device 1014, including alphanumeric and other keys, can be coupled to bus 1002 for communicating information and command selections to processor 1004. Another type of user input device is a cursor control 1016, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 1004 and for controlling cursor movement on display 1012. This input device 1014 typically has two degrees of freedom in two axes, a first axis (i.e., x) and a second axis (i.e., y), that allows the device to specify positions in a plane. However, it should be understood that input devices 1014 allowing for three-dimensional (x, y, and z) cursor movement are also contemplated herein.

[0101] Consistent with certain implementations of the present teachings, results can be provided by computer system 1000 in response to processor 1004 executing one or more sequences of one or more instructions contained in memory 1006. Such instructions can be read into memory 1006 from another computer-readable medium or computer-readable storage medium, such as storage device 1010. Execution of the sequences of instructions contained in memory 1006 can cause processor 1004 to perform the processes described herein. Alternatively, hard-wired circuitry can be used in place of or in combination with software instructions to implement the present teachings. Thus, implementations of the present teachings are not limited to any specific combination of hardware circuitry and software.

[0102] The term“computer-readable medium” (e.g., data store, data storage, etc.) or “computer-readable storage medium” as used herein refers to any media that participates in providing instructions to processor 1004 for execution. Such a medium can take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Examples of non-volatile media can include, but are not limited to, optical, solid state, and magnetic disks, such as storage device 1010. Examples of volatile media can include, but are not limited to, dynamic memory, such as memory 1006. Examples of transmission media can include, but are not limited to, coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 1002.

[0103] Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other tangible medium from which a computer can read.

[0104] In addition to a computer-readable medium, instructions or data can be provided as signals on transmission media included in a communications apparatus or system to provide sequences of one or more instructions to processor 1004 of computer system 1000 for execution. For example, a communication apparatus may include a transceiver having signals indicative of instructions and data. The instructions and data are configured to cause one or more processors to implement the functions outlined in the disclosure herein. Representative examples of data communications transmission connections can include, but are not limited to, telephone modem connections, wide area networks (WAN), local area networks (LAN), infrared data connections, NFC connections, etc.

[0105] It should be appreciated that the methodologies described herein such as flow charts, diagrams, and the accompanying disclosure can be implemented using computer system 1000 as a standalone device or on a distributed network of shared computer processing resources such as a cloud computing network.

[0106] The methodologies described herein may be implemented by various means depending upon the application. For example, these methodologies may be implemented in hardware, firmware, software, or any combination thereof. For a hardware implementation, the processing unit may be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or a combination thereof.

[0107] In various embodiments, the methods of the present teachings may be implemented as firmware and/or a software program and applications written in conventional programming languages such as C, C++, Python, etc. If implemented as firmware and/or software, the embodiments described herein can be implemented on a non-transitory computer-readable medium in which a program is stored for causing a computer to perform the methods described above. It should be understood that the various engines described herein can be provided on a computer system, such as computer system 1000 of Appendix B, whereby processor 1004 would execute the analyses and determinations provided by these engines, subject to instructions provided by any one of, or a combination of, memory components 1006/1008/1010 and user input provided via input device 1014.

[0108] While the present teachings are described in conjunction with various embodiments, it is not intended that the present teachings be limited to such embodiments. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be appreciated by those of skill in the art.

[0109] Further, in describing various embodiments, the specification may have presented a method and/or process as a particular sequence of steps. However, to the extent that the method or process does not rely on the particular order of steps set forth herein, the method or process should not be limited to the particular sequence of steps described. As one of ordinary skill in the art would appreciate, other sequences of steps may be possible. Therefore, the particular order of the steps set forth in the specification should not be construed as limitations on the claims. In addition, the claims directed to the method and/or process should not be limited to the performance of their steps in the order written, and one skilled in the art can readily appreciate that the sequences may be varied and still remain within the spirit and scope of the various embodiments.

[0110] The embodiments described herein can be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The embodiments can also be practiced in distributing computing environments where tasks are performed by remote processing devices that are linked through a network. [0111] It should also be understood that the embodiments described herein can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing.

[0112] Any of the operations that form part of the embodiments described herein are useful machine operations. The embodiments, described herein, also relate to a device or an apparatus for performing these operations. The systems and methods described herein can be specially constructed for the required purposes or it may be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0113] Certain embodiments can also be embodied as computer-readable code on a computer-readable medium. The computer-readable medium is any data storage device that can store data, which can thereafter be read by a computer system. Examples of the computer- readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical, FLASH memory and non-optical data storage devices. The computer-readable medium can also be distributed over a network coupled to computer systems so that the computer-readable code is stored and executed in a distributed fashion.

Claims

CLAIMS What is claimed is:

1. A method for quantification of brain regions, comprising:

obtaining an MRI image of a brain, the MRI image comprising multiple slabs, each slab comprising multiple slices, each slice comprising a gray-scale image of the brain taken at a different cross-section;

performing a series of convolutions, pooling, and upsampling on each slab to obtain a segmented output, the segmented output comprising groups of segmented slabs segmented according to a class;

combining the groups of segmented slabs together to obtain a segmented MRI image, the segmented MRI image comprising a voxel matrix; and

calculating volumes, surface areas, and thicknesses of the brain regions based on the voxel matrix.

2. The method of claim 1, further comprising:

generating risk models and detecting diseases based on the calculated volumes and surface areas.

3. The method of claim 1, wherein obtaining the MRI image further comprises:

deriving mesh data from the MRI image, the mesh data comprising coordinates, vertices, and a category for each of the vertices.

4. The method of claim 1, further comprising:

generating a database comprising labeled data for training purposes, the labeled data comprising triplicates of the MRI image paired with corresponding labeled cortical surface images and volumetric measurements of the brain.

5. The method of claim 1, wherein calculating the volumes comprises:

summing voxel elements of the voxel matrix corresponding to specific brain regions, the specific brain regions determined by the segmented MRI image.

6. The method of claim 1, wherein calculating the surface areas comprises:

calculating a total surface area of the brain; and

calculating a partial surface area of the brain.

7. The method of claim 1, further comprising:

calculating a cortical thickness of the brain based on the voxel matrix.

8. The method of claim 1, further comprising:

segmenting and localizing abnormalities in the brain by overlaying masks of the abnormalities over the segmented MRI image, the abnormalities comprising lesion hyperintensities.

9. The method of claim 1, further comprising:

validating the segmented MRI image when an overlap between the segmented image and a manually labeled MRI image is within a defined threshold, the defined threshold based on a geodesic error.

10. The method of claim 1, wherein performing the series of convolutions, pooling, and upsampling comprises:

contracting features of the MRI image to capture context through repeated convolutions, each convolution followed by a rectified linear unit (ReLU) and a max pooling operation for downsampling; and

expanding features of the MRI image to localize the context through upsampling of a feature map followed by convolutions, the expanding including concatenation with a correspondingly cropped feature map from the contracting.

11. A system for quantification of brain regions, comprising:

a magnetic resonance imaging device configured to obtain an MRI image of a brain, the MRI image comprising multiple slabs, each slab comprising multiple slices, each slice comprising a gray-scale image of the brain taken at a different cross-section; and

a computing device communicatively connected to the magnetic resonance imaging device, comprising:

a mask generator configured to perform a series of convolutions, pooling, and upsampling on each slab to obtain a segmented output, the segmented output comprising groups of segmented slabs segmented according to a class, the mask generator further configured to combine the groups of segmented slabs together to obtain a segmented MRI image, the segmented MRI image comprising a voxel matrix; and

a voxel calculator configured to calculate volumes, surface areas, and thicknesses of the brain regions based on the voxel matrix.

12. The system of claim 12, further comprising a profile generator configured to generate risk models and detect diseases based on the calculated volumes and surface areas.

13. The system of claim 12, wherein the magnetic resonance imaging device is further configured to:

derive mesh data from the MRI image, the mesh data comprising coordinates, vertices, and a category for each of the vertices.

14. The system of claim 12, further comprising a database generator configured to: generate a database comprising labeled data for training purposes, the labeled data comprising triplicates of the MRI image paired with corresponding labeled cortical surface images and volumetric measurements of the brain.

15. The system of claim 12, wherein the voxel calculator is further configured to:

sum voxel elements of the voxel matrix corresponding to specific brain regions, the specific brain regions determined by the segmented MRI image.

16. The system of claim 12, wherein the voxel calculator is further configured to:

calculate a total surface area of the brain; and

calculate a partial surface area of the brain.

17. The system of claim 12, wherein the voxel calculator is further configured to:

calculate cortical thickness of the brain based on the voxel matrix.

18. The system of claim 12, wherein the mask generator is further configured to:

segment and localize abnormalities in the brain by overlaying masks of the abnormalities over the segmented MRI image, the abnormalities comprising lesion hyperintensities.

19. The system of claim 12, wherein the mask generator is further configured to:

validate the segmented MRI image when an overlap between the segmented image and a manually labeled MRI image is within a defined threshold, the defined threshold based on a geodesic error.

20. The system of claim 12, wherein the mask generator is further configured to:

contract features of the MRI image to capture context through repeated convolutions, each convolution followed by a rectified linear unit (ReLU) and a max pooling operation for downsampling; and

expand features of the MRI image to localize the context through upsampling of a feature map followed by convolutions, the expanding including concatenation with a correspondingly cropped feature map from the contracting.