US11164067B2 - Systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging - Google Patents

Systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging Download PDF

Info

Publication number
US11164067B2
US11164067B2 US16/556,130 US201916556130A US11164067B2 US 11164067 B2 US11164067 B2 US 11164067B2 US 201916556130 A US201916556130 A US 201916556130A US 11164067 B2 US11164067 B2 US 11164067B2
Authority
US
United States
Prior art keywords
layer block
layer
decoder
blocks
neural network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US16/556,130
Other versions
US20200074271A1 (en
Inventor
Jianming Liang
Zongwei Zhou
Md Mahfuzur Rahman Siddiquee
Nima Tajbakhsh
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Arizona Board of Regents of ASU
Original Assignee
Arizona Board of Regents of ASU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US201862724089P priority Critical
Application filed by Arizona Board of Regents of ASU filed Critical Arizona Board of Regents of ASU
Priority to US16/556,130 priority patent/US11164067B2/en
Assigned to ARIZONA BOARD OF REGENTS ON BEHALF OF ARIZONA STATE UNIVERSITY reassignment ARIZONA BOARD OF REGENTS ON BEHALF OF ARIZONA STATE UNIVERSITY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: RAHMAN SIDDIQUEE, MD MAHFUZUR, TAJBAKHSH, NIMA, LIANG, JIANMING, ZHOU, Zongwei
Publication of US20200074271A1 publication Critical patent/US20200074271A1/en
Application granted granted Critical
Publication of US11164067B2 publication Critical patent/US11164067B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/10Segmentation; Edge detection
    • G06T7/11Region-based segmentation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Computing arrangements based on biological models using neural network models
    • G06N3/04Architectures, e.g. interconnection topology
    • G06K9/0014
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KGRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for recognising patterns
    • G06K9/62Methods or arrangements for pattern recognition using electronic means
    • G06K9/6267Classification techniques
    • G06K9/6268Classification techniques relating to the classification paradigm, e.g. parametric or non-parametric approaches
    • G06K9/627Classification techniques relating to the classification paradigm, e.g. parametric or non-parametric approaches based on distances between the pattern to be recognised and training or reference patterns
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Computing arrangements based on biological models using neural network models
    • G06N3/04Architectures, e.g. interconnection topology
    • G06N3/0454Architectures, e.g. interconnection topology using a combination of multiple neural nets
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Computing arrangements based on biological models using neural network models
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Computing arrangements based on biological models using neural network models
    • G06N3/08Learning methods
    • G06N3/084Back-propagation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/60Type of objects
    • G06V20/69Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/695Preprocessing, e.g. image segmentation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Computing arrangements based on biological models using neural network models
    • G06N3/08Learning methods
    • G06N3/082Learning methods modifying the architecture, e.g. adding or deleting nodes or connections, pruning
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00Computing arrangements based on specific mathematical models
    • G06N7/005Probabilistic networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20016Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]

Abstract

Disclosed are provided systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging. For example, a system having means to execute a neural network model formed from a plurality of layer blocks including an encoder layer block which precedes a plurality of decoder layer blocks includes: means for associating a resolution value with each of the plurality of layer blocks; means for processing via the encoder layer block a respective layer block input including a down-sampled layer block output processing, via decoder layer blocks, a respective layer block input including an up-sampled layer block output and a layer block output of a previous layer block associated with a prior resolution value of a layer block which precedes the respective decoder layer block; and generating the respective layer block output by summing or concatenating the processed layer block inputs.

Description

CLAIM OF PRIORITY
This non-provisional U.S. Utility Patent Application is related to, and claims priority to the U.S. Provisional Patent Application No. 62/724,089, entitled “MULTI-RESOLUTION NEURAL NETWORKS,” filed Aug. 29, 2018, the entire contents of which are incorporated herein by reference.
GOVERNMENT RIGHTS AND GOVERNMENT AGENCY SUPPORT NOTICE
None.
COPYRIGHT NOTICE
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
TECHNICAL FIELD
Embodiments of the invention relate generally to the field of imaging segmentation, and more particularly, to systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging.
BACKGROUND
The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also correspond to embodiments of the claimed inventions.
Machine learning models operate to receive input and responsively generate output or predicted outputs, based on the received input. Some machine learning models are parametric models which generate output based on the received input and on values of the parameters of the machine learning model. Other machine learning models utilize so called “deep learning” or “deep machine learning models” which utilize multiple layers to progressively extract higher level features from received input to generate an output. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden or intermediate layers that each apply a non-linear transformation to a received input to generate an output.
Machine learning models have various applications to automatically process inputs and produce outputs considering situational factors and learned information to improve output quality. One area where machine learning models, and neural networks in particular, provide high utility is in the field of imaging segmentation, including for use with segmenting medical images. Models used for medical image segmentation commonly include encoder-decoder networks such as U-Net and fully convolution networks (FCNs), each of which use skip connection pathways that have information gaps in gradient flow.
Problematically, segmenting medical images to visualize them as a basis for diagnosing abnormalities requires a higher level of accuracy than is possible with raw input images. Any misdiagnosis or failure to recognize abnormalities may result in potentially devastating impacts on patient morbidity and mortality. Specifically, segmenting lesions and other abnormalities in medical images requires a high level of accuracy so as to ensure effective diagnosis and to avoid missing pathological hallmarks such as speculation patterns on nodules indicative of malignancy and further to avoid rendering a diagnosis based on inaccurate medical images such as an inaccurate display of the number of lymph nodes in an image which results in erroneous scoring of diagnostic criteria.
Embodiments of the invention therefore improve upon conventionally known image segmentation techniques and thus improve diagnostic accuracy in the medical field through the re-designing of network skip connections to allow for progressive feature adaptation. These improvements in turn improve patient health and save lives, as well as improving image recognition in a wide array of technical fields. Stated differently, the methodologies described herein provide solutions to improving systems for dealing with image representations of objects that are subject to unknown, ever-changing, and varying parameters, conditions, and characteristics, such as pathology in the human body.
The present state of the art may therefore benefit from systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging as is described herein.
BRIEF DESCRIPTION OF THE DRAWINGS
Embodiments are illustrated by way of example, and not by way of limitation, and can be more fully understood with reference to the following detailed description when considered in connection with the figures in which:
FIG. 1A depicts an exemplary overview of UNet++ architecture 100 in accordance with described embodiments;
FIG. 1B depicts an exemplary detailed analysis of the first skip connection pathway (top connection pathway) of UNet++ 110 in accordance with described embodiments;
FIG. 1C depicts an exemplary overview of UNet++ pruning with deep supervision in accordance with described embodiments;
FIG. 2 depicts an exemplary qualitative comparison of image segmentation results between U-Net, wide U-Net, and U-Net++ architectures for polyp, liver and cell nuclei datasets, in a 2D-view for distinct visualization, in accordance with described embodiments;
FIG. 3 depicts an exemplary comparison of complexity, speed, and accuracy of UNet++ after pruning on segmentation tasks for various image segmentation tasks, in accordance with described embodiments;
FIG. 4A depicts an exemplary architecture of UNet++ in accordance with described embodiments;
FIG. 4B depicts exemplary topologies of U-Net like, UNet+, and UNet++, in accordance with described embodiments;
FIG. 5 depicts feature-maps for (a) U-Net, (b) UNet+ and (c) UNet++ including intermediate layers based on the same input, with the outputs of each architecture compared to a truth, in accordance with described embodiments;
FIG. 6 depicts validation loss and dice coefficient for different network structures along each epoch training stage, in accordance with described embodiments.
FIG. 7 is a depiction of UNet+ architecture in accordance with described embodiments;
FIG. 8 depicts visual comparisons between varying depth U-Net architectures ranging from 1-depth to 4-depth, in accordance with described embodiments;
FIG. 9 depicts further visual comparisons between UNet{circumflex over ( )}e and UNet+, in accordance with described embodiments;
FIG. 10 is a visualization of intermediate feature-maps concatenation and flow along the top components of a nested network, in accordance with described embodiments;
FIG. 11A depicts examples of cell segmentation intermediate feature-maps and predictions of U-Net, UNet+ and UNet++ architectures without deep supervision, in accordance with described embodiments;
FIG. 11B depicts examples of cell segmentation intermediate feature-maps and predictions of UNet+ and UNet++ with deep supervision, in accordance with described embodiments;
FIG. 12 depicts a flow diagram illustrating a method for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging, in accordance with disclosed embodiments;
FIG. 13 shows a diagrammatic representation of a system within which embodiments may operate, be installed, integrated, or configured in accordance with described embodiments; and
FIG. 14 illustrates a diagrammatic representation of a machine in the exemplary form of a computer system, in accordance with one embodiment.
DETAILED DESCRIPTION
Described herein are systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging. For example, there are described in accordance with certain embodiments, a system having at least a processor and memory therein and being specifically configured to carry out operations including: executing, via the processor, a neural network model stored within the memory of the system; in which the neural network model is formed from a plurality of layer blocks including an encoder layer block which precedes a plurality of decoder layer blocks; associating a resolution value with each of the plurality of layer blocks defining a resolution of the layer block input to be processed by the layer block to generate the layer block output; processing, via the encoder layer block, a respective layer block input including a down-sampled layer block output from a higher resolution layer block associated with a resolution value greater than the resolution value for the encoder layer block; processing, via each of the plurality of decoder layer blocks, a respective layer block input including both (i) an up-sampled layer block output from a lower resolution layer block associated with a resolution value less than the resolution value for the respective decoder layer block and (ii) a layer block output of a previous layer block associated with a prior resolution value of a layer block which precedes the respective decoder layer block; and generating the respective layer block output from the processing via the encoder layer block and the plurality of decoder layer blocks by summing or concatenating the processed layer block inputs.
According to described embodiments, there are provided system and methodologies for implementing an improved a neural network which includes multiple layer blocks. Each layer block includes one or more neural network layers and each layer block is configured to process a respective layer block input to generate a respective layer block output. Each layer block is associated with a resolution value defining a resolution of the respective layer block input processed by the layer block to generate the respective layer block output.
For a particular resolution value, the layer blocks in the neural network associated with the respective resolution value have an ordering and include an encoder layer block and multiple decoder layer blocks. The encoder layer block precedes the decoder layer blocks in the ordering. The encoder layer block is configured to process a respective layer block input including a down-sampled layer block output of a higher resolution layer block associated with a resolution value that is greater than the particular resolution value. Each decoder layer block is configured to process a respective layer block input including: (i) an up-sampled layer block output of a lower resolution layer block associated with a resolution value that is less than the particular resolution value, and (ii) a layer block output of a previous layer block associated with the particular resolution value that is prior to the decoder layer block in the ordering. According to described embodiments, processing inputs to any layer block includes summing the inputs or linking (e.g., “concatenating”) the inputs.
According to such embodiments, each of the decoder layer blocks associated with the highest resolution value generates a layer block output defining a decoder output of the neural network. The neural network then generates a final output based on the decoder outputs generated by these decoder layer blocks associated with the highest resolution value, which may include averaging their outputs according to such embodiments or alternatively selecting a particular decoder output, based on the particular implementation preferences.
Practice of the disclosed embodiments yield several distinct advantages over conventional methodologies. For example, the neural networks described herein may include, for example, multiple different resolution values at each encoder layer block followed by multiple decoder layer blocks which may be densely connected via “skip connections,” whereas prior solutions utilize only a single encoder layer block of a particular resolution followed by a single decoder layer block. By including multiple decoder layer blocks at different resolutions, the neural network implementations set forth herein operate to smooth multiple resolution feature propagation and encourage feature-map reuse, thus attaining higher prediction accuracy (e.g., improved segmentation prediction accuracy) in comparison with prior known techniques.
Further still, the improved neural networks described herein consume fewer computational resources (e.g., memory and computing power) in comparison with conventional neural networks as they may be trained to achieve acceptable levels of performance using less training data and fewer iterations of machine learning training procedures than is possible with prior methodologies. The improved neural network architecture described herein facilitates such reductions in computational resource consumption, for example, via the utilization of multiple decoder layer blocks at each resolution which thus improves numerical stability during training of the neural network.
Further still, the neural network described herein may be trained using “deep supervision,” which results in the neural network being trained to generate outputs from multiple intermediate layers, each of which are trained to approximate the target output of the neural network. Training the neural network using deep supervision may therefore contribute to further reductions in computational resource consumption by the neural network during training compared to prior known techniques.
Certain embodiments disclosed herein therefore utilize deep supervision techniques and a multi-scale approach using segmentation branches with more than one loss layer to allow for model pruning. Deep supervision has been observed to be especially useful with segmenting components of organs and tissues that appear at varying scales in medical imaging such as cell nuclei and colon polyps.
Learning models have traditionally used plain skip connections that directly fast-forward high-resolution feature-maps from the encoder to the decoder network, resulting in the fusion of semantically dissimilar feature-maps, with recovery of some fine-grained details of target objects. Prior solutions simply do not modify skip connections through dense connections to create similar feature-maps for neural networks to improve the precision and other features of image segmentation.
The methodologies described herein further utilize neural network architectures capable of segmenting images based on nested and dense skip connections to more efficiently capture fine-grain details of foreground objects when high-resolution feature-maps from the encoder network are gradually enriched prior to fusion with corresponding semantically rich feature-maps from the decoder network. Described embodiments therefore improve the quality of image segmentation by optimizing an objective function (“loss term”) corresponding to each decoder output of the neural network which in turn better satisfies the stricter segmentation requirements for medical images. Further benefits realized through practice of the disclosed embodiments include accommodating for smooth multiple resolution feature propagation, re-use of feature-maps, less data augmentation and thus less computational resource consumption, and accelerated convergence speed.
In the following description, numerous specific details are set forth such as examples of specific systems, languages, components, etc., in order to provide a thorough understanding of the various embodiments. It will be apparent, however, to one skilled in the art that these specific details need not be employed to practice the embodiments disclosed herein. In other instances, well known materials or methods have not been described in detail in order to avoid unnecessarily obscuring the disclosed embodiments.
In addition to various hardware components depicted in the figures and described herein, embodiments further include various operations which are described below. The operations described in accordance with such embodiments may be performed by hardware components or may be embodied in machine-executable instructions, which may be used to cause a specialized and special-purpose processor having been programmed with the instructions to perform the operations described herein. Alternatively, the operations may be performed by a combination of hardware and software. In such a way, the embodiments of the invention provide a technical solution to a technical problem.
Embodiments also relate to an apparatus for performing the operations disclosed herein. This apparatus may be specially constructed for the required purposes, or it may be a special purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.
The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various customizable and special purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear as set forth in the description below. In addition, embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the embodiments as described herein.
Embodiments may be provided as a computer program product, or software, that may include a machine-readable medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to the disclosed embodiments. A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium (e.g., read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices, etc.), a machine (e.g., computer) readable transmission medium (electrical, optical, acoustical), etc.
Any of the disclosed embodiments may be used alone or together with one another in any combination. Although various embodiments may have been partially motivated by deficiencies with conventional techniques and approaches, some of which are described or alluded to within the specification, the embodiments need not necessarily address or solve any of these deficiencies, but rather, may address only some of the deficiencies, address none of the deficiencies, or be directed toward different deficiencies and problems which are not directly discussed.
In addition to various hardware components depicted in the figures and described herein, embodiments further include various operations which are described below. The operations described in accordance with such embodiments may be performed by hardware components or may be embodied in machine-executable instructions, which may be used to cause a special-purpose processor programmed with the instructions to perform the operations. Alternatively, the operations may be performed by a combination of hardware and software, including software instructions that perform the operations described herein via memory and one or more processors of a computing platform.
FIG. 1A depicts an exemplary overview of UNet++ architecture 100 in accordance with described embodiments.
As shown here, the UNet++ 100 architecture starts with an encoder sub-network or backbone 101 followed by a decoder sub-network 102 connected by a series of nested dense convolution blocks. Contrast this with a U-Net architecture which comprises encoder backbone 101, decoder sub-network 102 (e.g., decoder pathway 102) and linkage 104. The UNet++ is distinguished from U-Net by the nested dense convolution blocks 108 and re-designed skip connections, and skip connection pathways 105, which connect the two sub-networks to create the skip pathways for use with deep supervision 107. Utilization of UNet++ in such a way bridges the semantic gap between the feature-maps of the encoder backbone 101 and decoder sub-network 102 prior to fusion. For example, the semantic gap between (X0,0, X1,3) is bridged using a dense convolution block with three convolution layers.
Re-designed Skip Pathways: The re-designed skip pathways depicted here transform the connectivity of the encoder (e.g., backbone 101) and de-coder (e.g., decoder pathway 102) sub-networks. With U-Net, feature-maps of the encoder are directly received in the decoder. However, with use of UNet++, the feature-maps undergo a dense convolution block whose number of convolution layers depends on the pyramid level. For example, the skip pathway between nodes X0,0 and X1,3 consists of a dense convolution block with three convolution layers where each convolution layer is preceded by a concatenation layer that fuses the output from the previous convolution layer of the same dense block with the corresponding up-sampled output of the lower dense block. Consequently, the dense convolution blocks 108 bring the semantic level of the encoder feature-maps closer to that of the feature-maps awaiting in the decoder pathway 102. Complementary optimizers therefore face a simplified and therefore easier optimization problem when the received encoder feature-maps and the corresponding decoder feature-maps are semantically similar.
Formally, the skip pathways are formulated as follows: let xi,j denote the output of node Xi,j where i indexes the down-sampling layer along the encoder and j indexes the convolution layer of the dense block along the skip pathway. The stack of feature-maps represented by xi,j is computed in equation 1 below as follows:
x i , j = { ( x i - 1 , j ) , j = 0 ( [ [ x i , k ] k = 0 j - 1 , u ( x i + 1 , j - 1 ) ] ) , j > 0
where function
Figure US11164067-20211102-P00001
(·) is a convolution operation followed by an activation function,
Figure US11164067-20211102-P00002
(·) denotes an up-sampling layer, and [ ] denotes the concatenation layer. Nodes at level j=0 therefore receive only one input from the previous layer of the encoder, while nodes at level j=1 receive two inputs from the encoder sub-network, but at two consecutive levels, and further in which nodes at level j>1 receive j+1 inputs, of which j inputs are the outputs of the previous j nodes in the same skip pathway with the last input being the up-sampled output from the lower skip pathway. All prior feature-maps accumulate and arrive at the current node due to the use of a dense convolution block along each skip pathway.
FIG. 1B depicts an exemplary detailed analysis of the first skip connection pathway (top connection pathway) of UNet++ 110 in accordance with described embodiments.
Depicted in greater detail here is the manner in which the feature-maps travel through the top skip pathway of UNet++ 110, thus further clarifying equation 1 set forth above. Notably, the feature-maps travel through the top skip pathway from the encoder backbone 101 to the decoder sub-network 102 via convolution blocks connected by the first skip connection pathway 105. The path used in U-Net architecture is also shown, depicted via the solid arrow representing U-Net path 111.
Deep Supervision: Through use of deep supervision in UNet++, described methodologies enable the model to operate in two modes, including: (1) an accurate mode in which the outputs from all segmentation branches are averaged; and (2) a fast mode in which the final segmentation map is selected from only one of the segmentation branches, the choice of which determines the extent of model pruning and speed gain.
FIG. 1C depicts an exemplary overview of UNet++ pruning with deep supervision in accordance with described embodiments.
Depicted here is the manner by which the choice of segmentation branch when using the (2) fast mode results in architectures of varying complexity 120. Segmentation branch UNet++L1 124 involving nodes to X0,0 to X1,0 on encoder backbone 101 has the simplest architecture with only one convolution block 108, and thus the exhibits the greatest degree of model pruning consequently the greatest speed gain. In contrast, segmentation branch UNet++L4 121 involving nodes to X0,0 to X4,0 on encoder backbone 101 depicts the most complex architecture with six convolution blocks 108, and thus the least model pruning and least speed gain. Similarly, each of segmentation branch UNet++L2 123 involving nodes X0,0 to X2,0 on encoder backbone 101 and segmentation branch UNet++L3 122 involving nodes X0,0 to X3,0 to on encoder backbone 101, depict intermediate complexity in the architecture, and thus present an intermediate or balanced degrees of model pruning and speed gains.
Owing to the nested skip pathways, UNet++ generates full resolution feature-maps at multiple semantic levels, {x0,j,j∈{1,2,3,4}}, which are amenable to deep supervision. A combination of binary cross-entropy and dice coefficient is added as the loss function to each of the above four semantic levels, which is described as:
( Y , Y ^ ) = - 1 N b = 1 N ( 1 2 · Y b · log Y ^ b + 2 · Y b · Y ^ b Y b + Y ^ b )
where Ŷb and Yb denote the flattened predicted probabilities and the flattened ground truths of the bth image respectively, and N indicates the batch size.
The UNet++ architecture 100 depicted at FIG. 1A therefore differs from the original U-Net by (1) having convolution layers on skip connection pathways 105 which bridge the semantic gap between encoder backbone 101 and decoder sub-network 102 feature-maps; and further differ by (2) having dense skip connections on skip connection pathways 105 which improves gradient flow; and still further differ by (3) utilizing deep supervision 107 which enables model pruning and improves over the use of only one loss layer or at a minimum, attains comparable performance to using only one loss layer.
FIG. 2 depicts an exemplary qualitative comparison of image segmentation results between U-Net, wide U-Net, and U-Net++ architectures for polyp, liver and cell nuclei datasets, in a 2D-view for distinct visualization, in accordance with described embodiments.
Specifically, raw or input images 200 are segmented under each of the three architectures and compared to a ground truth 204 representation. Digital representation 208 depicts an exemplary digitized view of the raw or input images 200 which undergo processing via the three architectures.
As shown here, the UNet++ architecture 207 yields the closest output to accurately representing the ground truth images 204 for each of the polyp 201, liver 202, and nuclei 203. U-Net architecture 205 is the least accurate when compared to the ground truth images 204, followed by wide U-Net architecture 206 which is moderately accurate compared to ground truth images 204 but less accurate than the UNet++ architecture 207 described herein.
Experiments
Datasets: As shown in Table 1, four medical imaging datasets were utilized for experimental model evaluation, covering lesions/organs from different medical imaging modalities.
TABLE 1
Image segmentation datasets used for experiments.
Dataset Images Input Size Modality Provider
cell nuclei 670 96 × 96 microscopy Data Science Bowl
2018
colon polyp 7,379 224 × 224 RGB video ASU-Mayo
liver 331 512 × 512 CT MICCAI 2018 LiTS
lung nodule 1,012 64 × 64 × 64 CT LIDC-IDRI
Baseline models: For comparison, original U-Net and a customized wide U-Net architecture were utilized. The U-Net architecture was selected as it is a common performance baseline for image segmentation due to its wide adoption within the image processing and segmentation application space. A wide U-Net architecture with a similar number of parameters was further designed and utilized for experiments to ensure that the performance gains yielded by the proposed UNet++ architecture were not simply due to an increased number of parameters.
Table 2, depicts the quantity of convolutional kernels utilized for the U-Net and wide U-Net experiments.
TABLE 2
Number of convolutional kernels in U-Net and wide U-Net.
encoder/decoder
X0,0/X0,4 X1,0/X1,3 X2,0/X2,2 X3,0/X3,1 X4,0/X4,0
U-Net 32 64 128 256 512
Wide U-Net 35 70 140 280 560
Implementation details: For the purposes of the experiments, dice coefficient and Intersection over Union (IoU) were monitored. The experiments employed an early-stop mechanism on the validation set. Further utilized was an Adam optimizer with a learning rate of 3e−4.
The UNet++ architecture extended the original U-Net architecture by ensuring that convolutional layers along a skip pathway (Xi,j) use k kernels of size 3×3 (or 3×3×3 for 3D lung nodule segmentation) where k=32×2i. To enable deep supervision, a 1×1 convolutional layer followed by a sigmoid activation function was appended to each of the target nodes: {x0,j|j∈{1,2,3,4}}. As a result, the UNet++ architecture utilized for experimentation generated four segmentation maps given an input image, which were then further averaged to generate the final segmentation map.
Table 3 shows experimental results comparing U-Net, wide U-Net, and UNet++ architectures in terms of the number parameters and segmentation accuracy for the tasks of lung nodule segmentation, colon polyp segmentation, liver segmentation, and cell nuclei segmentation. Table 3 further provides the segmentation results (IoU %) for U-Net, wide U-Net and for the suggested architecture U-Net++ both with and without deep supervision (DS).
TABLE 3
Experimental results of varying architectures.
Dataset
Architecture Params cell nuclei colon polyp liver lung nodule
U-Net 7.76M 90.77 30.08 76.62 71.47
Wide U-Net 9.13M 90.92 30.14 76.58 73.38
UNet++ w/o 9.04M 92.63 33.45 79.70 76.44
DS
UNet++ w/ DS 9.04M 92.52 32.12 82.90 77.21
As shown here, wide U-Net consistently outperforms U-Net except for liver segmentation where the two architectures perform comparably. This improvement is attributed to the larger number of parameters in wide U-Net. The UNet++ architecture without deep supervision achieves a significant performance gain over both U-Net and wide U-Net, yielding average improvement of 2.8 and 3.3 points in IoU, respectively. The UNet++ architecture with deep supervision exhibits average improvement of 0.6 points over the UNet++ architecture without deep supervision. Notably, use of deep supervision leads to marked improvement for liver and lung nodule segmentation, but such improvements are not experimentally observed for cell nuclei and colon polyp segmentation due to the lung and liver nodules appearing at varying scales in video frames and CT slices. Consequently, a multi-scale approach using all segmentation branches (e.g., via deep supervision) is recommended to attain the most accurate segmentation from available imaging data inputs.
FIG. 3 depicts an exemplary comparison of complexity, speed, and accuracy of the UNet++ architecture after pruning on segmentation tasks for various image segmentation tasks, in accordance with described embodiments.
Model Pruning: Specifically depicted here are comparisons of inference times 301 vs. inference accuracy (inference mIoU (%)) 302 for each of cell nuclei, colon polyp, liver, and lung nodule respectively, at increasing pruning levels corresponding to UNet++L1 (circle 303) UNet++L2 (triangle 304), UNet++L3 (square 305), and UNet++L4 (pentagon 306).
Further depicted is the image segmentation performance of the UNet++ architecture after applying different levels of pruning. For representative experiments, UNet++ Li was utilized to denote the UNet++ architecture pruned at level i (see also, FIG. 1C). In each graph, the horizontal x-axis represents inference times 301, which is the time taken to process 10K test images using one NVIDIA TITAN X (Pascal) graphics card with 12 GB memory while the inference accuracy (inference mIoU (%)) 302 is presented via the vertical y-axis.
As depicted by graph (c), UNet++ L3, achieves on average a 32.2% reduction in inference time while degrading inference accuracy 302 by only 0.6 points. More aggressive pruning, seen in graph (d) further reduces the inference time but at the cost of significant accuracy degradation. Graph (d) UNet++ L4 exhibits the highest inference times 301 and a lower inference accuracy 302 when compared with graph (c). The lowest level of pruning is observed at graph (a) UNet++ L1 which has the highest inference accuracy 302 for all imaging segmentation.
Inference times 301 are generally observed to be proportionally constant for the various imaging segmentations, with the simpler images such as cell nuclei 303 and colon polyp 304 having lower inference times 301, and more complex images such as liver nodule 305 and lung nodule 306 having higher inference times 301. Cell nuclei 303 also resulted in lower inference accuracy 302 at various levels of pruning compared to other imaging segmentations.
UNet++ advantages: The suggested UNet++ architecture takes advantage of re-designed skip pathways and deep supervision, in which the re-designed skip pathways aim to reduce the semantic gap between the feature-maps of the encoder and decoder sub-networks, resulting in simpler optimization problems for the optimizer to solve. Deep supervision also enables more accurate segmentation, particularly for lesions that appear at multiple scales such as polyps in colonoscopy videos. Experiments demonstrate that the UNet++ architecture with deep supervision achieved an average IoU gain of 3.9 and 3.4 points over U-Net and wide U-Net, respectively.
Accurately segmenting organs and lesions is of great clinical significance in computer-aided diagnosis which may benefit from the application of U-Net and fully convolutional networks (FCNs) variants, and more particularly, use of skip connections. The skip architecture described herein combines deep, coarse, semantic information with shallow, fine, appearance information by encoding location and semantics in a nonlinear local-to-global pyramid. Skip connections therefore not only migrate context spatial information from layers in the contracting path to layers in the expanding path, but additionally help to overcome the vanishing gradient problem, resulting in outstanding performance when applied to medical image segmentation.
FIG. 4A depicts an exemplary architecture of UNet++ in accordance with described embodiments.
The skip connections described above may hide information gaps in the context of gradient flow, especially in the top skip connection, which connects the first layer in the contracting path and the last layer in the expanding path.
To mitigate information gaps with skip connections the intermediate layers are utilized which naturally form multiple new up-sampling expanding paths, leading to an ensemble of UNet{circumflex over ( )}e architectures (e.g., also referred to as an “Ensemble U-Net”) with a partially shared contracting path of different depths. Further still, nesting the UNet{circumflex over ( )}e architectures by densely connecting the intermediate layers at the same resolution across the UNet{circumflex over ( )}e architectures in the ensemble produces a new architecture as shown here, referred to herein as a UNet++ (e.g., nested ensemble nets). As a result, UNet++ naturally smoothes the gradient flows in skip connections. The UNet++ architecture utilizes more parameters, but offers several significant advantages, including: (1) learning a more precise output based on multi-scale information from input images, (2) strengthening multiple resolution features propagation, (3) encouraging feature-map reuse, (4) utilizing implicit multi-scale deep supervision, and (5) accelerating convergence speed.
As depicted here, the various boxes each correspond to a multi-channel feature-map. White boxes represent the result of copied feature-maps concatenated with any other same resolution feature-maps. The convolutional arrows 407 (solid arrows) with solid boxes seen in 404, 405 and 406 denote the basic bottleneck layer. The convolutional arrows 407 with shaded boxes denote the alternative convolutional operations, such as traditional conv-block (Conv-Block) 401, resolution block (Res-Block) 402, and dense block (Dense-Block) 403. The dashed downward arrows represent down-sampling 408, the solid upward arrows represent up-sampling 409, and the sideways (e.g., horizontal and curved) dashed arrows represent skip connections 410.
FIG. 4B depicts exemplary topologies of U-Net like, UNet+, and UNet++ architectures, in accordance with described embodiments.
Classical U-Net like architectures as seen in FIG. 4B, graph (a) are built from one single contracting path, extensive paths and d number of skip connections between matching size of feature-maps in contracting path and extensive path, where d is the number of downsampling within the network. The key component skip connections help the extensive path recover spatially detailed information from the contracting path, by reusing feature-maps. The darkness of the respective cells (e.g., the darkness of the shading) corresponds to different underlying quantities of downsampling operations.
Newly introduced intermediate layers in UNet+ depicted in FIG. 4B, graph (b) are only sequentially connected, whereas they are densely connected in UNet++, depicted in FIG. 4B, graph (c).
The model thus exploits the multiple-resolution feature-maps reuse by expanding all d number of level feature-maps to their own extensive path branch and then adding full skip connections among all the matching size feature-maps, resulting in a full ensemble of U-Nets.
Nested Ensemble Networks: To formulate the network's topological structures, consider a single image X that is passed through a d depth segmentation convolutional network to generate an output mask, in which
Figure US11164067-20211102-P00002
(·) is an up-sampling operation. As shown in FIG. 4B, aligning the contracting path to index i=0, and j indicates how many down-sampling operations are applied. The feature-map is defined from expanding path as Xi,j, where i,j∈[1, d−1]. Therefore, Xi,j in d−depth U-Net like architectures may be written as:
X i,j =X 0,j +u(X i−1,j+1), i=j
specifically, resulting in a standard U-Net when d=4. Accordingly, Xi,j in d−depth UNet+ can be written as:
X i,j =X i−1,j +u(X i−1,j+1), i≤j
and Xi,j in d−depth UNet++ may be written as:
X i,j =X 0,j +X 1,j + . . . +X i−1,j +u(X i−1,j+1), i≤j.
Two notable differences between the UNet+ and UNet++ like architectures networks may therefore be observed. Firstly, extensive path branches are added for all level of resolution feature-maps. Secondly, dense skip connections are made for all the matching feature-maps. In a horizontal view, each of the skip connection may be used as a densely connected feature-maps block which contains both long and short skip connections. Stated differently, in each horizontal direction, as much information as possible is information contained which then traverses through the dense connection, thus making better use of available multi-scaled information.
Backbone of the architecture: Convolutional blocks are utilized in both the contracting paths and expanding paths. With a single bottleneck layer (BN-ELU-1×1-Dropout-BN-ELU-3×3-Dropout) as
Figure US11164067-20211102-P00003
, different modified promising convolutional blocks are applied as feature extractors based on image classification results on ImageNet (e.g., Residual Unit and Dense Unit), to demonstrate the fully Nest-Net-like (e.g., UNet+ like architecture) family, extended as Fully Nest-ResNet, Fully-Nest-DenseNet. The nested structures were observed to outperform both the U-Net like architecture family, such as U-ResNet, U-DenseNet and the Nest-Net-like (e.g., UNet+ like) architecture family, such as Nest-ResNet, Nest-DenseNet. Given xl as the output of the lthlayer within one block, the same definition was followed as with previously known convolution blocks (Cony-Block) 401 methodologies, resolution blocks (Res-Block) 402 methodologies, and dense blocks (Dense-Block) 403 methodologies, as depicted at FIG. 4A, resulting in output xl=
Figure US11164067-20211102-P00003
(xl−1)+xl−1 for Res-Block 402 and xl=
Figure US11164067-20211102-P00003
([xl−1, xl−2, . . . , x0]) for Dense-Block 403. Each block follows up with a bottleneck layer
Figure US11164067-20211102-P00003
for a concatenated feature-maps transition from the previous block.
FIG. 5 depicts feature-maps for each of the (a) U-Net, (b) UNet+ and (c) UNet++ architectures, including intermediate layers based on the same input, with the outputs of each architecture compared to a truth, in accordance with described embodiments.
Depicted here are Feature Maps (FM) 1 to 4, each representing the expanding path of the (a) U-Net, (b) UNet+ and (c) UNet++ network architectures. As depicted here, all channels are averaged and the intermediate layers are intuitively visualized, in which the dotted arrows denote single skip connection 501 in U-Net and UNet+ architectures, while the dashed rows denote dense connections 502 in UNet++. Observing (a) U-Net feature-maps (e.g., Feature Maps 1 and 5), simply concatenating feature-maps between the one with rich row input information and the one with abstract feature information together causes network confusion and result in vanishing gradient. Based on the feature-maps changing from FM1 to FM 5 and segmentation performance, nesting different level feature-maps was observed to gradually help learn more accurate segmentation and grasp more information.
Intermediate feature-maps: Outputs of different lengths of paths were observed to be largely different in the various U-Net like architectures, as shown at FM 1 and FM 5. Directly adding first and last feature-maps as is done in previously known U-Net methodologies may confuse the network, whereas iteratively concatenating intermediate levels of feature-maps not only contains the information lost from upsampling and downsampling paths, but also provides reuse of the matching level feature information. In addition, deep supervision of the network is provided, by adding a 1×1 convolutional layer with softmax to each U-Net path, and applying back-propagation to minimize a multiple task object function.
Implementation Details: On all datasets, UNet++ has 4-depth nested contracting and expanding paths that each utilize the same type of convolution block (traditional convolution block, residual block or dense block). Following each depth of convolution block, one bottleneck layer is utilized as transition layer and a 1×1 average pooling layer. Experimental results depict an input patch of 96×96, however, the input patch may be any size of 2d×
Figure US11164067-20211102-P00004
, where
Figure US11164067-20211102-P00004
is an integer and d is the depth of the network. The dice coefficient and use of an early-stop was monitored on the validation set. To train the networks, weighted cross-entropy and dice coefficient were employed as the loss function, which is described in Equation 4 as:
L ( y , y ^ ) = - 1 N i = 1 N c = 1 C ( 1 2 · y i c · log y ^ i c + 2 · y i c · y ^ i c y i c + y ^ i c )
where ŷi c denotes the probability of pixel i belongs to class c (background or foreground), and yi c indicates the ground truth label for pixel i.
Resizing the input data from all database sources to 96×96 and scaling the intensity value into [0,1] for a fair experimental comparison, vertical/horizontal flips and random rotation degree (−10,10) and random scale (0.9,1.1) were then applied for data augmentation on all datasets.
Within a neural network with residual blocks, each layer feeds into the next layer and directly into layers multiple 2-3 hops away. Utilization of residual blocks within the neural network model can improve predictive performances on wide range of tasks.
While increasing a number of layers may improve the accuracy of results, there is nevertheless a practical limit on the overall number of layers that may be introduced to any neural network model. Certain problems such as vanishing gradients and the curse of dimensionality may arise with sufficiently deep networks, and the neural network may not be able to learn simple functions. Yet, simply increasing the number of layers within a neural network model will ultimately lead to accuracy that begins to saturate and then begin to degrade, popularly known as the degradation problem.
In certain ways, shallower networks perform better than the deeper counterparts that have few more layers added, and thus, it may be desirable in certain instances to skip layers. Through the use of skip-connections or residual connections it is therefore possible to skip training of certain layers. While the layers in a traditional network learn the true output. the layers in a residual network are learning the residual of a distribution, hence, the name.
FIG. 6 depicts validation loss and dice coefficient for different network structures along each epoch training stage, in accordance with described embodiments.
The number of epoch 601 is plotted against validation loss 602 (based on equation 4, above) and validation dice 603. The validation loss for UNet++ 604 is significantly smaller in magnitude and slope when compared to the validation loss for U-Net 606 and UNet+ 605 which has the sharpest validation loss. Validation loss for all three architectures converge around 12 Epochs and slowly decline, reaching a value around −0.87.
Validation dice 603 is a coefficient that measures similarity and overlap of images with a value of 1 indicating identical, fully overlapping images. The validation dice 603 for all three architectures merge around 10 epochs and plateau together at a validation dice of approximately 0.90 around 20 epochs. UNet+ Dice 608 has the sharpest initial slope and plateaus around 0.90 most quickly, followed by U-Net Dice 609 and UNet++ Dice 607 which has a sharp initial slope but then levels off slowly.
FIG. 7 is a depiction of UNet+ architecture in accordance with described embodiments.
The UNet+ architecture depicted here introduces intermediate layers via convolution blocks into skip connections between contracting 701 and expanding 702 paths of a general encoder-decoder network, which combines both long and short skip connections of the intermediate matching size feature-maps. Each dashed circle and solid circle is a convolution block corresponding to a convolution component Xi,j 707 where multi-channel feature-maps traverse. Bold arrows represent the down-sampling 704 and up-sampling 705 operations respectively within the encoder-decoder network, while the dashed arrows 706 represent skip connections, and the copied feature-maps merged with any other same resolution feature-maps via concatenation or summation. Encoder network 701 denotes generalized transfer learning which can either include an alternative feature generators backbone, such as VGG-16 and ResNet-101, or include a meta segmentation pipeline, for example, Mask RCNN.
FIG. 8 depicts visual comparisons between varying depth U-Net architectures ranging from 1-depth to 4-depth, in accordance with described embodiments.
Specifically, FIG. 8 depicts standard independent U-Net architectures of different depths via graphs (a)-(d), with each additional depth adding a row of intermediate layers comprising feature-maps.
FIG. 9 depicts further visual comparisons between UNet{circumflex over ( )}e and UNet+, in accordance with described embodiments.
At FIG. 9, Graph (e) depicts an ensemble of 4-different-depth U-Nets e.g., a UNet{circumflex over ( )}e) with a shared contracting path and graph (f) depicts the architecture of UNet+, which sequentially connects same resolution of feature-maps, without the long skip connections (arched arrows). Note that training the depicted nets in FIG. 8 and FIG. 9 utilizes explicit deep supervision, whereas deep supervision is an optional variant for training the nets depicted at FIG. 4A.
Differences between UNet++ and U-Net networks include the addition of expanding path branches for all levels of resolution feature-maps and the inclusion of dense skip connections for all the matching feature-maps.
The U-Net architecture and its variants as described herein provide a Fully Convolutional Network (FCN) capable of performing image segmentation, with the goal of predicting each pixel's class for all pixels of an input image. The U-Net architecture and its variants modify FCNs to yield better segmentation of images, especially with regard to use with medical imaging, with such modifications permitting U-Net architectures to operate symmetrically and to utilize skip connections between downsampling paths and upsampling paths by applying a concatenation operator rather than a sum. Such skip connections provide local information to the global information while upsampling and owing to the symmetry of the U-Net architecture variants, the resulting neural network has a large number of feature maps in the upsampling path, which enables greater transfer of information, whereas conventional FCN architectures are limited to only a number of classes feature maps in the upsampling path.
The U-Net architecture variants are generally separated in 3 parts, including the contracting/downsampling path, the bottleneck, and the expanding/upsampling path. The contracting/downsampling path operates to capture the context of the input image so as to facilitate image segmentation, resulting in coarse contextual information which is then transferred to the upsampling path by means of skip connections.
The bottleneck portion of the network resides between the contracting and expanding paths.
The expanding/upsampling path enables the neural network to perform precise localization combined with the contextual information transferred from the contracting/downsampling path by way of the skip connections.
In such a way, U-Net architectural variants combine the location information from the downsampling path with the contextual information in the upsampling path to obtain a general information combining localization and context, so as to improve predictive output rendered from a segmentation map.
Through the process of image segmentation, an appropriately configured system executing the neural network architectures as described herein operates to “partition” an input image different segments, each of them representing a different entity or element of interest within the input image. While medical imaging is somewhat complex, consider the more simplistic example of a self-driving vehicle, which seeks to separate, partition, or segment an input image into useable image components, such as differentiating vehicles and pedestrians from a background. Once the image is segmented, those identified image elements may then be utilized to perform some computational task, such as steering a vehicle. While medical imaging is more complex as the human physiology exhibits many nuances and image subtleties, which may be difficult for even trained physicians to differentiate, once the image elements are segmented and identified, various tasks may be performed, such as generating a predicted diagnosis based on the input image.
Thus, an appropriately configured machine or system, such as those described herein, through the execution of improved neural networks may therefore operate to identify and predict with greater accuracy, the elements of concern within an input image within the context of medical imaging.
FIG. 10 is a visualization of intermediate feature-maps concatenation and flow along the top components of a nested network, in accordance with described embodiments.
Directly summing up or concatenating first and last feature-maps as proposed in U-Net may confuse the network. Therefore, iteratively concatenating intermediate levels of feature-maps not only operates to contain the information lost from up-sampling and down-sampling paths, but also permits reuse of the matching level feature information. The nested network has the ability to smooth the information flow compared to U-Net architecture, which concatenates map A and map B together directly.
Map A (depicted at top left) represents the feature-map for the input image that goes through only one convolutional block, while map B (depicted at lower right) represents the feature-map for the input image which goes through a sequence of convolutional blocks including 4 downsampling and 4 upsampling blocks. Therefore, notwithstanding each having a same resolution, the information produced is asymmetrical. The depth d=1 condition is further shown; this phenomenon exists in other depth conditions, in which the larger the depth the more asymmetric the feature-map becomes.
Use of visualizing depth d=0 for feature-maps is further depicted, in which the standard U-Net architecture concatenates feature-map A and B together following up with a successive convolution layer. However, when the network becomes deeper, the information gap between feature-maps A and B will be enlarged and ultimately influence the gradient flow between them. To address this problem, intermediate layers and skip connections between them are introduced. Output for different path lengths were observed to be largely different in encoder-decoder architectures (see FIG. 3) with the path between X0,0 and X0,4.
The visualization of intermediate feature-maps concatenation and flow along the top components of nested network as depicted here is performed at depth d=0. Map A is the feature-map for the input image that goes through only one convolutional block, while map B is the feature-map for the input image that goes through a sequence of convolutional blocks including four downsampling and four upsampling blocks, resulting in asymmetrical feature maps, despite the same resolution input.
Deep supervision: Further depicted here is a deep supervision mechanism is introduced during the learning process to combat potential optimization difficulties, thus the model shown here attains a much faster convergence rate and more powerful discrimination capability. Advantages include accelerating the optimization convergence rate and further include improved prediction accuracy. During the testing phase, the network structure is pruned according to validation performance, achieving a balance between accuracy and speed.
Vanishing gradients can make the loss back-propagation ineffective in early layers, which may be even more severe with the use of 3D situations and may slow convergence rates and reduce discrimination capabilities of the model. So as to negate such potential issues, additional supervision is injected into the hidden layers to counteract the adverse effects of gradient vanishing.
Additionally, a lack of deep supervision may result in the features produced at hidden layers that less semantically meaningful. So as to negate any potential issues, certain embodiments utilize deep supervision layers to improve the learning ability of the hidden layers.
FIG. 11A depicts examples of cell segmentation intermediate feature-maps and predictions of U-Net, UNet+ and UNet++ without deep supervision, in accordance with described embodiments.
The dotted arrows denote single skip connections 1101 in U-Net architecture (a) and UNet+ architecture (b) the dashed arrows 1102 denote dense connections in UNet++ architecture (c).
Intermediate feature-maps visualization and analysis: Output for direct paths under U-Net like architecture is largely different than other architectures, as shown by U-Net architecture (a) at Feature-Maps FM 1 and FM 5. Directly adding the first and last feature-maps as proposed in U-Net may confuse the network, whereas iteratively concatenating intermediate level feature-maps, as seen in UNet+ architecture (b) and UNet++ architecture (c), respectively, not only contains the information normally lost from upsampling and downsampling paths, but also allows for reuse of matching level feature information. This is evidenced by progressively higher IoU and Dice values for UNet+ and UNet++ as compared to U-Net, seen when comparing the respective values in UNet+ architecture (b) and UNet++ architecture (c) to that of the U-Net architecture (a).
FIG. 11B depicts examples of cell segmentation intermediate feature-maps and predictions of UNet+ and UNet++ with deep supervision, in accordance with described embodiments.
The dotted arrows denote single skip connections 1103 in UNet+ architecture (a) and the dashed arrows 1104 denote dense connections in UNet++ architecture (b).
Deep supervision of the networks is introduced here by adding a 1×1 sigmoid convolutional layer to each U-Net path, and applying back-propagation to minimize a multiple task object function. As a result, slightly increased IoU scores (0.8892 vs. 0.8843) and similar dice scores (0.9505 vs. 0.9560) are observed for UNet++ (b) with deep supervision when compared to UNet++ without deep supervision as shown at UNet++ architecture (c) of FIG. 11A. When comparing UNet+ architecture (a) with deep supervision in FIG. 11B to UNet+ architecture (b) without deep supervision in FIG. 11A, slightly decreased IoU and dice scores are observed for UNet+ with deep supervision (decreasing from an IoU score of 0.8778 to 0.8036 and dice score of 0.9560 to 0.9505 when deep supervision is added to UNet+).
FIG. 12 depicts a flow diagram illustrating a method 1200 for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging, in accordance with disclosed embodiments. Method 1200 may be performed by processing logic that may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device) to perform various operations such as designing, defining, retrieving, parsing, persisting, exposing, loading, executing, operating, receiving, generating, storing, maintaining, creating, returning, presenting, interfacing, communicating, transmitting, querying, processing, providing, determining, triggering, displaying, updating, sending, etc., in pursuance of the systems and methods as described herein. For example, the system 1301 (see FIG. 13) and the machine 1401 (see FIG. 14) and the other supporting systems and components as described herein may implement the described methodologies. Some of the blocks and/or operations listed below are optional in accordance with certain embodiments. The numbering of the blocks presented is for the sake of clarity and is not intended to prescribe an order of operations in which the various blocks must occur.
With reference to the method 1200 depicted at FIG. 12 beginning at block 1205, there is a method performed by a system having at least a processor and a memory therein to execute instructions which cause the system to perform the following operations:
At block 1210, processing logic executes, via the processor, a neural network model stored within the memory of the system, in which the neural network model is formed from a plurality of layer blocks including an encoder layer block which precedes a plurality of decoder layer blocks.
At block 1215, processing logic associates a resolution value with each of the plurality of layer blocks defining a resolution of the layer block input to be processed by the layer block to generate the layer block output.
At block 1220, processing logic processes, via the encoder layer block, a respective layer block input including a down-sampled layer block output from a higher resolution layer block associated with a resolution value greater than the resolution value for the encoder layer block.
At block 1225, processing logic processes, via each of the plurality of decoder layer blocks, a respective layer block input including both (i) an up-sampled layer block output from a lower resolution layer block associated with a resolution value less than the resolution value for the respective decoder layer block and (ii) a layer block output of a previous layer block associated with a prior resolution value of a layer block which precedes the respective decoder layer block.
At block 1230, processing logic generates the respective layer block output from the processing via the encoder layer block and the plurality of decoder layer blocks by summing or concatenating the processed layer block inputs.
According to another embodiment of method 1200, each of the plurality of layer blocks include one or more neural network layers configured to process a respective layer block input and responsively generate a corresponding layer block output.
According to another embodiment of method 1200, generating the respective layer block output comprises concatenating the processed layer block inputs via a concatenation layer that fuses the respective layer block output from the previous layer block associated with the prior resolution value within a common dense block with a corresponding up-sampled layer block output from a lower dense block.
According to another embodiment, method 1200 further includes: outputting feature-maps from the processing via the encoder layer block; and receiving the output feature-maps from the encoder layer block for processing via the plurality of decoder layer blocks via skip pathways between nodes within the neural network model.
According to another embodiment, method 1200 further includes: ordering the plurality of layer blocks of the neural network to position the encoder layer block prior to the plurality of decoder layer blocks; and ordering each of the plurality of decoder layer blocks of different resolution values based on the associated resolution value for each decoder layer block.
According to another embodiment of method 1200, processing, via each of the plurality of decoder layer blocks further comprises each decoder layer block processing layer block outputs from every previous layer block associated with a resolution value preceding the resolution value associated with the respective decoder layer block performing the processing based on the ordering.
According to another embodiment, method 1200 further includes: generating, from each of the plurality of decoder layer blocks associated with a highest resolution value, a layer block output defining a decoder output of the neural network model; and generating a final output from the neural network model using the defined decoder outputs generated by the plurality of decoder layer blocks, each associated with the highest resolution value.
According to another embodiment of method 1200, generating the final output from the neural network model using the defined decoder outputs generated by the plurality of decoder layer blocks, each associated with the highest resolution value comprises one of: averaging the defined decoder outputs to generate the final output from the neural network model; or alternatively selecting one of the defined decoder outputs generated by the plurality of decoder layer blocks based on configurable implementation parameters of the neural network model.
According to another embodiment, method 1200 further includes: training the neural network model using machine learning training to optimize an objective function; and wherein the objective function comprises a respective loss term corresponding to layer block outputs from each of the plurality of decoder layer blocks of the executing neural network model.
According to another embodiment of method 1200, each layer block output from each of the plurality of decoder layer blocks of the executing neural network model is a respective estimate of a segmentation map of an input image processed by the neural network model; and wherein the respective loss term corresponding to each layer block output from each of the plurality of decoder layer blocks of the executing neural network model comprises a cross-entropy term and a dice coefficient term.
According to another embodiment of method 1200, the method further comprises: receiving the input image for image segmentation processing at the system via the executing neural network model; and wherein the input image comprises a medical image.
According to another embodiment of method 1200, one or more of the plurality of layer blocks include a plurality of convolutional layers; wherein each of the plurality of convolutional layers is configured to process a layer input comprising the outputs of each previous convolutional layer in the layer block; and wherein a different one or more of the plurality of layer blocks are residual blocks.
According to a particular embodiment, there is a non-transitory computer readable storage medium having instructions stored thereupon that, when executed by a system having at least a processor and a memory therein, the instructions cause the processor to perform operations including: executing, via the processor, a neural network model stored within the memory of the system; wherein the neural network model is formed from a plurality of layer blocks including an encoder layer block which precedes a plurality of decoder layer blocks; associating a resolution value with each of the plurality of layer blocks defining a resolution of the layer block input to be processed by the layer block to generate the layer block output; processing, via the encoder layer block, a respective layer block input including a down-sampled layer block output from a higher resolution layer block associated with a resolution value greater than the resolution value for the encoder layer block; processing, via each of the plurality of decoder layer blocks, a respective layer block input including both (i) an up-sampled layer block output from a lower resolution layer block associated with a resolution value less than the resolution value for the respective decoder layer block and (ii) a layer block output of a previous layer block associated with a prior resolution value of a layer block which precedes the respective decoder layer block; and generating the respective layer block output from the processing via the encoder layer block and the plurality of decoder layer blocks by summing or concatenating the processed layer block inputs.
FIG. 13 shows a diagrammatic representation of a system 1301 within which embodiments may operate, be installed, integrated, or configured. In accordance with one embodiment, there is a system 1301 having at least a processor 1390 and a memory 1395 therein to execute implementing application code 1396. Such a system 1301 may communicatively interface with and cooperatively execute with the benefit of remote systems, such as a user device sending instructions and data, a user device to receive segmented image 1343 output from the model output manager 1385 of the system, or systems within a networked or within a client-server environment, etc.
According to the depicted embodiment, the system 1301, includes the processor 1390 and the memory 1395 to execute instructions at the system 1301 and operates to implement a multi-resolution neural network for use with imaging intensive applications including medical imaging. According to such an embodiment, the system 1301 further includes the processor 1390 to execute a neural network model 1365 stored within the memory 1395 of the system 1301, in which the neural network model 1365 is formed from a plurality of layer blocks including an encoder layer block 1368 which precedes a plurality of decoder layer blocks 1370. Within such a system, the processor is further to execute an image resolution manager 1350 to associate a resolution value with each of the plurality of layer blocks defining a resolution of the layer block input to be processed by the layer block to generate the layer block output. The processor 1390 of the system 1301 is further to execute the encoder layer block 1368 to process a respective layer block input including a down-sampled layer block 1340 output from a higher resolution layer block associated with a resolution value greater than the resolution value for the encoder layer block. The processor is to further execute each of the plurality of decoder layer blocks 1370 to process a respective layer block input including both (i) an up-sampled layer block 1341 output from a lower resolution layer block associated with a resolution value less than the resolution value for the respective decoder layer block and (ii) a layer block output of a previous layer block 1342 associated with a prior resolution value of a layer block which precedes the respective decoder layer block. According to such an embodiment of the system 1301, the processor 1390 is further to execute a model output manager 1385 to generate the respective layer block output from the processing via the encoder layer block and the plurality of decoder layer blocks by summing or concatenating the processed layer block inputs.
The model output manager 1385 may further transmit output back to a user device or other requestor, for example, via the user interface 1326, including sending a segmented image 1343 representation of the input image 1339 (e.g., a medical image) to a user device or other requestor, or such information may alternatively be stored within the database system storage 1345 of the system 1301.
According to another embodiment of the system 1301, a user interface 1326 communicably interfaces with a user client device remote from the system and communicatively interfaces with the system via a public Internet.
Bus 1316 interfaces the various components of the system 1301 amongst each other, with any other peripheral(s) of the system 1301, and with external components such as external network elements, other machines, client devices, cloud computing services, etc. Communications may further include communicating with external devices via a network interface over a LAN, WAN, or the public Internet.
FIG. 14 illustrates a diagrammatic representation of a machine 1401 in the exemplary form of a computer system, in accordance with one embodiment, within which a set of instructions, for causing the machine/computer system 1401 to perform any one or more of the methodologies discussed herein, may be executed. In alternative embodiments, the machine may be connected (e.g., networked) to other machines in a Local Area Network (LAN), an intranet, an extranet, or the public Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, as a server or series of servers within an on-demand service environment. Certain embodiments of the machine may be in the form of a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, computing system, or any machine capable of executing a set of instructions (sequential or otherwise) that specify and mandate the specifically configured actions to be taken by that machine pursuant to stored instructions. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines (e.g., computers) that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.
The exemplary computer system 1401 includes a processor 1402, a main memory 1404 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc., static memory such as flash memory, static random access memory (SRAM), volatile but high-data rate RAM, etc.), and a secondary memory 1418 (e.g., a persistent storage device including hard disk drives and a persistent database and/or a multi-tenant database implementation), which communicate with each other via a bus 1430. Main memory 1404 includes a neural network model 1424 and an image resolution manager 1423 and a model output manager 1425 which operate to implement a multi-resolution neural network for use with imaging intensive applications including medical imaging in support of the methodologies and techniques described herein. Main memory 1404 and its sub-elements are further operable in conjunction with processing logic 1426 and processor 1402 to perform the methodologies discussed herein.
Processor 1402 represents one or more specialized and specifically configured processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processor 1402 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processor 1402 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processor 1402 is configured to execute the processing logic 1426 for performing the operations and functionality which is discussed herein.
The computer system 1401 may further include a network interface card 1408. The computer system 1401 also may include a user interface 1410 (such as a video display unit, a liquid crystal display, etc.), an alphanumeric input device 1412 (e.g., a keyboard), a cursor control device 1414 (e.g., a mouse), and a signal generation device 1416 (e.g., an integrated speaker). The computer system 1401 may further include peripheral device 1436 (e.g., wireless or wired communication devices, memory devices, storage devices, audio processing devices, video processing devices, etc.).
The secondary memory 1418 may include a non-transitory machine-readable storage medium or a non-transitory computer readable storage medium or a non-transitory machine-accessible storage medium 1431 on which is stored one or more sets of instructions (e.g., software 1422) embodying any one or more of the methodologies or functions described herein. The software 1422 may also reside, completely or at least partially, within the main memory 1404 and/or within the processor 1402 during execution thereof by the computer system 1401, the main memory 1404 and the processor 1402 also constituting machine-readable storage media. The software 1422 may further be transmitted or received over a network 1420 via the network interface card 1408.
While the subject matter disclosed herein has been described by way of example and in terms of the specific embodiments, it is to be understood that the claimed embodiments are not limited to the explicitly enumerated embodiments disclosed. To the contrary, the disclosure is intended to cover various modifications and similar arrangements as are apparent to those skilled in the art. Therefore, the scope of the appended claims are to be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements. It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the disclosed subject matter is therefore to be determined in reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims (20)

What is claimed is:
1. A method performed by a system having at least a processor and a memory therein to execute instructions, wherein the method comprises:
executing, via the processor, a neural network model stored within the memory of the system;
wherein the neural network model is formed from a plurality of layer blocks including an encoder layer block which precedes a plurality of decoder layer blocks;
associating a resolution value with each of the plurality of layer blocks defining a resolution of the layer block input to be processed by the layer block to generate the layer block output;
processing, via the encoder layer block, a respective layer block input including a down-sampled layer block output from a higher resolution layer block associated with a resolution value greater than the resolution value for the encoder layer block;
processing, via each of the plurality of decoder layer blocks, a respective layer block input including both (i) an up-sampled layer block output from a lower resolution layer block associated with a resolution value less than the resolution value for the respective decoder layer block and (ii) a layer block output of a previous layer block associated with a prior resolution value of a layer block which precedes the respective decoder layer block; and
generating the respective layer block output from the processing via the encoder layer block and the plurality of decoder layer blocks by summing or concatenating the processed layer block inputs.
2. The method of claim 1, wherein each of the plurality of layer blocks include one or more neural network layers configured to process a respective layer block input and responsively generate a corresponding layer block output.
3. The method of claim 1, wherein generating the respective layer block output comprises concatenating the processed layer block inputs via a concatenation layer that fuses the respective layer block output from the previous layer block associated with the prior resolution value within a common dense block with a corresponding up-sampled layer block output from a lower dense block.
4. The method of claim 1, further comprising:
outputting feature-maps from the processing via the encoder layer block; and
receiving the output feature-maps from the encoder layer block for processing via the plurality of decoder layer blocks via skip pathways between nodes within the neural network model.
5. The method of claim 1, further comprising:
ordering the plurality of layer blocks of the neural network to position the encoder layer block prior to the plurality of decoder layer blocks; and
ordering each of the plurality of decoder layer blocks of different resolution values based on the associated resolution value for each decoder layer block.
6. The method of claim 5, wherein processing, via each of the plurality of decoder layer blocks further comprises each decoder layer block processing layer block outputs from every previous layer block associated with a resolution value preceding the resolution value associated with the respective decoder layer block performing the processing based on the ordering.
7. The method of claim 1, further comprising:
generating, from each of the plurality of decoder layer blocks associated with a highest resolution value, a layer block output defining a decoder output of the neural network model; and
generating a final output from the neural network model using the defined decoder outputs generated by the plurality of decoder layer blocks, each associated with the highest resolution value.
8. The method of claim 7, wherein generating the final output from the neural network model using the defined decoder outputs generated by the plurality of decoder layer blocks, each associated with the highest resolution value comprises one of:
averaging the defined decoder outputs to generate the final output from the neural network model; or alternatively
selecting one of the defined decoder outputs generated by the plurality of decoder layer blocks based on configurable implementation parameters of the neural network model.
9. The method of claim 1, further comprising:
training the neural network model using machine learning training to optimize an objective function; and
wherein the objective function comprises a respective loss term corresponding to layer block outputs from each of the plurality of decoder layer blocks of the executing neural network model.
10. The method of claim 9:
wherein each layer block output from each of the plurality of decoder layer blocks of the executing neural network model is a respective estimate of a segmentation map of an input image processed by the neural network model; and
wherein the respective loss term corresponding to each layer block output from each of the plurality of decoder layer blocks of the executing neural network model comprises a cross-entropy term and a dice coefficient term.
11. The method of claim 10, wherein the method further comprises:
receiving the input image for image segmentation processing at the system via the executing neural network model; and
wherein the input image comprises a medical image.
12. The method of claim 1:
wherein one or more of the plurality of layer blocks include a plurality of convolutional layers;
wherein each of the plurality of convolutional layers is configured to process a layer input comprising the outputs of each previous convolutional layer in the layer block; and
wherein a different one or more of the plurality of layer blocks are residual blocks.
13. A system comprising:
a memory to store instructions;
a processor to execute instructions stored within the memory;
the processor to execute a neural network model stored within the memory of the system;
wherein the neural network model is formed from a plurality of layer blocks including an encoder layer block which precedes a plurality of decoder layer blocks;
the processor to execute an image resolution manager to associate a resolution value with each of the plurality of layer blocks defining a resolution of the layer block input to be processed by the layer block to generate the layer block output;
the processor to execute the encoder layer block to process a respective layer block input including a down-sampled layer block output from a higher resolution layer block associated with a resolution value greater than the resolution value for the encoder layer block;
the processor to execute each of the plurality of decoder layer blocks to process a respective layer block input including both (i) an up-sampled layer block output from a lower resolution layer block associated with a resolution value less than the resolution value for the respective decoder layer block and (ii) a layer block output of a previous layer block associated with a prior resolution value of a layer block which precedes the respective decoder layer block; and
the processor to execute a model output manager to generate the respective layer block output from the processing via the encoder layer block and the plurality of decoder layer blocks by summing or concatenating the processed layer block inputs.
14. The system of claim 13, wherein each of the plurality of layer blocks include one or more neural network layers configured to process a respective layer block input and responsively generate a corresponding layer block output.
15. The system of claim 13, further comprising:
the processor to order the plurality of layer blocks of the neural network to position the encoder layer block prior to the plurality of decoder layer blocks;
the processor to order each of the plurality of decoder layer blocks of different resolution values based on the associated resolution value for each decoder layer block; and
wherein the processor to execute each of the plurality of decoder layer blocks to process layer block outputs from every previous layer block associated with a resolution value preceding the resolution value associated with the respective decoder layer block performing the processing based on the ordering.
16. The system of claim 13:
wherein the processor is to generate from each of the plurality of decoder layer blocks associated with a highest resolution value, a layer block output defining a decoder output of the neural network model;
wherein the processor is to generate a final output from the neural network model using the defined decoder outputs generated by the plurality of decoder layer blocks, each associated with the highest resolution value;
wherein the model output manager is to generate the final output based on: (i) the processor averaging the defined decoder outputs to generate the final output from the neural network model or (ii) the processor selecting one of the defined decoder outputs generated by the plurality of decoder layer blocks based on configurable implementation parameters of the neural network model.
17. Non-transitory computer readable storage media having instructions stored thereupon that, when executed by a system having at least a processor and a memory therein, the instructions cause the processor to perform operations including:
executing, via the processor, a neural network model stored within the memory of the system;
wherein the neural network model is formed from a plurality of layer blocks including an encoder layer block which precedes a plurality of decoder layer blocks;
associating a resolution value with each of the plurality of layer blocks defining a resolution of the layer block input to be processed by the layer block to generate the layer block output;
processing, via the encoder layer block, a respective layer block input including a down-sampled layer block output from a higher resolution layer block associated with a resolution value greater than the resolution value for the encoder layer block;
processing, via each of the plurality of decoder layer blocks, a respective layer block input including both (i) an up-sampled layer block output from a lower resolution layer block associated with a resolution value less than the resolution value for the respective decoder layer block and (ii) a layer block output of a previous layer block associated with a prior resolution value of a layer block which precedes the respective decoder layer block; and
generating the respective layer block output from the processing via the encoder layer block and the plurality of decoder layer blocks by summing or concatenating the processed layer block inputs.
18. The non-transitory computer readable storage media of claim 17, wherein the instructions cause the system to perform operations further comprising:
outputting feature-maps from the processing via the encoder layer block; and
receiving the output feature-maps from the encoder layer block for processing via the plurality of decoder layer blocks via skip pathways between nodes within the neural network model.
19. The non-transitory computer readable storage media of claim 17, wherein the instructions cause the system to perform operations further comprising:
ordering the plurality of layer blocks of the neural network to position the encoder layer block prior to the plurality of decoder layer blocks;
ordering each of the plurality of decoder layer blocks of different resolution values based on the associated resolution value for each decoder layer block; and
wherein processing, via each of the plurality of decoder layer blocks further comprises each decoder layer block processing layer block outputs from every previous layer block associated with a resolution value preceding the resolution value associated with the respective decoder layer block performing the processing based on the ordering.
20. The non-transitory computer readable storage media of claim 17, wherein the instructions cause the system to perform operations further comprising:
generating, from each of the plurality of decoder layer blocks associated with a highest resolution value, a layer block output defining a decoder output of the neural network model;
generating a final output from the neural network model using the defined decoder outputs generated by the plurality of decoder layer blocks, each associated with the highest resolution value; and
wherein generating the final output from the neural network model using the defined decoder outputs generated by the plurality of decoder layer blocks, each associated with the highest resolution value comprises one of: (i) averaging the defined decoder outputs to generate the final output from the neural network model or alternatively (ii) selecting one of the defined decoder outputs generated by the plurality of decoder layer blocks based on configurable implementation parameters of the neural network model.
US16/556,130 2018-08-29 2019-08-29 Systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging Active US11164067B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US201862724089P true 2018-08-29 2018-08-29
US16/556,130 US11164067B2 (en) 2018-08-29 2019-08-29 Systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US16/556,130 US11164067B2 (en) 2018-08-29 2019-08-29 Systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging

Publications (2)

Publication Number Publication Date
US20200074271A1 US20200074271A1 (en) 2020-03-05
US11164067B2 true US11164067B2 (en) 2021-11-02

Family

ID=69641419

Family Applications (1)

Application Number Title Priority Date Filing Date
US16/556,130 Active US11164067B2 (en) 2018-08-29 2019-08-29 Systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging

Country Status (1)

Country Link
US (1) US11164067B2 (en)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10956785B2 (en) 2018-04-27 2021-03-23 Arizona Board Of Regents On Behalf Of Arizona State University Methods, systems, and media for selecting candidates for annotation for use in training classifiers
US11100685B2 (en) 2018-08-29 2021-08-24 Arizona Board Of Regents On Behalf Of Arizona State University Method and apparatus for detection and visualization of pulmonary embolism
US20200137380A1 (en) * 2018-10-31 2020-04-30 Intel Corporation Multi-plane display image synthesis mechanism
EP3651117B1 (en) * 2018-11-09 2021-02-17 Oxipit, UAB Methods, systems and use for detecting irregularities in medical images by means of a machine learning model
CN111652886B (en) * 2020-05-06 2022-07-22 哈尔滨工业大学 Liver tumor segmentation method based on improved U-net network
CN111882514A (en) * 2020-07-27 2020-11-03 中北大学 Multi-modal medical image fusion method based on double-residual ultra-dense network
CN112200029B (en) * 2020-09-27 2022-03-25 电子科技大学 Remote sensing image building extraction method based on improved UNet + + network
CN112258488A (en) * 2020-10-29 2021-01-22 山西三友和智慧信息技术股份有限公司 Medical image focus segmentation method
CN112508013B (en) * 2020-12-02 2021-07-16 哈尔滨市科佳通用机电股份有限公司 Lock catch loss fault detection method, system and device
CN112598656A (en) * 2020-12-28 2021-04-02 长春工业大学 Brain tumor segmentation algorithm based on UNet + + optimization and weight budget
CN113256500B (en) * 2021-07-02 2021-10-01 北京大学第三医院(北京大学第三临床医学院) Deep learning neural network model system for multi-modal image synthesis
CN114004836B (en) * 2022-01-04 2022-04-01 中科曙光南京研究院有限公司 Self-adaptive biomedical image segmentation method based on deep learning
CN114170249B (en) * 2022-02-08 2022-04-19 湖南大学 Image semantic segmentation method based on CNUNet3+ network

Citations (66)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012109670A1 (en) 2011-02-11 2012-08-16 Arizona Board Of Regents, For And On Behalf Of, Arizona State University Systems, methods, and media for detecting an anatomical object in a medical device image
WO2012109676A1 (en) 2011-02-11 2012-08-16 Arizona Board Of Regents For And On Behalf Of Arizona State University Methods, systems, and media for determining carotid intima-media thickness
WO2013116865A1 (en) 2012-02-02 2013-08-08 Arizona Board Of Regents, For And On Behalf Of, Arizona State University Systems, methods, and media for updating a classifier
WO2013116867A1 (en) 2012-02-03 2013-08-08 Arizona Board Of Regents, For And On Behalf Of, Arizona State University Systems, methods, and media for monitoring the condition of a patient's heart
WO2015031641A1 (en) 2013-08-29 2015-03-05 Mayo Foundation For Medical Education And Research System and method for boundary classification and automatic polyp detection
WO2015113014A1 (en) 2014-01-27 2015-07-30 Arizona Board Of Regents On Behalf Of Arizona State University Diagnosing pulmonary embolism by integrating patient-level diagnosis and embolus-level detection
WO2015142808A1 (en) 2014-03-17 2015-09-24 Arizona Board Of Regents On Behalf Of Arizona State University System and method for measuring artery thickness using ultrasound imaging
WO2015164768A1 (en) 2014-04-24 2015-10-29 Arizona Board Of Regents On Behalf Of Arizona State University System and method for detecting polyps from learned boundaries
WO2015164724A1 (en) 2014-04-24 2015-10-29 Arizona Board Of Regents On Behalf Of Arizona State University System and method for quality assessment of optical colonoscopy images
US9330336B2 (en) 2011-09-16 2016-05-03 Arizona Board of Regents, a body corporate of the State of Arizona, acting for and on behalf of, Arizona State University Systems, methods, and media for on-line boosting of a classifier
US9449381B2 (en) 2012-09-10 2016-09-20 Arizona Board Of Regents, A Body Corporate Of The State Of Arizona, Acting For And On Behalf Of Arizona State University Methods, systems, and media for generating and analyzing medical images having elongated structures
WO2016161115A1 (en) 2015-03-31 2016-10-06 Mayo Foundation For Medical Education And Research System and methods for automatic polyp detection using convolutional neural networks
WO2017027475A1 (en) 2015-08-07 2017-02-16 Jianming Liang Methods, systems, and media for simultaneously monitoring colonoscopic video quality and detecting polyps in colonoscopy
US9700213B2 (en) 2014-09-12 2017-07-11 Mayo Foundation For Medical Education And Research System and method for automatic polyp detection using global geometric constraints and local intensity variation patterns
US9710916B2 (en) 2013-03-14 2017-07-18 Arizona Board Of Regents, A Body Corporate Of The State Of Arizona, Acting For And On Behalf Of Arizona State University Kernel sparse models for automated tumor segmentation
US9924927B2 (en) 2016-02-22 2018-03-27 Arizona Board Of Regents On Behalf Of Arizona State University Method and apparatus for video interpretation of carotid intima-media thickness
US20180101957A1 (en) * 2016-10-06 2018-04-12 Qualcomm Incorporated Neural network for image processing
US9959615B2 (en) 2015-07-01 2018-05-01 Arizona Board Of Regents On Behalf Of Arizona State University System and method for automatic pulmonary embolism detection
US20180177461A1 (en) * 2016-12-22 2018-06-28 The Johns Hopkins University Machine learning approach to beamforming
US20180218497A1 (en) * 2017-01-27 2018-08-02 Arterys Inc. Automated segmentation utilizing fully convolutional networks
US10045728B2 (en) 2013-10-07 2018-08-14 Arizona Board Of Regents On Behalf Of Arizona State University Kidney glomeruli measurement systems and methods
US20180259608A1 (en) * 2015-11-29 2018-09-13 Arterys Inc. Automated cardiac volume segmentation
US20180268220A1 (en) * 2017-03-17 2018-09-20 Magic Leap, Inc. Room layout estimation methods and techniques
US10096125B1 (en) * 2017-04-07 2018-10-09 Adobe Systems Incorporated Forecasting multiple poses based on a graphical image
US20180314943A1 (en) 2017-04-27 2018-11-01 Jianming Liang Systems, methods, and/or media, for selecting candidates for annotation for use in training a classifier
US10157467B2 (en) 2015-08-07 2018-12-18 Arizona Board Of Regents On Behalf Of Arizona State University System and method for detecting central pulmonary embolism in CT pulmonary angiography images
US20190005670A1 (en) * 2017-06-28 2019-01-03 Magic Leap, Inc. Method and system for performing simultaneous localization and mapping using convolutional image transformation
US20190035113A1 (en) * 2017-07-27 2019-01-31 Nvidia Corporation Temporally stable data reconstruction with an external recurrent neural network
US20190049540A1 (en) * 2017-08-10 2019-02-14 Siemens Healthcare Gmbh Image standardization using generative adversarial networks
US20190064378A1 (en) * 2017-08-25 2019-02-28 Wei D. LIU Automated Seismic Interpretation Using Fully Convolutional Neural Networks
US20190108634A1 (en) * 2017-10-09 2019-04-11 The Board Of Trustees Of The Leland Stanford Junior University Contrast Dose Reduction for Medical Imaging Using Deep Learning
US20190139216A1 (en) * 2017-11-03 2019-05-09 Siemens Healthcare Gmbh Medical Image Object Detection with Dense Feature Pyramid Network Architecture in Machine Learning
US20190142519A1 (en) * 2017-08-15 2019-05-16 Holo Surgical Inc. Graphical user interface for displaying automatically segmented individual parts of anatomy in a surgical navigation system
US20190147255A1 (en) * 2017-11-15 2019-05-16 Uber Technologies, Inc. Systems and Methods for Generating Sparse Geographic Data for Autonomous Vehicles
US20190155973A1 (en) * 2017-11-02 2019-05-23 Airworks Solutions, Inc. Methods and apparatus for automatically defining computer-aided design files using machine learning, image analytics, and/or computer vision
US10328282B2 (en) 2015-07-13 2019-06-25 Mayo Foundation For Medical Education And Research System and method for novel chance-constrained optimization in intensity-modulated proton therapy planning to account for range and patient setup uncertainties
US20190213779A1 (en) * 2018-01-10 2019-07-11 The Board Of Trustees Of The University Of Illinois Apparatus and method for producing three-dimensional models from magnetic resonance imaging
US20190223725A1 (en) * 2018-01-25 2019-07-25 Siemens Healthcare Gmbh Machine Learning-based Segmentation for Cardiac Medical Imaging
US20190261945A1 (en) * 2018-02-26 2019-08-29 Siemens Medical Solutions Usa, Inc. Three-Dimensional Segmentation from Two-Dimensional Intracardiac Echocardiography Imaging
US20190295223A1 (en) * 2018-03-22 2019-09-26 Adobe Inc. Aesthetics-guided image enhancement
US20190311202A1 (en) * 2018-04-10 2019-10-10 Adobe Inc. Video object segmentation by reference-guided mask propagation
US20190332896A1 (en) 2018-04-27 2019-10-31 Jianming Liang Methods, systems, and media for selecting candidates for annotation for use in training classifiers
US20190340810A1 (en) * 2018-05-03 2019-11-07 Adobe Inc. Utilizing an object relighting neural network to generate digital images illuminated from a target lighting direction
US20190340152A1 (en) * 2018-05-04 2019-11-07 Cornami Inc. Reconfigurable reduced instruction set computer processor architecture with fractured cores
US20190369190A1 (en) * 2018-06-04 2019-12-05 Korea Advanced Institute Of Science And Technology Method for processing interior computed tomography image using artificial neural network and apparatus therefor
US10521927B2 (en) * 2017-08-15 2019-12-31 Siemens Healthcare Gmbh Internal body marker prediction from surface data in medical imaging
US20200005511A1 (en) * 2018-06-28 2020-01-02 Adobe Inc. Determining Image Handle Locations
US20200020098A1 (en) * 2017-04-19 2020-01-16 Siemens Healthcare Gmbh Target detection in latent space
US20200041612A1 (en) * 2018-08-02 2020-02-06 Metawave Corporation Recurrent super-resolution radar for autonomous vehicles
US20200041597A1 (en) * 2018-08-01 2020-02-06 Siemens Healthcare Gmbh Magnetic Resonance Fingerprinting Image Reconstruction and Tissue Parameter Estimation
US20200065969A1 (en) * 2018-08-27 2020-02-27 Siemens Healthcare Gmbh Medical image segmentation from raw data using a deep attention neural network
US20200074701A1 (en) 2018-08-29 2020-03-05 Arizona Board Of Regents On Behalf Of Arizona State University Method and apparatus for detection and visualization of pulmonary embolism
US20200085382A1 (en) * 2017-05-30 2020-03-19 Arterys Inc. Automated lesion detection, segmentation, and longitudinal identification
US10643360B2 (en) 2017-02-10 2020-05-05 Arizona Board Of Regents On Behalf Of Arizona State University Real-time medical image visualization systems and related methods
US20200160065A1 (en) * 2018-08-10 2020-05-21 Naver Corporation Method for training a convolutional recurrent neural network and for semantic segmentation of inputted video using the trained convolutional recurrent neural network
US20200250794A1 (en) * 2017-07-31 2020-08-06 Institut Pasteur Method, device, and computer program for improving the reconstruction of dense super-resolution images from diffraction-limited images acquired by single molecule localization microscopy
US20200286263A1 (en) * 2018-01-26 2020-09-10 Tencent Technology (Shenzhen) Company Limited Image processing method, storage medium, and computer device
US20200311914A1 (en) * 2017-04-25 2020-10-01 The Board Of Trustees Of Leland Stanford University Dose reduction for medical imaging using deep convolutional neural networks
US20200320748A1 (en) * 2017-10-24 2020-10-08 L'oreal System and method for image processing using deep neural networks
US10803591B2 (en) * 2018-08-28 2020-10-13 International Business Machines Corporation 3D segmentation with exponential logarithmic loss for highly unbalanced object sizes
US10810469B2 (en) * 2018-05-09 2020-10-20 Adobe Inc. Extracting material properties from a single image
US10812825B2 (en) * 2016-11-14 2020-10-20 Google Llc Video frame synthesis with deep learning
US20200364477A1 (en) 2019-05-16 2020-11-19 Arizona Board Of Regents On Behalf Of Arizona State University Methods, systems, and media for discriminating and generating translated images
US20200380675A1 (en) * 2017-11-22 2020-12-03 Daniel Iring GOLDEN Content based image retrieval for lesion analysis
US20200380695A1 (en) 2019-05-28 2020-12-03 Zongwei Zhou Methods, systems, and media for segmenting images
WO2021016087A1 (en) 2019-07-19 2021-01-28 Arizona Board Of Regents On Behalf Of Arizona State University Systems for the generation of source models for transfer learning to application specific models

Patent Citations (78)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012109670A1 (en) 2011-02-11 2012-08-16 Arizona Board Of Regents, For And On Behalf Of, Arizona State University Systems, methods, and media for detecting an anatomical object in a medical device image
WO2012109676A1 (en) 2011-02-11 2012-08-16 Arizona Board Of Regents For And On Behalf Of Arizona State University Methods, systems, and media for determining carotid intima-media thickness
US9684957B2 (en) 2011-02-11 2017-06-20 Arizona Board Of Regents, A Body Corporate Of The State Of Arizona, Acting For And On Behalf Of Arizona State University Systems methods, and media for detecting an anatomical object in a medical device image using a multi-stage classifier
US10610203B2 (en) 2011-02-11 2020-04-07 The Arizona Board Of Regents On Behalf Of Arizona State University Methods, systems, and media for determining carotid intima-media thickness
US9330336B2 (en) 2011-09-16 2016-05-03 Arizona Board of Regents, a body corporate of the State of Arizona, acting for and on behalf of, Arizona State University Systems, methods, and media for on-line boosting of a classifier
WO2013116865A1 (en) 2012-02-02 2013-08-08 Arizona Board Of Regents, For And On Behalf Of, Arizona State University Systems, methods, and media for updating a classifier
US9152926B2 (en) 2012-02-02 2015-10-06 Arizona Board Of Regents On Behalf Of Arizona State University Systems, methods, and media for updating a classifier
WO2013116867A1 (en) 2012-02-03 2013-08-08 Arizona Board Of Regents, For And On Behalf Of, Arizona State University Systems, methods, and media for monitoring the condition of a patient's heart
US9603554B2 (en) 2012-02-03 2017-03-28 The Arizona Board Of Regents Systems, methods, and media for monitoring the condition of a patient's heart
US9449381B2 (en) 2012-09-10 2016-09-20 Arizona Board Of Regents, A Body Corporate Of The State Of Arizona, Acting For And On Behalf Of Arizona State University Methods, systems, and media for generating and analyzing medical images having elongated structures
US9710916B2 (en) 2013-03-14 2017-07-18 Arizona Board Of Regents, A Body Corporate Of The State Of Arizona, Acting For And On Behalf Of Arizona State University Kernel sparse models for automated tumor segmentation
WO2015031641A1 (en) 2013-08-29 2015-03-05 Mayo Foundation For Medical Education And Research System and method for boundary classification and automatic polyp detection
US9741116B2 (en) 2013-08-29 2017-08-22 Mayo Foundation For Medical Education And Research System and method for boundary classification and automatic polyp detection
US10045728B2 (en) 2013-10-07 2018-08-14 Arizona Board Of Regents On Behalf Of Arizona State University Kidney glomeruli measurement systems and methods
US10120980B2 (en) 2014-01-27 2018-11-06 Arizona Board Of Regents On Behalf Of Arizona State University Diagnosing pulmonary embolism by integrating patient-level diagnosis and embolus-level detection
WO2015113014A1 (en) 2014-01-27 2015-07-30 Arizona Board Of Regents On Behalf Of Arizona State University Diagnosing pulmonary embolism by integrating patient-level diagnosis and embolus-level detection
US20170124701A1 (en) 2014-03-17 2017-05-04 Arizona Board Of Regents On Behalf Of Arizona State University System and method for measuring artery thickness using ultrasound imaging
WO2015142808A1 (en) 2014-03-17 2015-09-24 Arizona Board Of Regents On Behalf Of Arizona State University System and method for measuring artery thickness using ultrasound imaging
WO2015164768A1 (en) 2014-04-24 2015-10-29 Arizona Board Of Regents On Behalf Of Arizona State University System and method for detecting polyps from learned boundaries
US9747687B2 (en) 2014-04-24 2017-08-29 Arizona Board Of Regents On Behalf Of Arizona State University System and method for detecting polyps from learned boundaries
WO2015164724A1 (en) 2014-04-24 2015-10-29 Arizona Board Of Regents On Behalf Of Arizona State University System and method for quality assessment of optical colonoscopy images
US9978142B2 (en) 2014-04-24 2018-05-22 Arizona Board Of Regents On Behalf Of Arizona State University System and method for quality assessment of optical colonoscopy images
US9700213B2 (en) 2014-09-12 2017-07-11 Mayo Foundation For Medical Education And Research System and method for automatic polyp detection using global geometric constraints and local intensity variation patterns
US10052027B2 (en) 2014-09-12 2018-08-21 Mayo Foundation For Medical Education And Research System and method for automatic polyp detection using global geometric constraints and local intensity variation patterns
WO2016161115A1 (en) 2015-03-31 2016-10-06 Mayo Foundation For Medical Education And Research System and methods for automatic polyp detection using convolutional neural networks
US10055843B2 (en) 2015-03-31 2018-08-21 Mayo Foundation For Medical Education And Research System and methods for automatic polyp detection using convulutional neural networks
US9959615B2 (en) 2015-07-01 2018-05-01 Arizona Board Of Regents On Behalf Of Arizona State University System and method for automatic pulmonary embolism detection
US10328282B2 (en) 2015-07-13 2019-06-25 Mayo Foundation For Medical Education And Research System and method for novel chance-constrained optimization in intensity-modulated proton therapy planning to account for range and patient setup uncertainties
US10861151B2 (en) 2015-08-07 2020-12-08 The Arizona Board Of Regents On Behalf Of Arizona State University Methods, systems, and media for simultaneously monitoring colonoscopic video quality and detecting polyps in colonoscopy
WO2017027475A1 (en) 2015-08-07 2017-02-16 Jianming Liang Methods, systems, and media for simultaneously monitoring colonoscopic video quality and detecting polyps in colonoscopy
US10157467B2 (en) 2015-08-07 2018-12-18 Arizona Board Of Regents On Behalf Of Arizona State University System and method for detecting central pulmonary embolism in CT pulmonary angiography images
US20180259608A1 (en) * 2015-11-29 2018-09-13 Arterys Inc. Automated cardiac volume segmentation
US9924927B2 (en) 2016-02-22 2018-03-27 Arizona Board Of Regents On Behalf Of Arizona State University Method and apparatus for video interpretation of carotid intima-media thickness
US20180101957A1 (en) * 2016-10-06 2018-04-12 Qualcomm Incorporated Neural network for image processing
US10812825B2 (en) * 2016-11-14 2020-10-20 Google Llc Video frame synthesis with deep learning
US20180177461A1 (en) * 2016-12-22 2018-06-28 The Johns Hopkins University Machine learning approach to beamforming
US20180218497A1 (en) * 2017-01-27 2018-08-02 Arterys Inc. Automated segmentation utilizing fully convolutional networks
US10643360B2 (en) 2017-02-10 2020-05-05 Arizona Board Of Regents On Behalf Of Arizona State University Real-time medical image visualization systems and related methods
US20180268220A1 (en) * 2017-03-17 2018-09-20 Magic Leap, Inc. Room layout estimation methods and techniques
US10096125B1 (en) * 2017-04-07 2018-10-09 Adobe Systems Incorporated Forecasting multiple poses based on a graphical image
US20200020098A1 (en) * 2017-04-19 2020-01-16 Siemens Healthcare Gmbh Target detection in latent space
US20200311914A1 (en) * 2017-04-25 2020-10-01 The Board Of Trustees Of Leland Stanford University Dose reduction for medical imaging using deep convolutional neural networks
US20180314943A1 (en) 2017-04-27 2018-11-01 Jianming Liang Systems, methods, and/or media, for selecting candidates for annotation for use in training a classifier
US20200085382A1 (en) * 2017-05-30 2020-03-19 Arterys Inc. Automated lesion detection, segmentation, and longitudinal identification
US20190005670A1 (en) * 2017-06-28 2019-01-03 Magic Leap, Inc. Method and system for performing simultaneous localization and mapping using convolutional image transformation
US20190035113A1 (en) * 2017-07-27 2019-01-31 Nvidia Corporation Temporally stable data reconstruction with an external recurrent neural network
US20200250794A1 (en) * 2017-07-31 2020-08-06 Institut Pasteur Method, device, and computer program for improving the reconstruction of dense super-resolution images from diffraction-limited images acquired by single molecule localization microscopy
US20190049540A1 (en) * 2017-08-10 2019-02-14 Siemens Healthcare Gmbh Image standardization using generative adversarial networks
US10521927B2 (en) * 2017-08-15 2019-12-31 Siemens Healthcare Gmbh Internal body marker prediction from surface data in medical imaging
US20190142519A1 (en) * 2017-08-15 2019-05-16 Holo Surgical Inc. Graphical user interface for displaying automatically segmented individual parts of anatomy in a surgical navigation system
US20190064378A1 (en) * 2017-08-25 2019-02-28 Wei D. LIU Automated Seismic Interpretation Using Fully Convolutional Neural Networks
US20190108634A1 (en) * 2017-10-09 2019-04-11 The Board Of Trustees Of The Leland Stanford Junior University Contrast Dose Reduction for Medical Imaging Using Deep Learning
US20200320748A1 (en) * 2017-10-24 2020-10-08 L'oreal System and method for image processing using deep neural networks
US20190155973A1 (en) * 2017-11-02 2019-05-23 Airworks Solutions, Inc. Methods and apparatus for automatically defining computer-aided design files using machine learning, image analytics, and/or computer vision
US20190139216A1 (en) * 2017-11-03 2019-05-09 Siemens Healthcare Gmbh Medical Image Object Detection with Dense Feature Pyramid Network Architecture in Machine Learning
US20190147255A1 (en) * 2017-11-15 2019-05-16 Uber Technologies, Inc. Systems and Methods for Generating Sparse Geographic Data for Autonomous Vehicles
US20200380675A1 (en) * 2017-11-22 2020-12-03 Daniel Iring GOLDEN Content based image retrieval for lesion analysis
US20190213779A1 (en) * 2018-01-10 2019-07-11 The Board Of Trustees Of The University Of Illinois Apparatus and method for producing three-dimensional models from magnetic resonance imaging
US20190223725A1 (en) * 2018-01-25 2019-07-25 Siemens Healthcare Gmbh Machine Learning-based Segmentation for Cardiac Medical Imaging
US20200286263A1 (en) * 2018-01-26 2020-09-10 Tencent Technology (Shenzhen) Company Limited Image processing method, storage medium, and computer device
US20190261945A1 (en) * 2018-02-26 2019-08-29 Siemens Medical Solutions Usa, Inc. Three-Dimensional Segmentation from Two-Dimensional Intracardiac Echocardiography Imaging
US20190295223A1 (en) * 2018-03-22 2019-09-26 Adobe Inc. Aesthetics-guided image enhancement
US20190311202A1 (en) * 2018-04-10 2019-10-10 Adobe Inc. Video object segmentation by reference-guided mask propagation
US20190332896A1 (en) 2018-04-27 2019-10-31 Jianming Liang Methods, systems, and media for selecting candidates for annotation for use in training classifiers
US20190340810A1 (en) * 2018-05-03 2019-11-07 Adobe Inc. Utilizing an object relighting neural network to generate digital images illuminated from a target lighting direction
US20190340152A1 (en) * 2018-05-04 2019-11-07 Cornami Inc. Reconfigurable reduced instruction set computer processor architecture with fractured cores
US10810469B2 (en) * 2018-05-09 2020-10-20 Adobe Inc. Extracting material properties from a single image
US20190369190A1 (en) * 2018-06-04 2019-12-05 Korea Advanced Institute Of Science And Technology Method for processing interior computed tomography image using artificial neural network and apparatus therefor
US20200005511A1 (en) * 2018-06-28 2020-01-02 Adobe Inc. Determining Image Handle Locations
US20200041597A1 (en) * 2018-08-01 2020-02-06 Siemens Healthcare Gmbh Magnetic Resonance Fingerprinting Image Reconstruction and Tissue Parameter Estimation
US20200041612A1 (en) * 2018-08-02 2020-02-06 Metawave Corporation Recurrent super-resolution radar for autonomous vehicles
US20200160065A1 (en) * 2018-08-10 2020-05-21 Naver Corporation Method for training a convolutional recurrent neural network and for semantic segmentation of inputted video using the trained convolutional recurrent neural network
US20200065969A1 (en) * 2018-08-27 2020-02-27 Siemens Healthcare Gmbh Medical image segmentation from raw data using a deep attention neural network
US10803591B2 (en) * 2018-08-28 2020-10-13 International Business Machines Corporation 3D segmentation with exponential logarithmic loss for highly unbalanced object sizes
US20200074701A1 (en) 2018-08-29 2020-03-05 Arizona Board Of Regents On Behalf Of Arizona State University Method and apparatus for detection and visualization of pulmonary embolism
US20200364477A1 (en) 2019-05-16 2020-11-19 Arizona Board Of Regents On Behalf Of Arizona State University Methods, systems, and media for discriminating and generating translated images
US20200380695A1 (en) 2019-05-28 2020-12-03 Zongwei Zhou Methods, systems, and media for segmenting images
WO2021016087A1 (en) 2019-07-19 2021-01-28 Arizona Board Of Regents On Behalf Of Arizona State University Systems for the generation of source models for transfer learning to application specific models

Non-Patent Citations (53)

* Cited by examiner, † Cited by third party
Title
Al-hinnawi, A.R.M., et al., "Collaboration between interactive three-dimensional visualization and computer aided detection of pulmonary embolism on computed tomography pulmonary angiography views" (2018), Radiological Physics and Technology 11, pp. 61-72.
Armato, S.G., et al., "The lung image database consortium (Iidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans," Medical physics, (2011) pp. 915-931.
Badrinarayanan, V., et al., "Segnet, A deep convolutional encoder-decoder architecture for image segmentation," IEEE transactions on pattern analysis and machine intelligence (2017), pp. 39(12):2481-2495.
Bouma, H., et al., "Automatic detection of pulmonary embolism in cta images" (2009), Medical Imaging, IEEE Transactions on 28, pp. 1223-1230.
Buhmann, S., et al., "Clinical evaluation of a computer-aided diagnosis (cad) prototype for the detection of pulmonary embolism" (2007), Academic radiology 14, pp. 651-658.
Calder, K.K., et al., "The mortality of untreated pulmonary embolism in emergency department patients" (2005), Annals of emergency medicine 45, pp. 302-310.
Cardona, A., et al., "An integrated micro- and macroarchitectural analysis of the drosophila brain by computer-assisted serial section electron microscopy," PLoS biology (2010), 17 pages.
Chen, L.-C., et al., "Encoder-decoder with atrous separable convolution for semantic image segmentation," (2018), 18 pages.
Ciresan, D., et al., "Multi-column deep neural networks for image classification" (2012), Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, IEEE. pp. 3642-3649.
Ding, J., et al., "Accurate pulmonary nodule detection in computed tomography images using deep convolutional neural networks" (2017), International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 559-567.
Dou, Q., et al., "3d deeply supervised net-work for automatic liver segmentation from ct volumes" (2016), International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 149-157.
Drozdzal, M., et al., "The importance of skip connections in biomedical image segmentation," Deep Learning and Data Labeling for Medical Applications (2016), pp. 179-187.
Engelke, C., et al., "Computer assisted detection of pulmonary embolism: performance evaluation in consensus with experienced and inexperienced chest radiologists" (2008), European radiology 18, pp. 298-307.
Fairfield, J., "Toboggan contrast enhancement for collrast segmentation," Pattern Recognition, 1990. Proceedings., 10th International Conference, IEEE. pp. 712-716.
Fourure, D., et al., "Residual conv-deconv grid network for semantic segmentation," arXiv preprint arXiv: 1707.07958, (2017), 12 pages.
Glorot, X., et al., "Understanding the difficulty of training deep feedforward neural networks" (2010), International conference on artificial intelligence and statistics, pp. 249-256.
Greenspan, H., et al., "Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique," IEEE Transactions on Medical Imaging (2016), pp. 1153-1159.
He, K., et al., "Deep residual learning for image recognition," Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 770-778.
He, K., et al.,"Mask r-cnn," Computer Vision (ICCV), 2017 IEEE International Conference, (2017), pp. 2980-2988.
Huang, G., et al., "Densely connected convolutional networks," Proceedings of the IEEE conference on computer vision and pattern recognition (2017), vol. 1, p. 3.
Jegou, S., et al., "The one hundred layers tiramisu: Fully convulutional densenets for semantic segmentation," Computer Vision and Pattern Recognition Workshops (CVPRW) (2017), pp. 1175-1183.
Krizhevsky, A., et al., "Imagenet classification with deep convolutional neural networks" (2012), Advances in neural information processing systems, pp. 1097-1105.
LeCun, Y., et al., "Backpropagation applied to handwritten zip code recognition" (1989), Neural computation 1, pp. 541-551.
Lee, C.-Y., et al., "Deeply-supervised nets," Artificial Intelligence and Statistics (2015), pp. 562-570.
Li, X., et al., "H-denseunet: Hybrid densely connected unet for liver and liver tumor segmentation from ct volumes," arXiv preprint arXiv:1709.07330 (2017), 10 pages.
Liang, J., et al., "Computer aided detection of pulmonary embolism with tobogganing and multiple instance classification in CT pulmonary angiography" (2007), Information Processing in Medical Imaging, Springer, pp. 630-541.
Long, J., "Fully convolutional networks for semantic segmentation," Proceedings of the IEEE conference on computer vision and pattern recognition (2015), pp. 3431-3440.
Maizlin, Z.V., et al., "Computer aided detection of pulmonary embolism on ct angiography: initial experience" (2007), Journal of thoracic imaging 22, pp. 324-329.
Masoudi, M., et al., "A new dataset of computed-tomography angiography images for computer-aided detection of pulmonary embolism" (2018), Scientific data.
Milletari, F., et al., "V-net: Fully convolutional neural networks for volumetric medical image segmentation," in: 3D Vision (3DV), 2016 Fourth International Conference on, IEEE (2016), pp. 565-571.
NesenNet ("NestNet"), https://openreview.net/forum?id=ryPLSWnsM, Apr. 11, 2018, entire document.
NesenNet ("NestNet"), https://openreview.net/pdf?id=ryPLSWnsM, Apr. 11, 2018, entire document.
Ozkan, H., et al., "A novel method for pulmonary embolism detection in cta images" (2014), Computer methods and programs in biomedicine 113, pp. 757-766.
Park, S.C., et al., "A multistage approach to improve performance of computer-aided detection of pulmonary embolisms depicted on CT images: Preliminary investigation" (2011), Biomedical Engineering, IEEE Transactions on 58, pp. 1519-1527.
Prasoon, A., et al., "Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network," Medical Image Computing and Computer-Assisted Intervention-MICCAI 2013, Springer, pp. 246-253.
Quan, T.M., et al., "Fusionnet: A deep fully residual convolutional neural network for image segmentation in connectomics" (2016), arXiv preprint arXiv:1612.05360, 10 pages.
Ronneberger, O., et al., "U-net: Convolutional networks for biomedical image segmentation," International Conference on Medical image computing and computer-assisted intervention (2015), pp. 234-241.
Roth, H., et al., "A new 2.5d representation for lymph node detection using random sets of deep convolutional neural network observations," in Golland, P., Hata, N., Barillot, C., Homegger, J., Howe, R. (Eds.), Medical Image Computing and Computer-Assisted Intervention MICCAI 2014, Springer International Publishing, vol. 8673 of Lecture Notes in Computer Science, pp. 520-527.
Roth, H., et al., "Detection of sclerotic spine metastases via random aggregation of deep convolutional neural network classifications," in: Yao, J., Glocker, B., Klinder, T., Li, S. (Eds.), Recent Advances in Computational Methods and Clinical Applications for Spine Imaging (2015), Springer International Publishing, vol. 20 of Lecture Notes in Computational Vision and Biomechanics, pp. 3-12.
Roth, H.R., et al., "Deep convolutional networks for pancreas segmentation in ct imaging" (2015), SPIE Medical Imaging, International Society for Optics and Photonics, pp. 94131G-94131G.
Roth, H.R., et al., "Improving computer-aided detection using convolutional neural networks and random view aggregation," (2016) To appear in IEEE Transaction in Medical Imaging, 12 pages.
Sadigh, G., et al., "Challenges, controversies, and hot topics in pulmonary embolism imaging" (2011), American Journal of Roentgenology 196,19 pages.
Setio, A.A.A., et al., "Pulmonary nodule detection in ct images: false positive reduction using multiview convolutional networks" (2016), IEEE transactions on medical imaging 35, pp. 1160-1169.
Simonyan, K., et al., "Very deep convolutional networks for large-scale image recognition," (2014), 13 pages.
Szegedy, C., "Going deeper with convolutions," Cvpr (2015), 9 pages.
Tajbakhsh, N., "Convulutional neural networks for medical image analysis: Full training or fine tuning?" IEEE transactions on medical imaging (2016), pp. 1299-1312.
Tajbakhsh, N., et al., "A comprehensive computeraided polyp detection system for colonoscopy videos" (2015), Information Processing in Medical Imaging, Springer. pp. 327-338.
Tajbakhsh, N., et al., "Automated polyp detection in colonoscopy videos using shape and context information" (2015), Medical Imaging, IEEE Transactions on pp. 1-1.
Tajbakhsh, N., et al., "Computer-aided pulmonary embolism detection using a novel vessel-aligned multi-planar image representation and convolutional neural networks" (2015), International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 62-69.
Tajbakhsh, N., et al., "Surrogate supervision for medical image analysis: Effective deep learning from limited quantities of labeled data" (2019), arXiv preprint arXiv:1901.08707.
Tong et al., "Image Super-Resolution Using Dense Skip Connections," IEEE International Conference on Computer Vision (ICCV), 2017. *
U.S. Appl. No. 17/098,422, filed Nov. 15, 2020, Hosseinzadeh Taher et al.
Zhou, Z., et al., "Fine-tuning convolutional neural networks for biomedical image anaysis: actively incrementally," IEEE conference on computer vision and pattern recognition (CVPR) (2017), pp. 7340-7351.

Also Published As

Publication number Publication date
US20200074271A1 (en) 2020-03-05

Similar Documents

Publication Publication Date Title
US11164067B2 (en) Systems, methods, and apparatuses for implementing a multi-resolution neural network for use with imaging intensive applications including medical imaging
Chen et al. MRI tumor segmentation with densely connected 3D CNN
CN110674866B (en) Method for detecting X-ray breast lesion images by using transfer learning characteristic pyramid network
Schuegraf et al. Automatic building footprint extraction from multi-resolution remote sensing images using a hybrid FCN
US20200380695A1 (en) Methods, systems, and media for segmenting images
WO2019167884A1 (en) Machine learning method and device, program, learned model, and identification device
CN110443818B (en) Graffiti-based weak supervision semantic segmentation method and system
Kang et al. Nuclei segmentation in histopathological images using two-stage learning
Lee et al. Scribble2label: Scribble-supervised cell segmentation via self-generating pseudo-labels with consistency
Dong et al. Self-supervised representation learning for remote sensing image change detection based on temporal prediction
Fayyaz et al. STFCN: spatio-temporal fully convolutional neural network for semantic segmentation of street scenes
Zhai et al. Linking convolutional neural networks with graph convolutional networks: application in pulmonary artery-vein separation
Lu et al. A coarse-to-fine fully convolutional neural network for fundus vessel segmentation
CN112634296A (en) RGB-D image semantic segmentation method and terminal for guiding edge information distillation through door mechanism
Karabağ et al. Semantic segmentation of HeLa cells: An objective comparison between one traditional algorithm and four deep-learning architectures
Zhao et al. Modeling heterogeneity in mode-switching behavior under a mobility-on-demand transit system: An interpretable machine learning approach
Schorr et al. Neuroscope: An explainable ai toolbox for semantic segmentation and image classification of convolutional neural nets
Ma et al. Attention-guided deep graph neural network for longitudinal alzheimer’s disease analysis
Gao et al. Nuclei grading of clear cell renal cell carcinoma in histopathological image by composite high-resolution network
Wang et al. Salient object detection by robust foreground and background seed selection
Xu et al. TE-SAGAN: An Improved Generative Adversarial Network for Remote Sensing Super-Resolution Images
Lai et al. Segmentation of brain mr images by using fully convolutional network and gaussian mixture model with spatial constraints
Chen et al. Vectorization of historical maps using deep edge filtering and closed shape extraction
Li et al. Render u-net: A unique perspective on render to explore accurate medical image segmentation
Yang et al. Glomerular disease classification and lesion identification by machine learning

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: MICROENTITY

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: MICROENTITY

Free format text: ENTITY STATUS SET TO MICRO (ORIGINAL EVENT CODE: MICR); ENTITY STATUS OF PATENT OWNER: MICROENTITY

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO MICRO (ORIGINAL EVENT CODE: MICR); ENTITY STATUS OF PATENT OWNER: MICROENTITY

AS Assignment

Owner name: ARIZONA BOARD OF REGENTS ON BEHALF OF ARIZONA STATE UNIVERSITY, ARIZONA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIANG, JIANMING;ZHOU, ZONGWEI;RAHMAN SIDDIQUEE, MD MAHFUZUR;AND OTHERS;SIGNING DATES FROM 20190910 TO 20190920;REEL/FRAME:050507/0674

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO MICRO (ORIGINAL EVENT CODE: MICR); ENTITY STATUS OF PATENT OWNER: MICROENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE