JP2021507327A - Features Systems and methods for interactive expression learning transfer by deep learning of ontology - Google Patents

Abstract

A method for interactive feature learning transfer to a convolutional neural network (CNN) is presented. The method comprises obtaining at least the first and second input image datasets from the first and second imaging modality. Further, the method is a first supervised learning CNN based on the label associated with the first input image data set, and a second supervised learning CNN based on the label associated with the second input image data set. Collaborative training of learning CNNs to generate one or more feature primitives and corresponding mapping functions, and first and second inputs to the first unsupervised learning CNN and the second unsupervised learning CNN, respectively. Performing at least one of jointly training on an image dataset to learn a compressed representation of the input image dataset, the compressed representation containing common feature primitives and corresponding mapping functions. Includes storing common feature primitives and corresponding mapping functions in the feature primitive repository. [Selection diagram] Fig. 2

Description

Embodiments herein relate to systems and methods for generating transferable feature learning of medical image data of various anatomical structures obtained from various imaging modality for use in learning networks. .. Specifically, the system and method determine the expression learning of medical image data as a set of feature primitives based on the biology of the physics and anatomical structure of the first and / or second imaging modalities, and others. The purpose is to construct a new convolutional network for learning issues such as classification and segmentation of medical image data from the imaging modalities of.

As is understood, machine learning is a subfield of computer science that "gives the computer the ability to learn without being explicitly programmed." Machine learning explores the research and construction of algorithms that can learn from data and predict it. In machine learning, feature learning or expression learning is a set of techniques that transform raw data inputs into expressions that can be effectively used in machine learning tasks. Expression learning is motivated by the fact that machine learning tasks such as classification often require convenient inputs to process both mathematically and computationally. However, actual data such as images, videos, and sensor measurements are usually complex, verbose, and highly variable. Therefore, it is desirable to identify useful features or representations from the raw data. Currently, manual feature identification methods require expensive human labor and rely on expert knowledge. Also, manually generated expressions are usually not suitable for generalization, thus motivating the design of efficient expression learning methods to automate and generalize feature or expression learning.

Furthermore, in machine learning, a convolutional neural network (CNN or ConvNet) or a deep convolutional neural network is a type of feed-forward artificial neural network whose connection patterns between neurons are inspired by the tissues of the animal's visual cortex. .. A convolutional neural network (CNN) is a biologically inspired variant of a multi-layer perceptron designed to emulate the behavior of the visual cortex with minimal pretreatment. Note that the Multilayer Perceptron Network (MLP) is a feedforward artificial neural network model that maps a set of input data to a suitable set of outputs. The MLP contains multiple layers of nodes in the directed graph, each layer fully connected to the next layer. In its current deployment, CNNs have a wide range of applications in image and video recognition, recommender systems, and natural language processing. CNNs are also known as shift-invariant or space-invariant artificial neural networks (SIANNs) based on weight-sharing architectures and translational invariant properties.

Note that in the deep CNN architecture, the convolutional layer is the core building block. The parameters associated with the convolutional layer include a set of learnable filters or kernels. During the forward path, each filter is convoluted over the width and height of the input volume, calculating the dot product between the items of the filter and the input, resulting in a two-dimensional (2D) activation map of the filter. .. Therefore, the network learns a filter that is activated when the network detects a particular type of feature at a given spatial position at the input. Deep CNN architectures are typically built specifically for deep learning problems. Deep learning is known to be effective in many tasks involving human perception and decision making. Typical applications for deep learning are handwriting recognition, image recognition, and speech recognition. Deep learning techniques construct a network model by using training data and the outcomes corresponding to that data. Once the network model is configured, it can be used with new data to determine outcomes. Furthermore, it will be appreciated that deep learning networks once learned for a particular outcome can be reused favorably for related outcomes.

In addition, CNNs can be used for supervised learning with labeled input datasets and for unsupervised learning with unlabeled input datasets. A labeled input dataset is a dataset in which the elements of the dataset are pre-associated with the classification scheme represented by the label. Therefore, CNNs can be trained on a labeled subset of the dataset and tested on another subset to verify accurate classification results.

The deep CNN architecture is a multi-layered structure in which layers are connected hierarchically. As the number of input-to-output mappings and filters for each layer increases, the multi-layer deep CNN can provide a huge number of parameters that need to be configured for its operation. If such CNN training data is lacking, the problem to learn is undecided. In this situation, it is advantageous to transfer specific parameters from the pre-trained CNN model. Transfer learning reduces the number of parameters optimized by freezing pretrained parameters in a subset of layers and provides proper initialization for adjusting the remaining layers. In the domain of medical imaging problems, preconfiguring CNNs for various classification and identification problems using transfer learning depends on data shortage situations, multiple imaging modality and anatomical structures. Benefits in improving heterogeneous data type challenges, as well as other clinical challenges.

According to one aspect of the specification, a method for interactive feature learning transfer to a convolutional neural network (CNN) is presented. The method comprises obtaining at least a first input image data set from a first imaging modality and a second input image data set from a second imaging modality. In addition, the method is a first supervised learning CNN based on the label associated with the first input image data set, and a second supervised learning CNN based on the label associated with the second input image data set. Collaborative training of learning CNNs to generate one or more feature primitives and corresponding mapping functions, and first unsupervised learning CNNs with a first input image dataset and a second unsupervised learning. Performing at least one of jointly training a CNN with a second input image dataset to learn a compressed representation of the input image dataset, which is a common feature of one or more. Includes primitives and corresponding mapping functions. In addition, the method involves storing at least one or more common feature primitives and corresponding mapping functions in the feature primitive repository.

According to another aspect of the specification, an interactive feature learning transfer (IRLT) unit for interactive feature learning transfer to a convolutional neural network (CNN) is presented. The IRLT unit obtains at least the first input image data set from the first imaging modality and the second input image data set from the second imaging modality, and the label associated with the first input image data set. Jointly train a first supervised learning CNN based on and a second supervised learning CNN based on the label associated with the second input image data set, and one or more feature primitives and correspondences. To generate mapping functions, and to jointly train the first unsupervised learning CNN in the first input image dataset and the second unsupervised learning CNN in the second input image dataset to input images. Performs at least one of learning a compressed representation of a data set, and the compressed representation includes an interactive learning network configurator configured to include one or more common feature primitives and corresponding mapping functions. In addition, the IRLT unit includes a feature primitive repository configured to store at least one or more common feature primitives and corresponding mapping functions.

According to yet another aspect of the present specification, a multi-modality transfer learning system is presented. The multi-modality transfer learning system includes a processor unit and a memory unit communicatively and operably coupled to the processor unit. In addition, the multi-modality transfer learning system is operably coupled to the processor unit to obtain at least a first input image data set from the first imaging modality and a second input image data set from the second imaging modality. A first supervised learning CNN based on the label associated with the first input image data set, and a second supervised learning CNN based on the label associated with the second input image data set. Collaborative training to generate one or more feature primitives and corresponding mapping functions, and a first unsupervised learning CNN in the first input image dataset and a second unsupervised learning CNN. Perform at least one of joint training with two input image datasets to learn a compressed representation of the input image dataset, where the compressed representation has one or more common feature primitives and corresponding mapping functions. Includes an Interactive Expression Learning Transfer (IRLT) unit that includes an interactive learning network configurator configured to include. In addition, the IRLT unit includes a feature primitive repository configured to store at least one or more common feature primitives and corresponding mapping functions.

These and other features and aspects of the embodiments herein will be better understood by reviewing the following detailed description with reference to the accompanying drawings, in the accompanying drawings, similar reference numerals. Represents similar parts throughout the drawing.

FIG. 6 is a schematic diagram of an exemplary system for interactive feature learning transfer by deep learning of feature ontology according to aspects of the present specification. FIG. 5 is a flow chart illustrating a method for constructing a set of feature primitives from unlabeled image data to enhance the feature primitive repository according to aspects of the present specification. It is a flowchart which shows the method for constructing the set of the feature primitive from the labeled image data, and enhancing the feature primitive repository according to the aspect of this specification. It is a flowchart which shows the method for learning an invisible data set based on the selection of a feature primitive by preconfiguring CNN with a mapping function according to the aspect of this specification. It is the schematic of one Embodiment of the interactive learning network configurator according to the aspect of this specification.

Various embodiments of exemplary systems and methods for interactive feature learning transfer by deep learning of feature ontology are presented, as described in detail below. Efforts to provide a concise description of these embodiments do not make it possible to describe all the features of the actual embodiments herein. In the development of actual embodiments, such as engineering or design projects, a number of implementation-specific decisions must be made to achieve a developer's specific objectives, such as addressing system-related and business-related constraints. Please understand that it must be done.

When describing the elements of various embodiments herein, the articles "one (a, an)", "this (the)", and "said" have one of those elements or It is intended to mean that there are multiple. The terms "comprising," "inclusion," and "having" are intended to be inclusive, and that additional elements may exist in addition to those listed. means. Moreover, the terms "construct" and "construct" and their variants are intended to mean mathematical decisions or calculations of mathematical constructs. The term "data drawn on any domain" or "data on any domain" is intended to mean data that corresponds to a domain, such as social media data, sensor data, corporate data, etc. There is.

As used herein, the term "transfer learning" or "inductive transfer" is used in machine learning that focuses on applying knowledge or learning gained while solving different related problems from one problem. Intended to mean an approach. This knowledge or learning can be characterized and / or expressed in various ways, typically by the combination and transformation of transfer functions, mapping functions, graphs, matrices, and other primitives. Also, as used herein, the term "transfer learning primitive" is intended to mean the characterization and / or expression of knowledge or learning gained by solving the machine learning problems described above.

In addition, the term "feature primitive" as used herein characterizes the aspects of the input dataset, typically the appearance and shape geometry of the region of interest (ROI) that corresponds to the image of the anatomical structure. , Or is intended to mean morphology, the images are ultrasound imaging system, computed tomography (CT) imaging system, positron emission tomography-CT (PET-CT) imaging system, magnetic resonance (MR) imaging. It can be obtained from imaging modalities such as systems. The anatomical structure may include, but is not limited to, internal organs of the human body such as lungs, liver, kidneys, stomach, heart, and brain.

An exemplary system 100 for interactive feature learning transfer by deep learning of feature ontology according to aspects of the present specification is shown in FIG. In the currently considered configuration of FIG. 1, system 100 includes a multi-modality transfer learning (MTL) subsystem 102. The MTL subsystem 102 includes an interactive feature learning transfer (IRLT) unit 104, a processor unit 108, a memory unit 110, and a user interface 106. The processor unit 108 is communicably coupled to the memory unit 110. The user interface 106 is operably coupled to the IRLT unit 104. Further, the IRLT unit 104 is operably coupled to the processor unit 108 and the memory unit 110. The system 100 and / or the MTL subsystem 102 may include a display unit 134. It should be noted that the MTL subsystem 102 may include other components or hardware and is not limited to the components shown in FIG.

The user interface 106 is configured to receive the user input 130 corresponding to the characteristics of the input image 128. User input 130 may include, but is not limited to, aspects or characteristics of input image 128 such as imaging modality, the anatomical structure commonly represented by input image 128, and the appearance of the ROI corresponding to input image 128.

In certain embodiments, the IRLT unit 104 can be executed via one or more processor units 108 and may be implemented as a software system or computer instruction stored in the memory unit 110. In other embodiments, the IRLT unit 104 may be implemented as a hardware system via, for example, an FPGA, a custom chip, an integrated circuit (IC), an application specific integrated circuit (ASIC), or the like.

As shown in FIG. 1, the IRLT unit 104 includes an interactive learning network configurator (ILNC) 112, one or more CNNs (unsupervised learning CNN) 114 configured for unsupervised learning, and supervised learning. It may include one or more CNNs (supervised learning CNNs) 116 configured in, and a feature primitive repository 118. ILNC112 is operably coupled to unsupervised learning CNN114, supervised learning CNN116, and feature primitive repository 118. In one embodiment, the INLC 112 may be a graphical user interface subsystem configured to allow the user to configure one or more supervised learning CNN116 or unsupervised learning CNN114.

The feature primitive repository 118 is configured to store one or more feature primitives and one or more corresponding mapping functions corresponding to the ROI of the input image 128. As used herein, the term "mapping function" is a transfer function or CNN filter that maps an ROI to a compressed representation so that the output of the CNN is a feature primitive that characterizes the ROI of the input image 128 based on aspects of the ROI. Is intended to represent. In one example, aspects of ROI may include shape geometry, appearance, morphology, and the like. Therefore, the feature primitive repository 118 is configured to store one or more mapping functions. These mapping functions, along with the corresponding feature primitives, can be used to preconfigure CNNs to learn new training sets. Therefore, the feature primitive repository 118 is configured to store feature primitives and mapping functions that are transfer learning obtained from other CNNs and preconfigure new CNNs to learn invisible datasets. As shown in FIG. 1, some non-limiting examples of feature primitives configured to be stored in the feature primitive repository 118 are appearance 120 corresponding to the ROI of image 128, shape corresponding to the ROI of image 128. Includes feature primitives that characterize the geometry 124 and the anatomy 126 corresponding to image 128.

In the currently considered configuration of FIG. 1, the ILNC 112 is configured to present the user with various tools and options for interactively characterizing one or more aspects of the ROI corresponding to the input image 128. An exemplary embodiment of ILNC112 is shown in FIG. 5, and the work of ILNC112 is described in more detail with reference to FIGS. 4 and 5.

System 100 is configured to develop a portfolio of feature primitives and mapping functions across one or more anatomical structures and imaging modality stored in the feature primitive repository 118. The system 100 then provides the ILNC 112 to the user based on a selection of modality, anatomical structure, shape geometry, morphology, etc. corresponding to one or more ROIs of the user invisible image dataset. Allows the CNN to be preconfigured to train invisible image datasets. As an example, one learning outcome of CNN could be a classification scheme that classifies image datasets. Other non-limiting examples of learning outcomes may include pixel-level segmentation, regression, and the like. The work of the system 100 will be described in more detail with reference to FIGS. 2-5.

In addition, the system 100 and / or the MTL subsystem 102 may be configured to visualize one or more of the feature primitives, mapping functions, image datasets, etc. on the display unit 134.

Next, with reference to FIG. 2, a flowchart 200 is presented that generally illustrates a method for constructing a set of feature primitives from unlabeled image data to augment the feature primitive repository, according to aspects herein. Method 200 will be described with reference to the components of FIG.

It should be noted that Flowchart 200 shows the main steps of a method for constructing a set of feature primitives from unlabeled image data and augmenting a feature primitive repository such as feature primitive repository 118. In some embodiments, the various steps of method 200 of FIG. 2, more specifically steps 202-208, may be performed by the processor unit 108 in conjunction with the memory unit 110 and the ILNC 112 of the IRLT unit 104. Further, step 210 may be performed by processor unit 108 in conjunction with memory unit 110 and one or more supervised learning CNN116. Also, steps 214-216 may be performed by the processor unit 108 in conjunction with the memory unit 110 and one or more unsupervised learning CNN 114s.

Method 200 begins in step 202 to obtain at least the first input image data set 220 and the second input image data set 222 corresponding to the first imaging modality and the second imaging modality. In one embodiment, the first and second input image datasets 220 and 222 may correspond to the anatomical structures of the human body such as the liver, lungs, kidneys, heart, brain and stomach. Further, the first and second imaging modality may include an ultrasonic imaging system, an MR imaging system, a CT imaging system, a PET-CT imaging system, or a combination thereof.

In step 204, a check is performed to determine if the input image data sets 220, 222 are labeled. As mentioned above, the label that references the input image dataset can generally represent a classification scheme or score that characterizes the aspect of the input image, such as shape geometry, appearance, and morphology. If it is determined in step 204 that the input image dataset is labeled, control moves to step 210, where the first CNN and the second CNN are for supervised learning of the input image dataset 220, 222. It is composed of. Step 210 will be described in more detail with reference to FIG. Following step 210, method 200 ends as indicated by step 212.

Referencing step 204 again, if it is determined that the input image data set is unlabeled, control moves to step 206, a second check is performed, the first input image data set 220 and the second. Determines if the input image dataset 222 contains sufficient data to properly train one or more CNNs. If in step 206 it is determined that the first and second input image datasets 220 and 222 contain sufficient data, control shifts to step 214. However, if in step 206 it is determined that the first and second input image datasets 220 and 222 do not have sufficient data, control shifts to step 208.

In one example, in step 208 it can be determined that the first input image data set 220 contains sufficient data and the second input image data set 222 does not contain sufficient data. Therefore, in this example, in step 208, the second input image data set 222 corresponding to the second imaging modality is augmented with additional data. Additional data for augmenting the second input image data set 222 can be obtained by processing the first input image data set 220 corresponding to the first imaging modality via an intensity mapping function. One non-limiting example of an intensity mapping function is to perform multi-modality acquisition through the use of one or more objects, eg, a first imaging modality and a second imaging modality corresponding to CT and MR. A regression that uses deep learning, handmade intensity features, or a combination thereof to learn patch-level mapping from the first modality to the second modality and maps the intensity of the first modality to the second modality. It can include a framework. After that, control shifts to step 214.

In step 214, the first unsupervised learning CNN and the second unsupervised learning CNN are jointly trained in the first input image data set 220 and the second input image data set 222 to input image data set 220, Learning 222 compressed representations, the compressed representation contains one or more common feature primitives and corresponding mapping functions.

One or more feature primitives characterize the image aspect of the first input image dataset 220. Note that the mapping function maps the input image dataset to the corresponding feature primitives. In one embodiment, the mapping function can be defined according to equation (1).

In equation (1), in h _CT , the region of interest of the image P _CT is the mapping function f and the weight.

A set of feature primitives obtained when mapped using. In this example, the image _PCT corresponds to an image obtained through the use of a CT imaging system.

Note that as a result of step 214, one or more mapping functions corresponding to the first imaging modality and the second imaging modality are generated. Note that these mapping functions map the first and second input image datasets 220 and 222 to the same feature primitive. In one embodiment, the second mapping function can be defined according to equation (2).

In equation (2), in h _MR , the region of interest of the image P _MR is the mapping function f and the weight.

A set of feature primitives obtained when mapped using. In this example, the image _PMR is obtained using an MR imaging system.

Further, in step 218, at least one or more feature primitives and corresponding mapping functions are stored in the feature primitive repository 118. Next, control proceeds to step 212 and ends method 200.

Next, with reference to FIG. 3, a flowchart 300 is presented that generally illustrates a method for constructing a set of feature primitives from labeled image data to augment a feature primitive repository, according to aspects herein. More specifically, Method 300 describes step 210 in FIG. 2 in more detail. The method 300 will be described with reference to the components of FIG.

It should be noted that Flowchart 300 shows the main steps of the method for constructing a set of feature primitives from labeled image data and augmenting the feature primitive repository. In some embodiments, the various steps of method 300 of FIG. 3, more specifically steps 302-308, may be performed by the processor unit 108 in conjunction with the memory unit 110 and the ILNC 112 of the IRLT unit 104. Further, steps 310-314 may be performed by the processor unit 108 in conjunction with the memory unit 110 and one or more supervised learning CNN 116.

Method 300 begins in step 302 to obtain at least the first input image data set 316 and the second input image data set 318 corresponding to at least the first imaging modality and the second imaging modality. In one embodiment, the first and second input image datasets 316, 318 may correspond to the anatomical structure of the human body, such as the liver, lungs, kidneys, heart, brain, stomach and the like. Further, the first and second imaging modality may include an ultrasonic imaging system, an MR imaging system, a CT imaging system, a PET-CT imaging system, or a combination thereof. Note that the learning outcome of one or more supervised CNNs can be the classification of images in the first and second input image datasets 316 and 318.

In step 304, a check is performed to determine if the first input image data set 316 and the second input image data set 318 contain sufficient data to properly train one or more CNNs. .. If in step 304 it is determined that the first and second input image datasets 316 and 318 contain sufficient data, control shifts to step 308. However, if in step 304 it is determined that the first and second input image data sets 316 and 318 do not have sufficient data, control shifts to step 306. In one example, in step 306 it can be determined that the first input image data set 316 contains sufficient data and the second input image data set 318 does not contain sufficient data. Therefore, in this example, in step 306, the second input image data set 318 corresponding to the second modality is augmented with additional data. Note that additional data is obtained by processing the first input image data set 316 corresponding to the first imaging modality via an intensity mapping function. As mentioned above, one non-limiting example of the intensity mapping function is multi through the use of one or more objects, eg, a first imaging modality and a second imaging modality corresponding to CT and MR. Perform modality acquisition and use deep learning, handmade strength features, or a combination thereof to learn patch level mapping from the first modality to the second modality, and the strength of the first modality to the second. It can include a regression framework that maps to modality. After that, control shifts to step 308.

Further, in step 308, the first supervised learning CNN and the second supervised learning CNN are the label associated with the first input image data set 316 and the label associated with the second input image data set 318. Jointly trained based on to generate one or more feature primitives and corresponding mapping functions.

In one embodiment, the learning outcome comprises one or more feature primitives that characterize the image aspect of the first input image data set 316 and the image aspect of the second input image data set 318 and the corresponding mapping function. The mapping function can map the corresponding first input image data set 316 and second input image data set 318 to one or more feature primitives. Therefore, the feature primitive is independent of the imaging modality used to acquire the first input image dataset 316 and the second input image dataset 318. Next, referring to step 312, one or more feature primitives and the corresponding mapping functions are stored in the feature primitive repository.

The methods 200 and 300 described above allow the creation of a portfolio of feature primitives and mapping functions corresponding to the images generated for multiple anatomical structures across multiple modality. The feature primitives and mapping functions are stored in the feature primitive repository 118. This portfolio of feature primitives and mapping functions also characterizes the learning gained by training CNNs with input image datasets. In addition, learning can be transferred to preconfigure new CNNs to obtain learning outcomes for different invisible datasets.

With the above in mind, FIG. 4 is a flow chart illustrating a method of preconfiguring a CNN with a mapping function to train an invisible dataset based on the selection of feature primitives, according to aspects herein. Shows 400. Method 400 will be described with reference to FIGS. 1, 2 and 3. It should be noted that Flowchart 400 shows the key steps of Method 400 for preconfiguring CNNs with mapping functions to learn invisible datasets based on the selection of feature primitives. In some embodiments, the various steps of method 400 of FIG. 4, more specifically steps 402-406, can be performed by the processor unit 108 in conjunction with the memory unit 110 and the ILNC 112 of the IRLT unit 104. Further, steps 408-410 may be performed by the processor unit 108 in conjunction with the memory unit 110 and one or more supervised learning CNN116.

Method 400 can be started in step 402 to obtain at least one input image data set 404. At least one input image data set 404 represents an invisible input image data set. In addition, at least one learning parameter 406 and learning outcome 408 corresponding to the input image data set 404 can be obtained. In certain embodiments, the input image data set 404, learning parameters 406, and learning outcome 408 may be obtained as user input 410. In one embodiment, the learning parameter 406 can include imaging modality, image anatomy, or a combination thereof. The learning outcome 408 may also include pixel-level output such as classification schemes, regression schemes, or segmentation.

In step 412, at least one feature primitive corresponding to the learning parameter 406 and the learning outcome 408 and the corresponding mapping function are obtained from the feature primitive repository 118. The CNN is then configured to train the input image dataset 404 using at least one feature primitive and at least one mapping function, as shown in step 414. In certain embodiments, the configuration of the CNN may involve setting one or more filters obtained from the feature primitive repository 118 in the mapping function. As a result of the process of step 414, a preconfigured CNN is generated. Further, in step 416, the preconfigured CNN is optimized with at least a training subset of the input image dataset 404.

In one embodiment, a trained convolutional autoencoder (CAE) for supervised learning using labeled data corresponding to the first imaging modality is an input image with several parameters according to equation (3). Fits to dataset 404.

In the formula (3), w is the region of interest of the image P ₁ mapping function f and the weight corresponding to the first imaging modality

Is a set of feature primitives obtained when mapped using, where α is a sparse set of CAE parameters. In this way, the number of filters can be reduced. The CAE parameter α can be further optimized with at least a training subset of the input image dataset 404. In this way, for supervised learning problems corresponding to the second imaging modality, a learning framework can be defined according to the following formulation.

In the formulation (4), the mapping function f obtained by the equation (3) is a mapping function corresponding to the region of interest of the image corresponding to the second imaging modality.

Applies to. Then, in step 418, the input image data set 404 is processed via an optimized CNN to obtain a learning outcome 420 corresponding to the requested learning outcome 408.

The workflow performed by method 400 will be described in more detail with reference to FIG. FIG. 5 is a schematic view of an embodiment of the interactive learning network configurator 112 of the interactive expression learning transfer unit 104 of FIG. 1 according to aspects of the present specification. As shown in FIG. 5, block diagram 500 generally represents the ILNC 112 shown in FIG. Reference numbers 502 to 508 generally represent visualization of feature primitives corresponding to imaging modality, anatomy, appearance, and shape geometry, respectively. The data of visualizations 502 to 508 can be obtained from the feature primitive repository 118 of FIG.

In FIG. 5, the ILNC 500 provides the user with an interactive menu selection. In particular, using the ILNC500, the user can select one or more aspects of the invisible image dataset learned by the CNN. Reference numbers 510-516 generally represent interactive menu options that may be available to the user to assist in the characterization of invisible image datasets. As an example, reference number 510 may relate to the imaging modality of an invisible image dataset. Menu options for block 510 may include CT, MR, PET, ultrasound and the like. Similarly, reference numbers 512-516 may indicate menu options related to the appearance, shape geometry, and anatomy of the invisible image dataset, respectively.

User-made selections from interactive menus allow preconfiguration of CNNs with feature primitives and mapping functions that correspond to menu selections. Reference number 518 generally represents a preconfigured visualization of the CNN. In certain embodiments, reference number 518 may correspond to one or more supervised learning CNN116 of IRLT unit 104 in FIG. In addition, the user can graphically browse, visualize, and combine the feature primitives of blocks 502-508 to create a preconfigured CNN518.

The systems and methods for interactive expression learning transfer by deep learning of the feature ontology described above can combine learned feature primitives and a portfolio of mapping functions to form a CNN and solve new medical image analysis problems. Provide a learning paradigm. Advantageously, the CNN can be trained to learn the appearance and morphology of the image. As an example, tumors can be classified as spots, cylinders, circles, bright / dark or striped. Overall, the network was trained for various combinations of anatomy, modality, appearance, and morphology (shape geometry) and was configured to instantly transfer pre-learned features to existing new problems. A rich portfolio can be generated.

It should be understood that according to any embodiment, not all such objects or advantages mentioned above can necessarily be achieved. Thus, for example, as will be apparent to those skilled in the art, the systems and techniques described herein do not necessarily achieve any other purpose or advantage as taught or suggested herein. It may be embodied or implemented in a manner that achieves or improves one advantage or a group of advantages taught in.

Although the art has been described in detail in relation to only a limited number of embodiments, it should be readily understood that the present specification is not limited to such disclosed embodiments. Rather, the technique may be modified to incorporate any number of modifications, substitutions, substitutions or equivalent configurations that have not been described so far but are commensurate with the spirit and scope of the claims. In addition, although various embodiments of the present technology have been described, it should be understood that aspects herein may include only some of the described embodiments. Therefore, this specification should not be limited by the above description, but only by the appended claims.

100 System 102 Multimodality Transfer Learning (MTL) Subsystem 104 Interactive Expression Learning Transfer (IRLT) Unit 106 User Interface 108 Processor Unit 110 Memory Unit 112 Interactive Learning Network Configurator (ILNC)
114 Unsupervised Learning Convolutional Neural Network (CNN)
116 Supervised Learning CNN
118 Features Primitive Repository 120 Appearance 124 Shape Geometry 126 Anatomical Structure 128 Input Image 130 User Input 134 Display Unit 200 Flow Flow, Method 202 Step 204 Step 206 Step 208 Step 210 Step 212 Step 214 Step 216 Step 218 Step 220 First Input Image data set 222 Second input image data set 300 Flow chart, method 302 Step 304 Step 306 Step 308 Step 310 Step 312 Step 314 Step 316 First input image data set 318 Second input image data set 400 Flow chart, method 402 Step 404 Input image data set 406 Learning parameter 408 Learning outcome 410 User input 412 Step 414 Step 416 Step 418 Step 420 Learning outcome 500 Schematic diagram, block diagram, ILNC
502 Visualization, Block 504 Visualization, Block 506 Visualization, Block 508 Visualization, Block 510 Block 512 Block 514 Block 516 Block 518 Preconfigured CNN

Claims (13)

A method (200, 300, 400) for interactive feature learning transfer to a convolutional neural network (CNN).
Obtaining at least the first input image data set (220, 316) from the first imaging modality and the second input image data set (222, 318) from the second imaging modality (202).
The first supervised learning CNN (116) is associated with the first supervised learning CNN (116) and the second input image data set (222, 318) based on the label associated with the first input image data set (220, 316). Jointly training a second supervised learning CNN (116) based on the label to generate one or more feature primitives and corresponding mapping functions (308), and a first unsupervised learning CNN (308). 114) was jointly trained in the first input image data set (220, 316) and the second unsupervised learning CNN (114) in the second input image data set (222, 318). Performing at least one of (214) learning a compressed representation of the input image data set (220, 222, 316, 318), said compressed representation being one or more common feature primitives and Including the corresponding mapping function and
A method (200, 300, 400) comprising storing at least one or more common feature primitives and the corresponding mapping functions in a feature primitive repository (118) (218, 312). To obtain at least one invisible input image dataset (404),
Obtaining at least one learning parameter and at least one learning outcome (408, 420) corresponding to the at least one invisible input image data set (404) (402).
Obtaining at least one feature primitive corresponding to the at least one learning parameter and the at least one learning outcome (408, 420) and a corresponding mapping function (412) via the feature primitive repository (118).
To construct a CNN for learning the at least one invisible input image dataset (404) based on the at least one feature primitive and the at least one mapping function (414).
Optimizing the configured CNN with at least a training subset of the invisible input image dataset (404) (416) and
Further comprising processing the at least one invisible input image data set (404) via the optimized CNN to obtain the at least one learning outcome (408, 420) (418). The method according to claim 1 (200, 300, 400). To jointly train the first supervised learning CNN (116) and the second supervised learning CNN (116) to generate the one or more common feature primitives and the corresponding mapping function ( 308) uses the first supervised learning CNN (116) in the first input image data set (220, 316) and the second supervised learning CNN (116) in the second input image. At least one or more of the commons that are jointly trained in the data set (222, 318) and characterize the first input image data set (220, 316) and the second input image data set (222, 318). The method of claim 1, comprising obtaining the feature primitives of, and the corresponding mapping functions of the first supervised learning CNN (116) and the second supervised learning CNN (116). 300, 400). It further comprises augmenting the second input image data set (222, 318) corresponding to the second imaging modality with additional data (208), the additional data being said via an intensity mapping function. The method (200, 300, 400) according to claim 1, which is obtained by processing the first input image data set (220, 316) corresponding to the first imaging modality. Processing the first input image data set (220, 316) corresponding to the first imaging modality via the intensity mapping function
Performing multi-modality acquisition from the first and second imaging modality corresponding to one or more objects through the regression framework of the intensity mapping function.
Through the regression framework of the intensity mapping function, patch level mapping from the first imaging modality to the second imaging modality is learned, and the intensity of the first imaging modality is determined by the second imaging modality. The method of claim 4 (200, 300, 400), comprising mapping to. An interactive expression learning transfer unit (104) for interactive expression learning transfer to a convolutional neural network (CNN).
Obtaining at least the first input image data set (220, 316) from the first imaging modality and the second input image data set (222, 318) from the second imaging modality,
The first supervised learning CNN (116) is associated with the first supervised learning CNN (116) and the second input image data set (222, 318) based on the label associated with the first input image data set (220, 316). To jointly train a second supervised learning CNN (116) based on the label to generate one or more common feature primitives and corresponding mapping functions, and to generate a first unsupervised learning CNN (114). ) Is jointly trained in the first input image data set (220, 316) and the second unsupervised learning CNN (114) is jointly trained in the second input image data set (222, 318). An interactive learning network configurator (112) that performs at least one of learning a compressed representation of an image data set and is configured to include one or more common feature primitives and corresponding mapping functions. )When,
An interactive feature learning transfer unit (104) comprising the at least one or more common feature primitives and a feature primitive repository (118) configured to store the corresponding mapping functions. The interactive learning network configurator (112)
Obtaining at least one invisible input image dataset (404),
Obtaining at least one learning parameter and at least one learning outcome (408, 420) corresponding to the at least one invisible input image data set (404),
Through the feature primitive repository (118), the at least one feature primitive corresponding to the at least one learning parameter and the at least one learning outcome (408, 420) and the corresponding mapping function are obtained.
A CNN for learning the at least one invisible input image dataset (404) based on the at least one feature primitive and the at least one mapping function is configured.
Optimizing the configured CNN with at least a training subset of the invisible input image dataset (404)
Claim that it is further configured to process the at least one invisible input image data set (404) via the optimized CNN to obtain the at least one learning outcome (408, 420). The interactive expression learning transfer unit (104) according to 6. The interactive learning network configurator (112) performs the first supervised learning CNN (116) with the first input image data set (220, 316) and the second supervised learning CNN (116). At least said that the first input image data set (220, 316) and the second input image data set (222, 318) are jointly trained with the second input image data set (222, 318). A claim further configured to obtain one or more common feature primitives, as well as said corresponding mapping functions of said first supervised learning CNN (116) and said second supervised learning CNN (116). Item 6. The interactive expression learning transfer unit (104). The interactive learning network configurator (112) is further configured to augment the second input image data set (222, 318) corresponding to the second imaging modality with additional data. The interactive expression learning transfer unit (104) according to claim 6, which is obtained by processing the first input image data set (220, 316) corresponding to the first imaging modality via an intensity mapping function. ). The intensity mapping function
Perform multi-modality acquisition from the first and second imaging modality corresponding to one or more objects,
It includes a regression framework configured to learn patch-level mapping from the first imaging modality to the second imaging modality and map the intensity of the first imaging modality to the second imaging modality. , The interactive expression learning transfer unit (104) according to claim 9. Processor unit (108) and
A memory unit (110) operably coupled to the processor unit (108),
Operatively coupled to the processor unit (108)
Obtaining at least the first input image data set (220, 316) from the first imaging modality and the second input image data set (222, 318) from the second imaging modality,
The first supervised learning CNN (116) is associated with the first supervised learning CNN (116) and the second input image data set (222, 318) based on the label associated with the first input image data set (220, 316). To jointly train a second supervised learning CNN (116) based on the label to generate one or more common feature primitives and corresponding mapping functions, and to generate a first unsupervised learning CNN (114). ) Is jointly trained in the first input image data set (220, 316) and the second unsupervised learning CNN (114) is jointly trained in the second input image data set (222, 318). An interactive learning network configurator (112) that performs at least one of learning a compressed representation of an image dataset, the compressed representation being configured to include one or more common feature primitives and corresponding mapping functions. ), And a feature primitive repository (118) configured to store the at least one or more common feature primitives and the corresponding mapping function.
A multi-modality transfer learning system (100) comprising an interactive expression learning transfer unit (104). The interactive learning network configurator (112)
Obtaining at least one invisible input image dataset (404),
Obtaining at least one learning parameter and at least one learning outcome (408, 420) corresponding to the at least one invisible input image data set (404),
Through the feature primitive repository (118), the at least one feature primitive corresponding to the at least one learning parameter and the at least one learning outcome (408, 420) and the corresponding mapping function are obtained.
A CNN for learning the at least one invisible input image dataset (404) based on the at least one feature primitive and the at least one mapping function is configured.
Optimizing the configured CNN with at least a training subset of the invisible input image dataset (404)
Claim that the at least one invisible input image data set (404) is further configured to process the at least one invisible input image data set (404) via the optimized CNN to obtain the at least one learning outcome (408, 420). 11. The multi-modality transfer learning system (100). The interactive learning network configurator (112)
The first supervised learning CNN (116) is the first input image data set (220, 316), and the second supervised learning CNN (116) is the second input image data set (222). , 318), and at least one or more of the common feature primitives, which are jointly trained in the first input image data set (220, 316) and feature the second input image data set (222, 318). The multimodality transfer learning system according to claim 11, further configured to obtain the corresponding mapping functions of the first supervised learning CNN (116) and the second supervised learning CNN (116). (100).