CN110838108A - Medical image-based prediction model construction method, prediction method and device - Google Patents

Medical image-based prediction model construction method, prediction method and device Download PDF

Info

Publication number
CN110838108A
CN110838108A CN201911059064.8A CN201911059064A CN110838108A CN 110838108 A CN110838108 A CN 110838108A CN 201911059064 A CN201911059064 A CN 201911059064A CN 110838108 A CN110838108 A CN 110838108A
Authority
CN
China
Prior art keywords
image
predicted
segmentation
network
prediction model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911059064.8A
Other languages
Chinese (zh)
Inventor
卢东焕
马锴
郑冶枫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201911059064.8A priority Critical patent/CN110838108A/en
Publication of CN110838108A publication Critical patent/CN110838108A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30016—Brain

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Artificial Intelligence (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the application discloses a medical image-based prediction model construction method, a prediction method and a prediction device, wherein the medical image-based prediction model construction method comprises the following steps: obtaining training sample labeling results, wherein the training sample labeling results are obtained by performing classification labeling on first to-be-predicted images of a plurality of sample users; carrying out image registration on the template image to be registered and the first image to be predicted to obtain a segmentation result of the first image to be predicted; inputting the first to-be-predicted image, the training sample labeling result and the segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to construct the prediction model based on the medical image. The prediction model based on the medical image provided by the embodiment of the application is adopted to predict the Alzheimer's disease, so that the improvement of the prediction precision of the Alzheimer's disease is facilitated.

Description

Medical image-based prediction model construction method, prediction method and device
Technical Field
The application relates to the technical field of image recognition, in particular to a medical image-based prediction model construction method, a prediction method and a prediction device.
Background
With the research and progress of artificial intelligence technology, the artificial intelligence technology is widely applied to various fields, such as common smart homes, virtual assistants, unmanned driving, intelligent medical treatment, and the like. In the medical field, Alzheimer's Disease (AD) is a neurodegenerative disease whose onset progresses slowly and deteriorates with time, and its diagnosis is often achieved by medical images such as Magnetic Resonance Imaging (MRI) and Positron Emission Computed Tomography (PET). Currently, the prediction of alzheimer's disease based on medical images may be performed by dividing the brain into different regions according to the anatomical structure of the brain, extracting artificially designed features from each region, and then training a classifier to predict the regions, or by directly using a 3D-CNN (3D convolutional Neural Networks, 3D-convolutional Neural Networks) model shown in fig. 1 to perform a plurality of convolution operations on the input MRI image (using a convolution kernel of 3 × 3 and maximum pooling of 2 × 2), concatenating the output results of the convolution operations, and then outputting the prediction result of the MRI image (probability P of alzheimer's disease) through processing of a full-concatenation layer and a classifier softmaxADNormal probability PNCAnd probability of mild cognitive dysfunction PMCI). However, the former method reduces the dimension of the input features and loses part of feature information, and the 3D-CNN prediction method ignores prior information, and both methods are low in prediction accuracy of Alzheimer's disease.
Disclosure of Invention
In order to solve the problems, the medical image-based prediction model construction method and the medical image-based prediction model construction device are provided.
The embodiment of the application provides a prediction model construction method based on a medical image in a first aspect, and the method comprises the following steps:
obtaining training sample labeling results, wherein the training sample labeling results are obtained by performing classification labeling on first to-be-predicted images of a plurality of sample users;
carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted;
inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to construct the prediction model based on the medical image.
A second aspect of the embodiments of the present application provides a prediction method based on a medical image, including:
acquiring an original medical image, and aligning and normalizing the original medical image to obtain a second image to be predicted;
inputting the second image to be predicted into a pre-constructed feature extraction sub-network based on a prediction model of a medical image to perform down-sampling operation, and extracting feature information of the second image to be predicted;
performing an upsampling operation on the second image to be predicted through an image segmentation subnetwork of the medical image-based prediction model, and outputting an image segmentation result of the second image to be predicted;
and classifying the characteristic information of the second image to be predicted through the classifier of the prediction model based on the medical image to obtain the prediction result of the second image to be predicted.
A third aspect of the embodiments of the present application provides a prediction model building apparatus based on medical images, including:
the system comprises a sample marking module, a data processing module and a data processing module, wherein the sample marking module is used for obtaining training sample marking results, and the training sample marking results are obtained by performing classification marking on first to-be-predicted images of a plurality of sample users;
the image registration module is used for carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted;
and the model building module is used for inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to build the prediction model based on the medical image.
A fourth aspect of the embodiments of the present application provides a prediction apparatus based on a medical image, including:
the image acquisition module is used for acquiring an original medical image, and aligning and normalizing the original medical image to obtain a second image to be predicted;
the feature extraction module is used for inputting the second image to be predicted into a pre-constructed feature extraction sub-network based on a prediction model of a medical image to perform down-sampling operation, and extracting feature information of the second image to be predicted;
the image segmentation module is used for performing upsampling operation on the second image to be predicted through an image segmentation subnetwork of the medical image-based prediction model and outputting an image segmentation result of the second image to be predicted;
and the result prediction module is used for classifying the characteristic information of the second image to be predicted through the classifier of the medical image-based prediction model to obtain the prediction result of the second image to be predicted.
A fifth aspect of embodiments of the present application provides an electronic device, which includes an input device, an output device, and a processor, and is adapted to implement one or more instructions; and the number of the first and second groups,
a computer storage medium storing one or more instructions adapted to be loaded by the processor and to perform the steps of the method of the first or second aspect.
A sixth aspect of embodiments of the present application provides a computer storage medium storing one or more instructions adapted to be loaded by a processor and to perform the steps of the method according to the first or second aspect.
According to the method, training sample labeling results are obtained, and the training sample labeling results are obtained by performing classification labeling on first to-be-predicted images of a plurality of sample users; this enables the deep neural network based on multi-objective learning to quickly learn the classification when used for predicting Alzheimer's disease, namely, Alzheimer's disease or Alzheimer's disease which will be suffered or will be suffered recently, or Alzheimer's disease which is not suffered. Carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted; in this way, the first image to be predicted is segmented by adopting the image registration method, and a standard segmented image with higher segmentation precision can be obtained. Inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to construct the prediction model based on the medical image; therefore, the deep neural network based on multi-target learning is trained by adopting the easily-learned two-classification labeling result and the standard segmentation image with higher segmentation precision, and the prediction precision of the constructed prediction model based on the medical image can be improved. Therefore, the prediction model based on the medical image, which is constructed by performing classification labeling on the first to-be-predicted image and performing image segmentation by adopting image registration, is beneficial to improving the prediction accuracy of the Alzheimer's disease when used for predicting the Alzheimer's disease.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a 3D-CNN provided in an embodiment of the present application;
FIG. 2a is a diagram of an application architecture according to an embodiment of the present application;
FIG. 2b is a diagram of an example of a framework for constructing a prediction model based on medical images according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a method for constructing a prediction model based on a medical image according to an embodiment of the present application;
fig. 4 is an exemplary diagram of image registration provided by an embodiment of the present application;
FIG. 5 is an exemplary diagram of solving a spatial transformation relation T according to an embodiment of the present application;
FIG. 6 is a diagram illustrating an example of an image segmentation result provided in an embodiment of the present application;
fig. 7 is a flowchart illustrating a flowchart of another medical image-based prediction model construction method according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a deep neural network based on multi-target learning according to an embodiment of the present application;
FIG. 9 is an exemplary diagram of a separable convolution as provided by an embodiment of the present application;
fig. 10 is a flowchart illustrating a medical image-based prediction method according to an embodiment of the present application;
fig. 11 is a schematic structural diagram of a prediction model construction apparatus based on medical images according to an embodiment of the present application;
fig. 12 is a schematic structural diagram of a prediction apparatus based on medical images according to an embodiment of the present application;
fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "comprising" and "having," and any variations thereof, as appearing in the specification, claims and drawings of this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus. Furthermore, the terms "first," "second," and "third," etc. are used to distinguish between different objects and are not used to describe a particular order.
The embodiment of the application provides a medical image-based prediction model construction scheme and a medical image-based prediction scheme, and a medical image-based prediction model constructed by the construction scheme is used for predicting a medical image, taking alzheimer's disease as an example, the accuracy of the medical image-based prediction model provided by the application for predicting that a user suffers from or will suffer from alzheimer's disease in the near future is higher, and certainly, the medical image-based prediction model is not limited to alzheimer's disease, and as long as the predicted classification result and some parts of the image are correlated, the image segmentation and classification prediction can be performed by using the medical image-based prediction model, for example, other brain diseases such as parkinson's disease and frontotemporal dementia, and therefore, the scheme provided by the application has strong clinical significance in the medical field. The medical image-based prediction model construction scheme and the medical image-based prediction scheme can be applied to electronic equipment, the electronic equipment includes but is not limited to a smart phone, a desktop computer, a tablet computer, a supercomputer, a server and the like, specifically, the electronic equipment can acquire medical images from medical imaging equipment to perform prediction model construction or direct prediction, and can acquire medical images from open source data to perform prediction model construction or direct prediction, for example: when the medical image-based prediction model is constructed, the medical research room can acquire related data from ADNI (Alzheimer's disease Neuroimaging Initiative), and the electronic equipment executes model construction related operation; for another example: when a patient is examined, a hospital can acquire a medical image of the patient through a medical imaging device, and an electronic device performs examination-related operations through a constructed prediction model based on the medical image, and the like.
First, a network system architecture to which the solution of the embodiments of the present application may be applied will be described by way of example with reference to the accompanying drawings. Referring to fig. 2a, fig. 2a is an application architecture diagram according to an embodiment of the present application, and as shown in fig. 2a, the application architecture diagram includes an electronic device, a medical imaging device, and a database, where the electronic device, the medical imaging device, and the database are in communication through a wired or wireless network connection. The electronic device may be an electronic device in a medical research laboratory, or an electronic device in a hospital examination room, and any electronic device capable of performing model construction and image prediction may be applied to the embodiment of the present application; the medical imaging device may be a medical image acquiring device, such as a magnetic resonance imaging device, a positron emission Tomography device, an electronic Computed Tomography (CT) device, and the like, and the medical imaging device generally transmits acquired medical images to the electronic device in real time, or transmits acquired medical images only when receiving an upload instruction of the electronic device, or transmits acquired medical images to a database for storage, and the medical imaging device is not strictly limited to provide only the second image to be predicted. The database may be an internal database (including local and cloud databases) or an external source database, and similarly, the database is not strictly limited to provide only the first image to be predicted, and may also provide the second image to be predicted. Research shows that although alzheimer's disease causes atrophy of the whole brain, the hippocampus (hippcampus) is the region that is damaged first, and in one embodiment, when alzheimer's disease prediction is performed, after the electronic device acquires an image of a brain region of a patient acquired by medical imaging equipment, the hippocampus region can be segmented out with high precision through a pre-constructed prediction model based on a medical image, and classification prediction results are output.
In addition, in the embodiment of the application, a model construction framework shown in fig. 2b is adopted to construct a prediction model based on a medical image, the network structure firstly carries out manual labeling and image segmentation on a sample image, corresponding labels are used as labeling results, and a deep neural network based on multi-target learning is trained by using the labeling results, the sample image and the image segmentation results, so that the prediction model based on the medical image is obtained.
Compared with the scheme of extracting the characteristics of artificial design and then training the classifier to predict and using the 3D-CNN model to predict, the prediction model based on the medical image constructed by the prediction model construction scheme based on the medical image provided by the embodiment of the application is more practical in the inspection or prediction scenes of Alzheimer's disease, frontotemporal dementia and the like.
Based on the application architecture shown in fig. 2a and the overall network structure shown in fig. 2b, an embodiment of the present application provides a method for constructing a prediction model based on a medical image, where the method for constructing a prediction model based on a medical image can be executed by an electronic device, and is specifically described with alzheimer's disease, please refer to fig. 3, and the method for constructing a prediction model based on a medical image can include the following steps:
and S31, obtaining training sample labeling results, wherein the training sample labeling results are obtained by performing classification labeling on the first to-be-predicted images of a plurality of sample users.
In this embodiment of the present application, the sample user refers to an object to which the first image to be predicted belongs, for example: the ADNI data set includes subjects to which the brain image data belongs, subjects to which the brain image data stored in the hospital laboratory belongs, and the like, including subjects who have or are about to suffer from alzheimer's disease in the near future, or subjects who do not have or do not suffer from alzheimer's disease in the near future, and the first image to be predicted is an image of a brain region of a sample user acquired by the medical imaging apparatus. The second classification labeling is to label the first to-be-predicted image which has or is about to be suffered from the alzheimer's disease in the near future with a specific label, label the first to-be-predicted image which does not have or is not to suffer from the alzheimer's disease in the near future with another specific label, for example, the first to-be-predicted image which has or is about to suffer from the alzheimer's disease in the near future is commonly labeled with 1, the first to-be-predicted image which does not have or is not to suffer from the alzheimer's disease in the near future is labeled with 0, the sample labeling result refers to the above labeling label of the first to-be-predicted image, and the labeling of the second classification is adopted to make the constructed prediction model based on the medical image only have the probability of.
And S32, performing image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted.
In an embodiment of the present application, the template image to be registered is a pre-selected standard image, and each region to be segmented in the brain of the template image is labeled with a segmentation tag, for example: if the first to-be-predicted image needs to be divided into three regions, a pixel point of each region is assigned with one of three values, namely 1, 2 and 3, and the assigned value of the pixel point is the division label of each region. The first segmentation result of the first to-be-predicted image refers to segmented images of all brain regions obtained by an image registration method, the first to-be-predicted image may only reflect the characteristics of the middle region of the brain due to different acquisition conditions, as shown in fig. 4, a rectangular frame may represent the whole brain region, an irregular figure in the frame may represent several preset regions to be segmented from the brain region, the first to-be-predicted image needs to be spatially transformed to accurately segment the preset regions from the first to-be-predicted image, the first to-be-predicted image is aligned with a template image to be registered, that is, image registration, and the registering the first to-be-predicted image with a sample image to be registered defines a similarity measure and finds a spatial transformation relation T (T is a matrix) so that the similarity of the two images after the spatial transformation is maximized, is expressed by the formula:
S(T)＝S(l1(x)，l0(T(x)))
where S represents the similarity measure, T represents the spatial transformation relationship, l1Indicates the first one to be expectedMeasuring an image, /)0And representing a standard image to be registered, x represents a pixel point in the image, and the registration process is a process of seeking optimal spatial transformation.
In one possible embodiment, image registration of the template image to be registered with the first image to be predicted includes:
extracting feature information of the template image to be registered and the first image to be predicted to form a feature space;
determining a spatial transformation relation T by adopting an optimization algorithm according to the extracted feature space, so that the similarity between the template image to be registered and the first image to be predicted after transformation reaches a preset value;
and transforming the segmentation labels on the template image to be registered to the first image to be predicted.
Specifically, the method for extracting the feature information of the first image to be predicted and the template image to be registered is not limited, and for example, the method includes: the extraction can be performed by using a U-shaped convolution neural network, or can also be performed by using a residual error network, and the like. A space in which all feature information of the template image to be registered and the first image to be predicted exist, i.e. a feature space, an optimization algorithm is adopted in the feature space to determine a spatial transformation relation T, so that the similarity between the template image to be registered and the first image to be predicted after transformation reaches a maximum value, wherein the process of determining the spatial transformation relation T can be as shown in fig. 5, firstly, an initial T is obtained, and then, for the first image to be predicted l1Carrying out spatial transformation to obtain a transformed image, and carrying out similarity measure evaluation between the transformed image and a template image to be registered by adopting an optimization algorithm, wherein the formula is as follows:
Figure BDA0002254022080000081
and finally, the original T is updated, and the segmentation labels of all the regions on the template image to be registered can be transformed to the first image to be predicted through the space transformation of the optimal T, so that the first segmentation result of the first image to be predicted is obtained. As shown in fig. 6, the first image to be predicted on the left side is subjected to image segmentation by an image registration method, and the obtained first segmentation result is the image shown on the right side.
And S33, inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to construct the medical image-based prediction model.
In this embodiment of the application, after the first segmentation results of all the first to-be-predicted images are obtained in step S32, the first to-be-predicted images, the training sample labeling result, and the first segmentation result of the first to-be-predicted images are input into a preset deep neural network based on multi-target learning to be trained, and a back propagation method (back propagation) is used to optimize network parameters until the network parameters are fixed, so as to obtain a constructed prediction model based on a medical image.
According to the embodiment of the application, the training sample labeling result is obtained by performing classification labeling on the first to-be-predicted images of a plurality of sample users; this enables the deep neural network based on multi-objective learning to quickly learn the classification when used for predicting Alzheimer's disease, namely, Alzheimer's disease or Alzheimer's disease which will be suffered or will be suffered recently, or Alzheimer's disease which is not suffered. Carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted; in this way, the first image to be predicted is segmented by adopting the image registration method, and a standard segmented image with higher segmentation precision can be obtained. Inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to construct the prediction model based on the medical image; therefore, the deep neural network based on multi-target learning is trained by adopting the easily-learned two-classification labeling result and the standard segmentation image with higher segmentation precision, and the prediction precision of the constructed prediction model based on the medical image can be improved. Therefore, the prediction model based on the medical image, which is constructed by performing classification labeling on the first to-be-predicted image and performing image segmentation by adopting image registration, is beneficial to improving the prediction accuracy of the Alzheimer's disease when used for predicting the Alzheimer's disease.
Referring to fig. 7, fig. 7 is a schematic flowchart of another medical image-based prediction model construction method according to an embodiment of the present application, and as shown in fig. 7, the method includes steps S71-S76:
s71, obtaining training sample labeling results, wherein the training sample labeling results are obtained by performing classification labeling on first to-be-predicted images of a plurality of sample users;
s72, carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted;
steps S71-S72 are already described in the embodiment shown in fig. 3, and are not repeated here to avoid repetition.
S73, inputting the first to-be-predicted image into a feature extraction sub-network of the deep neural network based on multi-target learning to perform down-sampling operation, extracting feature information of the first to-be-predicted image, inputting the extracted feature information into a classifier of the deep neural network based on multi-target learning to perform classification processing, and outputting a classification result;
s74, inputting the extracted feature information into an image segmentation sub-network of the deep neural network based on the multi-target learning for up-sampling operation, and outputting a second segmentation result of the first to-be-predicted image;
in the embodiment of the application, as shown in fig. 8, a preset deep neural network based on multi-target learning is used for constructing a prediction model based on a medical image, wherein the deep neural network based on multi-target learning comprises three parts, namely a feature extraction sub-network, an image segmentation sub-network and a classifier, the feature extraction sub-network is used for performing feature extraction on input to-be-segmented images, the image segmentation sub-network performs image segmentation on the basis of feature information extracted by the feature extraction sub-network, and the classifier performs classification prediction on the basis of feature information extracted by the feature extraction sub-network.
In one possible implementation, inputting the first image to be predicted into the feature extraction sub-network to perform a down-sampling operation, and extracting feature information of the first image to be predicted includes:
performing convolution processing on the first image to be predicted through a convolution block of the feature extraction sub-network;
performing maximum pooling on the feature map obtained through convolution processing through a maximum pooling layer of the feature extraction sub-network;
and extracting the characteristic information of the first to-be-predicted image through the alternate processing of M rolling blocks and N maximum pooling layers, wherein M is equal to N + 1.
As shown in fig. 8, the feature extraction sub-network is composed of 5 convolution blocks, each convolution block except for the last convolution block is followed by a maximum pooling layer (i.e. 4 maximum pooling layers), the first image to be predicted is first convolved by a convolution block to obtain a feature map, and then is maximally pooled by a maximum pooling layer, and the length, width and height of the obtained feature map are reduced to 1/2, so that the next convolution block can extract features in a larger receptive field range. Performing convolution processing on the 1/2 feature map with the original length, width and height reduced, to obtain another feature map, performing maximum pooling processing on the 1/2 feature map with the original length, width and height reduced, to obtain 1/2 feature map, and performing convolution processing on the last convolution block, to extract the high-dimensional feature information of the first image to be predicted, where the number of channels of the extracted features of each convolution block is twice that of the previous convolution block, for example: if the number of feature map channels extracted by the first convolution block is 64, the sequence is 128, 256, 512 and 1024.
The classifier of the deep neural network based on multi-target learning is composed of two fully-connected layers, high-dimensional feature information extracted by a feature extraction sub-network is input into the classifier, and before the classifier is input, the high-dimensional feature information is flattened, for example: and setting the length, width, height and channel number of the high-dimensional feature information as L, W, H, C respectively, wherein each feature value is unchanged after flattening, but the position is changed, so that a one-dimensional vector with the length of L multiplied by W multiplied by H multiplied by C is obtained, the one-dimensional vector is integrated into a value through convolution processing of a full connection layer of a classifier, and the value is output, so that the classification result of the first image to be predicted is obtained.
In a possible embodiment, inputting the extracted feature information into the image segmentation sub-network for an upsampling operation, and outputting a second segmentation result of the first image to be predicted, the method includes:
performing convolution processing on the feature information extracted by the feature extraction sub-network through an convolution layer;
merging the feature map obtained by the convolution processing with the feature map with the same size extracted by the convolution block of the feature extraction sub-network;
performing convolution processing on the feature map obtained by merging through a convolution block of the image segmentation sub-network;
and outputting a second segmentation result of the first to-be-predicted image through alternative processing of N upper convolution layers, N times of combination and N convolution blocks.
As shown in fig. 8, the image segmentation sub-network is composed of 4 convolution blocks, the 4 convolution blocks of the image segmentation sub-network correspond to the first 4 convolution blocks of the feature extraction sub-network one by one, the feature maps processed by the two corresponding convolution blocks have the same size, the input of the image segmentation sub-network is high-dimensional feature information obtained by the feature extraction sub-network, for the high-dimensional feature information, the length, the width and the height of the high-dimensional feature information are expanded to twice of the original length by an up-convolution layer (up-convolution layer), for the feature map obtained after the size expansion, since the maximum pooling operation of the maximum pooling layer in the feature extraction stage loses information of part of pixel points, in order to compensate the lost information, the feature map extracted by the corresponding convolution block of the feature extraction sub-network and having the same size is merged with the feature map obtained after the size expansion, and then the merged feature map is convolved by a convolution block, after 4 times of convolution-merge-convolution processing, the segmented image with the same size as the first to-be-predicted image, i.e. the second segmentation result of the first to-be-predicted image, is output by the last convolution block, and similarly, the number of channels of the extracted features of each convolution block is reduced to 1/2 of the previous one.
It should be noted that, the convolution blocks in the feature extraction sub-network and the image segmentation sub-network are each in a form of superposition of 4 convolution layers, the convolution kernel sizes of the 1 st layer and the 3 rd layer are 1 × 1 × 1, the convolution kernel sizes of the 2 nd layer and the 4 th layer are 3 × 3 × 3, each convolution layer uses relu (normalized linear unit) as an activation function, and the windows of maximum pooling and convolution are each 2 × 2 and the step size is 2. In order to improve the operation efficiency of the deep neural network based on the multi-target learning, when the convolution block in the embodiment of the present application performs convolution processing, the separable convolution (separable convolution) is adopted in the 2 nd layer and the 4 th layer, and one convolution operation is split into two steps to perform, for example, as shown in fig. 9, for a 12 × 12 × 3 feature map, first, features are extracted on the transverse plane by 3 convolution kernels with the size of 5 × 5 × 1 to obtain an 8 × 8 × 1 feature map, 3 channels are overlapped to obtain an 8 × 8 × 3 feature map, then, for the 8 × 8 × 3 feature map, convolution is performed by using a convolution kernel with 1 × 1 × 3 to obtain an 8 × 8 × 1 feature map, which reduces the original 5 × 5 × 3 × 75 parameters to 5+3 parameters 28 parameters, and if the feature map is 12 × 12 × 3, the embodiment of the present application first performs convolution by using a convolution kernel with 3 × 3 × 1 convolution kernel, the 10 × 10 × 3 feature map is obtained, then the 10 × 10 × 3 feature map is convolved by a 1 × 1 × 3 convolution kernel to obtain a 10 × 10 × 1 feature map, the original 3 × 3 × 3 × 3 ═ 27 parameters are reduced to 3 × 3+3 ═ 12 parameters, and the parameters of each convolution block are reduced by more than 50%, so that the calculation efficiency of the deep neural network based on the multi-target learning is greatly improved.
S75, constructing a loss function of the deep neural network based on the multi-target learning;
in an embodiment of the present application, the first image to be predicted is the brain region image of the sample user, and the constructing the loss function of the deep neural network based on the multi-target learning includes:
constructing a first loss function for the classification results of the brain region images of the sample user, the first loss function being defined as:
Lce＝-(ylog(p)+(1-y)log(1-p))
wherein L isceA value representing the first loss function, y represents a sample label, the sample label of the sample user who has or will not have alzheimer's disease within a preset time is 1, the sample label of the sample user who does not have and will not have alzheimer's disease within a preset time is 0, and p represents a probability that the classification result of the brain region image is a positive sample;
constructing a second loss function for a second segmentation result of the brain region image of the sample user, the second loss function defined as:
Figure BDA0002254022080000121
wherein L isdiceRepresenting the value of the second loss function, Ω representing a brain region in the brain region image, l representing a hippocampus subregion or other subregion in the brain region, S1(l) A first segmentation result, S, representing the image of the brain region2(l) A second segmentation result, ω, representing the image of the brain regionlRepresenting the weight of each sub-region in the brain region;
obtaining a loss function of the deep neural network based on the multi-target learning according to the first loss function and the second loss function:
Figure BDA0002254022080000122
wherein L represents a value of a loss function of the multi-objective learning based deep neural network,
Figure BDA0002254022080000123
and
Figure BDA0002254022080000124
is a preset coefficient.
The first predetermined condition can be understood as having or being suffered from alzheimer's disease in the near term, and the second predetermined condition can be understood as not having and not being suffered from alzheimer's disease in the near term, since the hippocampus (hipppampus) subregion is the first damaged region in each subregion of the brain, the hippocampus subregion is the key region for image segmentation, in the second loss function, we set the weight of the hippocampus subregion to 5, and the weights of the other subregions to 1. In addition, the embodiments of the present application will be described
Figure BDA0002254022080000132
The setting is 1 and the setting is carried out,and 2, the influence of the classification result on network training is improved, and the network is ensured to take the classification accuracy as a main target. And the weighted sum of the first loss function and the second loss function is the loss function of the whole deep neural network based on multi-target learning.
And S76, optimizing the network parameters of the deep neural network based on the multi-objective learning by adopting a gradient descent method based on Adam according to the constructed loss function until a classification result and a second segmentation result which accord with expectations are output.
In the specific embodiment of the application, parameters of the network are optimized by adopting an Adam-based gradient descent method in a back propagation algorithm, the initial learning rate is determined to be 0.001, every time 20 epochs are reduced to be one fifth, 1 epoch is used for completing one forward propagation operation and one back propagation operation on all first images to be predicted, the batch size is determined to be 5, namely, each iteration is trained by using 5 first to-be-predicted images, each iteration is one time of network parameter weight updating, each time of network parameter weight updating needs 5 first to-be-predicted images to carry out forward operation to obtain the value of the whole network loss function, the process of feature information extraction, classification processing and image segmentation is repeatedly executed, a classification result and a second segmentation result which are in line with expectations are obtained through 100 epochs of training, and the network parameter weight of the whole network is fixed, so that the well-constructed prediction model based on the medical image is obtained.
According to the method, training sample labeling results are obtained, and the training sample labeling results are obtained by performing classification labeling on first to-be-predicted images of a plurality of sample users; this enables the deep neural network based on multi-objective learning to quickly learn the classification when used for predicting Alzheimer's disease, namely, Alzheimer's disease or Alzheimer's disease which will be suffered or will be suffered recently, or Alzheimer's disease which is not suffered. Carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted; in this way, the first image to be predicted is segmented by adopting the image registration method, and a standard segmented image with higher segmentation precision can be obtained. Inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to construct the prediction model based on the medical image; therefore, the deep neural network based on multi-target learning is trained by adopting the easily-learned two-classification labeling result and the standard segmentation image with higher segmentation precision, and the prediction precision of the constructed prediction model based on the medical image can be improved. Therefore, the prediction model based on the medical image, which is constructed by performing classification labeling on the first to-be-predicted image and performing image segmentation by adopting image registration, is beneficial to improving the prediction accuracy of the Alzheimer's disease when used for predicting the Alzheimer's disease.
Referring to fig. 10, fig. 10 is a medical image-based prediction method provided in an embodiment of the present application, where the medical image-based prediction method can be implemented by using a medical image-based prediction model constructed in the embodiment of fig. 3 or fig. 7, and the method includes steps S1001-S1004:
s1001, acquiring an original medical image, and aligning and normalizing the original medical image to obtain a second image to be predicted;
in the embodiment of the application, an original medical image is an image of a brain region of an acquired object acquired by medical imaging equipment, the acquired object may be an experimental object in a medical laboratory or a patient in a hospital, after the original medical image is acquired, the original medical image is aligned and normalized to obtain a second image to be predicted with a better effect, and the second image to be predicted is an image to be classified and predicted by using a prediction model based on the medical image.
S1002, inputting the second image to be predicted into a pre-constructed feature extraction sub-network based on a prediction model of a medical image to perform down-sampling operation, and extracting feature information of the second image to be predicted;
in the embodiment of the present application, based on the network structure shown in fig. 8, similarly, a convolution block of the feature extraction subnetwork is used to perform convolution processing on the second image to be predicted, then a maximum pooling layer of the feature extraction subnetwork is used to perform maximum pooling processing on the feature map obtained through the convolution processing, the length, width and height of the obtained feature map are reduced to 1/2, and feature information of the second image to be predicted is extracted through alternative processing of 5 convolution blocks and 4 maximum pooling layers.
S1003, performing up-sampling operation on the second image to be predicted through the image segmentation sub-network based on the prediction model of the medical image, and outputting an image segmentation result of the second image to be predicted;
in the embodiment of the present application, based on the network structure shown in fig. 8, the feature information of the second image to be predicted extracted by the feature extraction sub-network is also first subjected to the convolution processing by one convolution layer, then the feature map obtained by the convolution processing is merged with the feature map with the same size extracted by the convolution block of the feature extraction sub-network, the feature map obtained by merging is subjected to the convolution processing by one convolution block of the image segmentation sub-network, and the segmented image with the same size as the second image to be predicted is output as the image segmentation result of the second image to be predicted after the alternation processing of 4 convolution layers, 4 times of merging and 4 convolution blocks.
And S1004, classifying the feature information of the second image to be predicted through the classifier based on the prediction model of the medical image to obtain the prediction result of the second image to be predicted.
In the embodiment of the application, the extracted feature information of the second image to be predicted is flattened to obtain a one-dimensional feature vector as the input of the classifier, the one-dimensional feature vector is integrated into a value through the processing of two full connection layers of the classifier, the value can be the probability p (a) of the label a suffering from or about to suffer from alzheimer's disease in the near future or the probability p (b) of the label b under other conditions, and the output value is the prediction result. It should be noted that the feature information extraction and the image segmentation in the present embodiment also take the form of separable convolution.
According to the method, a second image to be predicted is obtained by obtaining an original medical image and carrying out alignment and normalization processing on the original medical image; inputting the second image to be predicted into a pre-constructed feature extraction sub-network based on a prediction model of a medical image to perform down-sampling operation, and extracting feature information of the second image to be predicted; performing an upsampling operation on the second image to be predicted through an image segmentation subnetwork of the medical image-based prediction model, and outputting an image segmentation result of the second image to be predicted; and classifying the characteristic information of the second image to be predicted through the classifier of the prediction model based on the medical image to obtain the prediction result of the second image to be predicted. Because the prediction model based on the medical image adopts the binary labeling to obtain the training samples in the training stage and adopts the image registration method to perform accurate image segmentation, and the network parameters of the deep neural network based on the multi-target learning are optimized through 100 epochs of training, the prediction model based on the medical image is used for predicting the Alzheimer's disease, thereby being beneficial to improving the prediction precision of the Alzheimer's disease.
Based on the description of the embodiment of the medical image-based prediction model construction method, the embodiment of the present application further provides a medical image-based prediction model construction device, which may be a computer program (including a program code) running in a terminal. The medical image-based prediction model construction apparatus may perform the method shown in fig. 3 or fig. 7. Referring to fig. 11, the apparatus includes:
the sample labeling module 1101 is configured to obtain a training sample labeling result, where the training sample labeling result is obtained by performing classification labeling on a first to-be-predicted image of a plurality of sample users;
an image registration module 1102, configured to perform image registration on a template image to be registered and the first image to be predicted, so as to obtain a first segmentation result of the first image to be predicted;
a model building module 1103, configured to input the first to-be-predicted image, the training sample labeling result, and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training, so as to build the prediction model based on the medical image.
In a possible implementation, the image registration module 1102, in terms of performing image registration on the template image to be registered and the first image to be predicted, is specifically configured to:
extracting feature information of the template image to be registered and the first image to be predicted to form a feature space;
determining a spatial transformation relation T by adopting an optimization algorithm according to the extracted feature space, so that the similarity between the template image to be registered and the first image to be predicted after transformation reaches a preset value;
and transforming the segmentation labels on the template image to be registered to the first image to be predicted.
In one possible implementation, the multi-target learning based deep neural network comprises a feature extraction sub-network, an image segmentation sub-network and a classifier; the model building module 1103 is specifically configured to, in inputting the first to-be-predicted image, the training sample labeling result, and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training:
inputting the first image to be predicted into the feature extraction sub-network for down-sampling operation, extracting feature information of the first image to be predicted, inputting the extracted feature information into the classifier for classification, and outputting a classification result;
inputting the extracted feature information into the image segmentation sub-network for up-sampling operation, and outputting a second segmentation result of the first image to be predicted;
constructing a loss function of the deep neural network based on the multi-target learning;
and optimizing the network parameters of the deep neural network based on the multi-objective learning by adopting a gradient descent method based on Adam according to the constructed loss function until a classification result and a second segmentation result which accord with expectations are output.
In a possible implementation manner, the model building module 1103, in inputting the first image to be predicted into the feature extraction sub-network for down-sampling operation, and extracting feature information of the first image to be predicted, is specifically configured to:
performing convolution processing on the first image to be predicted through a convolution block of the feature extraction sub-network;
performing maximum pooling on the feature map obtained through convolution processing through a maximum pooling layer of the feature extraction sub-network;
and extracting the characteristic information of the first to-be-predicted image through the alternate processing of M rolling blocks and N maximum pooling layers, wherein M is equal to N + 1.
In a possible implementation manner, the model building module 1103, in inputting the extracted feature information into the image segmentation sub-network for upsampling, and outputting the second segmentation result of the first image to be predicted, is specifically configured to:
performing convolution processing on the feature information extracted by the feature extraction sub-network through an convolution layer;
merging the feature map obtained by the convolution processing with the feature map with the same size extracted by the convolution block of the feature extraction sub-network;
performing convolution processing on the feature map obtained by merging through a convolution block of the image segmentation sub-network;
and outputting a second segmentation result of the first to-be-predicted image through alternative processing of N upper convolution layers, N times of combination and N convolution blocks.
In a possible implementation manner, the first to-be-predicted image is an image of a brain region of the sample user, and the model building module 1103 is specifically configured to, in terms of building a loss function of the deep neural network based on multi-target learning:
constructing a first loss function for the classification results of the brain region images of the sample user, the first loss function being defined as:
Lce＝-(ylog(p)+(1-y)log(1-p))
wherein L isceA value representing the first loss function, y represents a sample label,
the sample label of the sample user who suffers from or will suffer from alzheimer's disease within a preset time is 1, the sample label of the sample user who does not suffer from and will not suffer from alzheimer's disease within a preset time is 0, and p represents the probability that the classification result of the second brain region image is a positive sample;
constructing a second loss function for a second segmentation result of the brain region image of the sample user, the second loss function defined as:
Figure BDA0002254022080000171
wherein L isdiceRepresenting the value of the second loss function, Ω representing a brain region in the brain region image, l representing a hippocampus subregion or other subregion in the brain region, S1(l) A first segmentation result, S, representing the image of the brain region2(l) A second segmentation result, ω, representing the image of the brain regionlRepresenting the weight of each sub-region in the brain region;
obtaining a loss function of the deep neural network based on the multi-target learning according to the first loss function and the second loss function:
Figure BDA0002254022080000181
wherein L represents a value of a loss function of the multi-objective learning based deep neural network,and
Figure BDA0002254022080000183
is a preset coefficient.
Based on the above description of the embodiment of the prediction method based on the medical image, an embodiment of the present application further provides a prediction apparatus based on the medical image, please refer to fig. 12, and the apparatus includes:
an image obtaining module 1201, configured to obtain an original medical image, and perform alignment and normalization processing on the original medical image to obtain a second image to be predicted;
a feature extraction module 1202, configured to input the second image to be predicted into a pre-constructed feature extraction sub-network based on a prediction model of a medical image to perform downsampling operation, and extract feature information of the second image to be predicted;
an image segmentation module 1203, configured to perform an upsampling operation on the second image to be predicted through an image segmentation subnetwork of the medical image-based prediction model, and output an image segmentation result of the second image to be predicted;
a result prediction module 1204, configured to perform classification processing on the feature information of the second image to be predicted through the classifier based on the prediction model of the medical image, so as to obtain a prediction result of the second image to be predicted.
According to an embodiment of the present application, the units in the medical image-based prediction model building apparatus shown in fig. 11 and the medical image-based prediction apparatus shown in fig. 12 may be respectively or entirely combined into one or several additional units to form the prediction model, or some of the unit(s) may be further split into a plurality of functionally smaller units to form the prediction model, which may achieve the same operation without affecting the achievement of the technical effect of the embodiment of the present invention. The units are divided based on logic functions, and in practical application, the functions of one unit can be realized by a plurality of units, or the functions of a plurality of units can be realized by one unit. In other embodiments of the present invention, the medical image-based prediction model building device and the medical image-based prediction device may also include other units, and in practical applications, these functions may also be implemented by the assistance of other units, and may be implemented by cooperation of a plurality of units.
According to another embodiment of the present application, the apparatus device shown in fig. 11 or fig. 12 may be constructed by running a computer program (including program codes) capable of executing steps involved in the respective methods shown in fig. 3, fig. 7 or fig. 10 on a general-purpose computing device such as a computer including a processing element such as a Central Processing Unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM) or the like, and a storage element, and the above-described method of the embodiment of the present invention may be implemented. The computer program may be recorded on a computer-readable recording medium, for example, and loaded and executed in the above-described computing apparatus via the computer-readable recording medium.
Based on the description of the method embodiment and the device embodiment, the embodiment of the invention also provides electronic equipment. Referring to fig. 13, the electronic device includes at least a processor 1301, an input device 1302, an output device 1303, and a computer storage medium 1304. The processor 1301, the input device 1302, the output device 1303, and the computer storage medium 1304 in the electronic device may be connected by a bus or other means.
A computer storage medium 1304 may be stored in the memory of the electronic device, the computer storage medium 1304 being for storing a computer program comprising program instructions, the processor 1301 being for executing the program instructions stored by the computer storage medium 1304. The processor 1301 (or CPU) is a computing core and a control core of the electronic device, and is adapted to implement one or more instructions, and in particular, to load and execute the one or more instructions so as to implement a corresponding method flow or a corresponding function.
In one embodiment, the processor 1301 of the electronic device provided in the embodiment of the present application may be configured to perform a series of prediction model building processes based on medical images, including:
obtaining training sample labeling results, wherein the training sample labeling results are obtained by performing classification labeling on first to-be-predicted images of a plurality of sample users;
carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted;
inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to construct the prediction model based on the medical image.
In one embodiment, the processor 1301 performs the image registration of the template image to be registered and the first image to be predicted, including:
extracting feature information of the template image to be registered and the first image to be predicted to form a feature space;
determining a spatial transformation relation T by adopting an optimization algorithm according to the extracted feature space, so that the similarity between the template image to be registered and the first image to be predicted after transformation reaches a preset value;
and transforming the segmentation labels on the template image to be registered to the first image to be predicted.
In one embodiment, the multi-objective learning based deep neural network comprises a feature extraction sub-network, an image segmentation sub-network and a classifier; the processor 1301 inputs the first to-be-predicted image, the training sample labeling result, and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training, including:
inputting the first image to be predicted into the feature extraction sub-network for down-sampling operation, extracting feature information of the first image to be predicted, inputting the extracted feature information into the classifier for classification, and outputting a classification result;
inputting the extracted feature information into the image segmentation sub-network for up-sampling operation, and outputting a second segmentation result of the first image to be predicted;
constructing a loss function of the deep neural network based on the multi-target learning;
and optimizing the network parameters of the deep neural network based on the multi-objective learning by adopting a gradient descent method based on Adam according to the constructed loss function until a classification result and a second segmentation result which accord with expectations are output.
In one embodiment, the processor 1301 performs the down-sampling operation of inputting the first image to be predicted into the feature extraction sub-network, and extracts feature information of the first image to be predicted, including:
performing convolution processing on the first image to be predicted through a convolution block of the feature extraction sub-network;
performing maximum pooling on the feature map obtained through convolution processing through a maximum pooling layer of the feature extraction sub-network;
and extracting the characteristic information of the first to-be-predicted image through the alternate processing of M rolling blocks and N maximum pooling layers, wherein M is equal to N + 1.
In one embodiment, the processor 1301 performs the operation of inputting the extracted feature information into the image segmentation sub-network for upsampling, and outputting the second segmentation result of the first image to be predicted, including:
performing convolution processing on the feature information extracted by the feature extraction sub-network through an convolution layer;
merging the feature map obtained by the convolution processing with the feature map with the same size extracted by the convolution block of the feature extraction sub-network;
performing convolution processing on the feature map obtained by merging through a convolution block of the image segmentation sub-network;
outputting a second segmentation result of the first to-be-predicted image through alternative processing of N upper convolution layers, N times of combination and N convolution blocks, wherein the second segmentation result comprises the following steps:
performing convolution processing on the feature information extracted by the feature extraction sub-network through an convolution layer;
merging the feature map obtained by the convolution processing with the feature map with the same size extracted by the convolution block of the feature extraction sub-network;
performing convolution processing on the feature map obtained by merging through a convolution block of the image segmentation sub-network;
and outputting a second segmentation result of the first to-be-predicted image through alternative processing of N upper convolution layers, N times of combination and N convolution blocks.
In one embodiment, the first to-be-predicted image is a brain region image of the sample user, and the processor 1301 performs the constructing the loss function of the deep neural network based on multi-target learning, including:
constructing a first loss function for the classification results of the brain region images of the sample user, the first loss function being defined as:
Lce＝-(ylog(p)+(1-y)log(1-p))
wherein L isceA value representing the first loss function, y represents a sample label, the sample label of the sample user who has or will not have alzheimer's disease within a preset time is 1, the sample label of the sample user who does not have and will not have alzheimer's disease within a preset time is 0, and p represents a probability that the classification result of the second brain region image is a positive sample;
constructing a second loss function for a second segmentation result of the brain region image of the sample user, the second loss function defined as:
Figure BDA0002254022080000211
wherein L isdiceRepresenting the value of the second loss function, Ω representing a brain region in the brain region image, l representing a hippocampus subregion or other subregion in the brain region, S1(l) A first segmentation result, S, representing the image of the brain region2(l) A second segmentation result, ω, representing the image of the brain regionlRepresenting the weight of each sub-region in the brain region;
obtaining a loss function of the deep neural network based on the multi-target learning according to the first loss function and the second loss function:
wherein L represents a value of a loss function of the multi-objective learning based deep neural network,
Figure BDA0002254022080000221
and
Figure BDA0002254022080000222
is a preset coefficient.
In one embodiment, the processor 1301 of the electronic device provided in the embodiment of the present application may be configured to perform a series of prediction processes based on medical images, including:
acquiring an original medical image, and aligning and normalizing the original medical image to obtain a second image to be predicted;
inputting the second image to be predicted into a pre-constructed feature extraction sub-network based on a prediction model of a medical image to perform down-sampling operation, and extracting feature information of the second image to be predicted;
performing an upsampling operation on the second image to be predicted through an image segmentation subnetwork of the medical image-based prediction model, and outputting an image segmentation result of the second image to be predicted;
and classifying the characteristic information of the second image to be predicted through the classifier of the prediction model based on the medical image to obtain the prediction result of the second image to be predicted.
Illustratively, the electronic device may be a computer, a notebook computer, a tablet computer, a palm computer, a server, or the like. Electronic devices may include, but are not limited to, a processor 1301, an input device 1302, an output device 1303, and a computer storage medium 1304. It will be appreciated by those skilled in the art that the schematic diagrams are merely examples of an electronic device and are not limiting of an electronic device and may include more or fewer components than those shown, or some components in combination, or different components.
It should be noted that, since the processor 1301 of the electronic device executes the computer program to implement the steps of the prediction model construction method based on the medical image and the prediction method based on the medical image, the embodiments of the prediction model construction method based on the medical image and the prediction method based on the medical image are both applicable to the electronic device, and can achieve the same or similar beneficial effects.
An embodiment of the present application further provides a computer storage medium (Memory), which is a Memory device in an electronic device and is used to store programs and data. It is understood that the computer storage medium herein may include a built-in storage medium in the terminal, and may also include an extended storage medium supported by the terminal. The computer storage medium provides a storage space that stores an operating system of the terminal. Also stored in the memory space are one or more instructions, which may be one or more computer programs (including program code), suitable for loading and execution by processor 1301. The computer storage medium may be a high-speed RAM memory, or may be a non-volatile memory (non-volatile memory), such as at least one disk memory; optionally, there may be at least one computer storage medium located remotely from the processor 1301. In one embodiment, one or more instructions stored in a computer storage medium may be loaded and executed by processor 1301 to perform the corresponding steps described above with respect to the medical image based prediction model construction method and the medical image based prediction method.
It should be noted that, since the computer program of the computer storage medium is executed by the processor to implement the steps of the medical image-based prediction model construction method and the medical image-based prediction method, all the embodiments or implementations of the medical image-based prediction model construction method and the medical image-based prediction method are applicable to the computer-readable storage medium, and can achieve the same or similar beneficial effects.
The foregoing detailed description of the embodiments of the present application has been presented to illustrate the principles and implementations of the present application, and the above description of the embodiments is only provided to help understand the method and the core concept of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims (13)

1. A medical image-based prediction model construction method is characterized by comprising the following steps:
obtaining training sample labeling results, wherein the training sample labeling results are obtained by performing classification labeling on first to-be-predicted images of a plurality of sample users;
carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted;
inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to construct the prediction model based on the medical image.
2. The method according to claim 1, wherein the image registering the template image to be registered with the first image to be predicted comprises:
extracting feature information of the template image to be registered and the first image to be predicted to form a feature space;
determining a spatial transformation relation T by adopting an optimization algorithm according to the extracted feature space, so that the similarity between the template image to be registered and the first image to be predicted after transformation reaches a preset value;
and transforming the segmentation labels on the template image to be registered to the first image to be predicted.
3. The method of claim 1, wherein the multi-objective learning based deep neural network comprises a feature extraction sub-network, an image segmentation sub-network, and a classifier; the inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training comprises:
inputting the first image to be predicted into the feature extraction sub-network for down-sampling operation, extracting feature information of the first image to be predicted, inputting the extracted feature information into the classifier for classification, and outputting a classification result;
inputting the extracted feature information into the image segmentation sub-network for up-sampling operation, and outputting a second segmentation result of the first image to be predicted;
constructing a loss function of the deep neural network based on the multi-target learning;
and optimizing the network parameters of the deep neural network based on the multi-objective learning by adopting a gradient descent method based on Adam according to the constructed loss function until a classification result and a second segmentation result which accord with expectations are output.
4. The method according to claim 3, wherein said inputting the first image to be predicted into the feature extraction sub-network for a down-sampling operation to extract feature information of the first image to be predicted comprises:
performing convolution processing on the first image to be predicted through a convolution block of the feature extraction sub-network;
performing maximum pooling on the feature map obtained through convolution processing through a maximum pooling layer of the feature extraction sub-network;
and extracting the characteristic information of the first to-be-predicted image through the alternate processing of M rolling blocks and N maximum pooling layers, wherein M is equal to N + 1.
5. The method according to claim 3, wherein the inputting the extracted feature information into the image segmentation sub-network for upsampling and outputting the second segmentation result of the first image to be predicted comprises:
performing convolution processing on the feature information extracted by the feature extraction sub-network through an convolution layer;
merging the feature map obtained by the convolution processing with the feature map with the same size extracted by the convolution block of the feature extraction sub-network;
performing convolution processing on the feature map obtained by merging through a convolution block of the image segmentation sub-network;
and outputting a second segmentation result of the first to-be-predicted image through alternative processing of N upper convolution layers, N times of combination and N convolution blocks.
6. The method according to any one of claims 3 to 5, wherein the first image to be predicted is an image of the brain region of the sample user, and the constructing the loss function of the deep neural network based on multi-objective learning comprises:
constructing a first loss function for the classification results of the brain region images of the sample user, the first loss function being defined as:
Lce＝-(ylog(p)+(1-y)log(1-p))
wherein L isceA value representing the first loss function, y represents a sample label, the sample user has or is presetA sample label which can suffer from the alzheimer disease in the interim is 1, a sample label which does not suffer from the alzheimer disease in the sample user within a preset time is 0, and p represents the probability that the classification result of the second brain region image is a positive sample;
constructing a second loss function for a second segmentation result of the brain region image of the sample user, the second loss function defined as:
Figure FDA0002254022070000031
wherein L isdiceRepresenting the value of the second loss function, Ω representing a brain region in the brain region image, l representing a hippocampus subregion or other subregion in the brain region, S1(l) A first segmentation result, S, representing the image of the brain region2(l) A second segmentation result, ω, representing the image of the brain regionlRepresenting the weight of each sub-region in the brain region;
obtaining a loss function of the deep neural network based on the multi-target learning according to the first loss function and the second loss function:
Figure FDA0002254022070000032
wherein L represents a value of a loss function of the multi-objective learning based deep neural network,and
Figure FDA0002254022070000034
is a preset coefficient.
7. A method for medical image-based prediction using a medical image-based prediction model constructed according to the method of any one of claims 1 to 6, the method comprising:
acquiring an original medical image, and aligning and normalizing the original medical image to obtain a second image to be predicted;
inputting the second image to be predicted into a pre-constructed feature extraction sub-network based on a prediction model of a medical image to perform down-sampling operation, and extracting feature information of the second image to be predicted;
performing an upsampling operation on the second image to be predicted through an image segmentation subnetwork of the medical image-based prediction model, and outputting an image segmentation result of the second image to be predicted;
and classifying the characteristic information of the second image to be predicted through the classifier of the prediction model based on the medical image to obtain the prediction result of the second image to be predicted.
8. An apparatus for constructing a prediction model based on a medical image, the apparatus comprising:
the system comprises a sample marking module, a data processing module and a data processing module, wherein the sample marking module is used for obtaining training sample marking results, and the training sample marking results are obtained by performing classification marking on first to-be-predicted images of a plurality of sample users;
the image registration module is used for carrying out image registration on the template image to be registered and the first image to be predicted to obtain a first segmentation result of the first image to be predicted;
and the model building module is used for inputting the first to-be-predicted image, the training sample labeling result and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training so as to build the prediction model based on the medical image.
9. The apparatus according to claim 8, wherein the image registration module, in image registration of the template image to be registered with the first image to be predicted, is specifically configured to:
extracting feature information of the template image to be registered and the first image to be predicted to form a feature space;
determining a spatial transformation relation T by adopting an optimization algorithm according to the extracted feature space, so that the similarity between the template image to be registered and the first image to be predicted after transformation reaches a preset value;
and transforming the segmentation labels on the template image to be registered to the first image to be predicted.
10. The apparatus of claim 8, wherein the multi-objective learning based deep neural network comprises a feature extraction sub-network, an image segmentation sub-network, and a classifier; the model building module is specifically configured to, in the aspect of inputting the first to-be-predicted image, the training sample labeling result, and the first segmentation result of the first to-be-predicted image into a preset deep neural network based on multi-target learning for training:
inputting the first image to be predicted into the feature extraction sub-network for down-sampling operation, extracting feature information of the first image to be predicted, inputting the extracted feature information into the classifier for classification, and outputting a classification result;
inputting the extracted feature information into the image segmentation sub-network for up-sampling operation, and outputting a second segmentation result of the first image to be predicted;
constructing a loss function of the deep neural network based on the multi-target learning;
and optimizing the network parameters of the deep neural network based on the multi-objective learning by adopting a gradient descent method based on Adam according to the constructed loss function until a classification result and a second segmentation result which accord with expectations are output.
11. A medical image-based prediction apparatus, characterized in that the apparatus comprises:
the image acquisition module is used for acquiring an original medical image, and aligning and normalizing the original medical image to obtain a second image to be predicted;
the feature extraction module is used for inputting the second image to be predicted into a pre-constructed feature extraction sub-network based on a prediction model of a medical image to perform down-sampling operation, and extracting feature information of the second image to be predicted;
the image segmentation module is used for performing upsampling operation on the second image to be predicted through an image segmentation subnetwork of the medical image-based prediction model and outputting an image segmentation result of the second image to be predicted;
and the result prediction module is used for classifying the characteristic information of the second image to be predicted through the classifier of the medical image-based prediction model to obtain the prediction result of the second image to be predicted.
12. An electronic device comprising an input device and an output device, further comprising:
a processor adapted to implement one or more instructions; and the number of the first and second groups,
a computer storage medium having one or more instructions stored thereon, the one or more instructions adapted to be loaded by the processor and to perform the method of any of claims 1-7.
13. A computer storage medium having stored thereon one or more instructions adapted to be loaded by a processor and to perform the method of any of claims 1-7.
CN201911059064.8A 2019-10-30 2019-10-30 Medical image-based prediction model construction method, prediction method and device Pending CN110838108A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911059064.8A CN110838108A (en) 2019-10-30 2019-10-30 Medical image-based prediction model construction method, prediction method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911059064.8A CN110838108A (en) 2019-10-30 2019-10-30 Medical image-based prediction model construction method, prediction method and device

Publications (1)

Publication Number Publication Date
CN110838108A true CN110838108A (en) 2020-02-25

Family

ID=69575840

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911059064.8A Pending CN110838108A (en) 2019-10-30 2019-10-30 Medical image-based prediction model construction method, prediction method and device

Country Status (1)

Country Link
CN (1) CN110838108A (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111488925A (en) * 2020-04-07 2020-08-04 北京百度网讯科技有限公司 Data labeling method and device, electronic equipment and storage medium
CN111738363A (en) * 2020-07-24 2020-10-02 温州大学 Alzheimer disease classification method based on improved 3D CNN network
CN112132841A (en) * 2020-09-22 2020-12-25 上海交通大学 Medical image cutting method and device
CN112734726A (en) * 2020-09-29 2021-04-30 首都医科大学附属北京天坛医院 Typing method, device and equipment for angiography
CN112802036A (en) * 2021-03-16 2021-05-14 上海联影医疗科技股份有限公司 Method, system and device for segmenting target area of three-dimensional medical image
CN113269774A (en) * 2021-06-09 2021-08-17 西南交通大学 Parkinson disease classification and lesion region labeling method of MRI (magnetic resonance imaging) image
CN113487656A (en) * 2021-07-26 2021-10-08 推想医疗科技股份有限公司 Image registration method and device, training method and device, control method and device
CN114708283A (en) * 2022-04-21 2022-07-05 推想医疗科技股份有限公司 Image object segmentation method and device, electronic equipment and storage medium
CN114937171A (en) * 2022-05-11 2022-08-23 复旦大学 Alzheimer's classification system based on deep learning

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111488925B (en) * 2020-04-07 2023-10-20 北京百度网讯科技有限公司 Data labeling method and device, electronic equipment and storage medium
CN111488925A (en) * 2020-04-07 2020-08-04 北京百度网讯科技有限公司 Data labeling method and device, electronic equipment and storage medium
CN111738363A (en) * 2020-07-24 2020-10-02 温州大学 Alzheimer disease classification method based on improved 3D CNN network
CN112132841A (en) * 2020-09-22 2020-12-25 上海交通大学 Medical image cutting method and device
CN112132841B (en) * 2020-09-22 2024-04-09 上海交通大学 Medical image cutting method and device
CN112734726A (en) * 2020-09-29 2021-04-30 首都医科大学附属北京天坛医院 Typing method, device and equipment for angiography
CN112734726B (en) * 2020-09-29 2024-02-02 首都医科大学附属北京天坛医院 Angiography typing method, angiography typing device and angiography typing equipment
CN112802036A (en) * 2021-03-16 2021-05-14 上海联影医疗科技股份有限公司 Method, system and device for segmenting target area of three-dimensional medical image
CN113269774A (en) * 2021-06-09 2021-08-17 西南交通大学 Parkinson disease classification and lesion region labeling method of MRI (magnetic resonance imaging) image
CN113269774B (en) * 2021-06-09 2022-04-26 西南交通大学 Parkinson disease classification and lesion region labeling method of MRI (magnetic resonance imaging) image
CN113487656A (en) * 2021-07-26 2021-10-08 推想医疗科技股份有限公司 Image registration method and device, training method and device, control method and device
CN113487656B (en) * 2021-07-26 2022-11-22 推想医疗科技股份有限公司 Image registration method and device, training method and device, control method and device
CN114708283A (en) * 2022-04-21 2022-07-05 推想医疗科技股份有限公司 Image object segmentation method and device, electronic equipment and storage medium
CN114937171B (en) * 2022-05-11 2023-06-09 复旦大学 Deep learning-based Alzheimer's classification system
CN114937171A (en) * 2022-05-11 2022-08-23 复旦大学 Alzheimer's classification system based on deep learning

Similar Documents

Publication Publication Date Title
CN110838108A (en) Medical image-based prediction model construction method, prediction method and device
CN109345575B (en) Image registration method and device based on deep learning
Zhang et al. Detecting anatomical landmarks from limited medical imaging data using two-stage task-oriented deep neural networks
WO2020228446A1 (en) Model training method and apparatus, and terminal and storage medium
Karasawa et al. Deep 3d convolutional neural network architectures for alzheimer’s disease diagnosis
CN108564127A (en) Image conversion method, device, computer equipment and storage medium
CN111429460B (en) Image segmentation method, image segmentation model training method, device and storage medium
CN110838125B (en) Target detection method, device, equipment and storage medium for medical image
CN113728335A (en) Method and system for classification and visualization of 3D images
CN110415250B (en) Overlapped chromosome segmentation method and device based on deep learning
CN111738363B (en) Alzheimer disease classification method based on improved 3D CNN network
CN110930378B (en) Emphysema image processing method and system based on low data demand
Abed et al. A modern deep learning framework in robot vision for automated bean leaves diseases detection
CN111080658A (en) Cervical MRI image segmentation method based on deformable registration and DCNN
CN113284088A (en) CSM image segmentation method, device, terminal equipment and storage medium
CN114445356A (en) Multi-resolution-based full-field pathological section image tumor rapid positioning method
CN115423754A (en) Image classification method, device, equipment and storage medium
CN114037699B (en) Pathological image classification method, equipment, system and storage medium
CN107590806B (en) Detection method and system based on brain medical imaging
CN111582449B (en) Training method, device, equipment and storage medium of target domain detection network
Tang et al. Lesion segmentation and RECIST diameter prediction via click-driven attention and dual-path connection
CN111210398A (en) White blood cell recognition system based on multi-scale pooling
CN113822323A (en) Brain scanning image identification processing method, device, equipment and storage medium
CN113111879B (en) Cell detection method and system
CN113379770B (en) Construction method of nasopharyngeal carcinoma MR image segmentation network, image segmentation method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40021479

Country of ref document: HK

SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination