CN114882996A - Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning - Google Patents

Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning Download PDF

Info

Publication number
CN114882996A
CN114882996A CN202210265257.4A CN202210265257A CN114882996A CN 114882996 A CN114882996 A CN 114882996A CN 202210265257 A CN202210265257 A CN 202210265257A CN 114882996 A CN114882996 A CN 114882996A
Authority
CN
China
Prior art keywords
prediction
feature map
mvi
feature
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210265257.4A
Other languages
Chinese (zh)
Other versions
CN114882996B (en
Inventor
黄炳升
林楚旋
陈嘉
陈佳兆
龙廷玉
陈玉莹
冯仕庭
王猛
周小琦
董帜
王霁朏
彭振鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen University
Original Assignee
Shenzhen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen University filed Critical Shenzhen University
Priority to CN202210265257.4A priority Critical patent/CN114882996B/en
Publication of CN114882996A publication Critical patent/CN114882996A/en
Application granted granted Critical
Publication of CN114882996B publication Critical patent/CN114882996B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • Pathology (AREA)
  • Computational Linguistics (AREA)
  • Primary Health Care (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Epidemiology (AREA)
  • Databases & Information Systems (AREA)
  • Biophysics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

The application discloses a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, which comprises the following steps: controlling a first feature extraction module, a second feature extraction module and a third feature extraction module to determine a first feature map, a second feature and a third feature map based on the MR image to be predicted; and the control prediction module determines a CK19 expression type and an MVI type of the MR image based on the first feature map, the second feature map and the third feature map. According to the method and the device, the abundant image characteristics carried by the MR image to be predicted are directly extracted through the prediction network model, and the problem that the prediction performance of the model is influenced due to subjectivity in image characteristic analysis can be avoided. Meanwhile, the CK19 expression features are extracted through the first feature extraction module, the shared features of CK19 expression and MVI are extracted through the second feature extraction module, the MVI features are extracted through the third feature extraction module, task parameters of a shared task and task parameters of an independent task are separated, and the problem that prediction results are unstable due to complex parameter sharing can be solved.

Description

Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning
Technical Field
The application relates to the technical field of biomedical engineering, in particular to a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning.
Background
Primary liver cancer is the sixth most common cancer worldwide, and the incidence rate thereof is on the rise worldwide. Hepatocellular carcinoma is the most common one of primary liver cancers, accounting for 90% of all primary liver malignancies. Hepatocellular Carcinoma (HCC) has high mortality and strong heterogeneity, and has significant global differences in morbidity and mortality. Surgical resection is currently the primary treatment of choice for HCC, however, most HCC patients treated by surgical resection have a recurrence of neogenetic tumors within 2 years. Among the many factors associated with the early recurrence of HCC post-surgery, positive expression of Cytokeratin19 (cytokine 19, CK19) and Microvascular Invasion (MVI) are important predictors of early recurrence and poor prognosis of HCC, and these two factors have a correlation, with HCC positive expression of CK19 being more prone to MVI.
HCC positively expressed by CK19 is more likely to have postoperative recurrent metastasis than HCC negatively expressed by CK19, and CK19 positive expression indicates that HCC is strongly invasive and is one of independent factors of HCC poor prognosis. When HCC patients are found to be at a high risk of poor prognosis, physicians need to reduce the postoperative recurrence rate by expanding the resection range of the tumor. Therefore, preoperatively predicting CK19 expression of liver cancer is important for making treatment strategies and judging prognosis of patients. According to literature reports, the risk of postoperative recurrence of HCC patients with MVI is high, and the MVI is an important prediction factor of HCC poor prognosis. Therefore, the preoperative diagnosis of the liver cancer MVI has important guiding function in the aspects of selection of treatment modes, prognosis and the like of liver cancer patients.
At present, CK19 expression is mainly diagnosed clinically through pathological examination of preoperative invasive puncture biopsy or operative excision of focus, MVI is also mainly diagnosed by depending on postoperative pathological examination, and the results of the two examinations serve as diagnostic gold standards of CK19 expression and MVI. The Imaging assessment method before the operation of liver cancer patients, which is commonly used in clinical practice, mainly includes ultrasound, electron computed tomography and Magnetic Resonance Imaging (MRI). Wherein, contrast agent enhanced MRI of gadolinate disodium (Gadolinium ethybenzyl diethylstillnamide Pentaacetic Acid, Gd-EOB-DTPA) is an important examination method for HCC preoperative diagnosis. The kit can accurately reflect the pathological condition of liver cancer tissue, provides rich information for evaluation and diagnosis of HCC, and can reflect the biological behavior and gene expression of HCC due to the imaging characteristics, thereby being beneficial to predicting CK19 expression and preoperative MVI. However, at present, doctors generally analyze imaging characteristics of MRI manually to predict the expression and MVI of the liver cancer CK19, so that the prediction results of the expression and MVI of the liver cancer CK19 are often influenced by clinical experience of doctors, and the prediction results may be different among different doctors, thereby causing inaccuracy of analysis and prediction results.
Thus, the prior art has yet to be improved and enhanced.
Disclosure of Invention
The present application is directed to provide a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, which overcome the disadvantages of the prior art.
In order to solve the technical problem, a first aspect of the embodiments of the present application provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, where the method is applied to a trained prediction network model, and the prediction network model includes a first feature extraction module for extracting CK19 expression features, a second feature extraction module for extracting shared features of CK19 expression and MVI, a third feature extraction module for extracting MVI features, and a prediction module; the method comprises the following steps:
controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted;
and controlling the prediction module to determine a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map.
The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the prediction module comprises a first prediction branch and a second prediction branch, and the first prediction branch and the second prediction branch respectively comprise a convolution unit, a spatial transformation unit and a prediction unit which are sequentially cascaded; the controlling the prediction module to determine, based on the first feature map, the second feature map and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted specifically includes:
controlling the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the first feature map and the second feature map;
and controlling the second prediction branch to determine the MVI category corresponding to the MR image to be predicted based on the second characteristic diagram and the third characteristic diagram.
The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the prediction module comprises a first prediction branch and a second prediction branch, the first prediction branch and the second prediction branch respectively comprise a feature transformation unit, a relation inference unit and a prediction unit in sequence, and the feature transformation unit comprises a convolution unit and a space transformation unit which are cascaded in sequence; the controlling the prediction module to determine, based on the first feature map, the second feature map and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted specifically includes:
controlling a feature transformation unit in the first prediction branch to determine a first affine feature map based on the first feature map and the second feature map;
controlling a feature transformation unit in the second prediction branch to determine a second affine feature map based on the second feature map and the third feature map;
controlling a relation inference unit in the first prediction branch to determine a first relation feature map based on the first affine feature map and the second affine feature map, and controlling a prediction unit in the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the first relation feature map;
and controlling a relation inference unit in the second prediction branch to determine a second relation feature map based on the first affine feature map and the second affine feature map, and controlling a prediction unit in the second prediction branch to determine the corresponding MVI type of the MR image to be predicted based on the second relation feature map.
The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the spatial transformation unit comprises a positioning network block, a network generator and a sampler, the positioning network block is connected with the network generator, the network generator is connected with the sampler, the positioning network block is used for generating an affine transformation coefficient matrix, the network generator is used for generating a sampling grid based on the affine transformation coefficient matrix, and the sampler is used for carrying out position mapping on a feature map input into the spatial transformation unit based on the sampling network so as to obtain an affine feature map.
The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the prediction module comprises a first prediction branch and a second prediction branch, and the first prediction branch and the second prediction branch respectively comprise a convolution unit, a relation inference unit and a prediction unit in sequence; the controlling the prediction module to determine, based on the first feature map, the second feature map and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted specifically includes:
controlling a convolution unit in the first prediction branch to determine a third affine feature map based on the first feature map and the second feature map;
controlling a convolution unit in the second prediction branch to determine a fourth affine feature map based on the second feature map and the third feature map;
controlling a relation inference unit in the first prediction branch to determine a third relation feature map based on the third affine feature map and the fourth affine feature map, and controlling a prediction unit in the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the third relation feature map;
and controlling a relation inference unit in the second prediction branch to determine a fourth relation feature map based on the third affine feature map and the fourth affine feature map, and controlling a prediction unit in the second prediction branch to determine the corresponding MVI type of the MR image to be predicted based on the fourth relation feature map.
The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the relational inference unit comprises a volume block, a global pooling layer and a multilayer perceptron; the convolution block is used for acquiring a feature relation graph of the two affine feature graphs input into the associated reasoning unit; the global pooling layer is used for converting the characteristic relation graph into a characteristic relation vector; the multilayer perceptron is configured to generate a relational feature map based on the feature relationship vector.
The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, wherein the CK19 expression category comprises CK19 expression negativity or CK19 expression positivity; the MVI category includes MVI or no MVI.
The embodiment of the application provides a hepatocellular carcinoma CK19 and MVI prediction device based on multitask learning, which is characterized in that the prediction device is configured with a trained prediction network model, the prediction network model comprises a first feature extraction module for extracting CK19 expression features, a second feature extraction module for extracting shared features of CK19 expression and MVI, a third feature extraction module for extracting MVI features and a prediction module, and the prediction device comprises:
the first control module is used for controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted;
and the second control module is used for controlling the prediction module to determine a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map.
A third aspect of embodiments of the present application provides a computer-readable storage medium storing one or more programs, which are executable by one or more processors to implement the steps of the method for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning as described in any one of the above.
A fourth aspect of the embodiments of the present application provides a terminal device, including: a processor, a memory, and a communication bus; the memory has stored thereon a computer readable program executable by the processor;
the communication bus realizes connection communication between the processor and the memory;
the processor, when executing the computer readable program, implements the steps in the method of hepatocellular carcinoma CK19 and MVI prediction based on multitask learning as described in any one of the above.
Has the advantages that: compared with the prior art, the application provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, and the method comprises the following steps: controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted; and controlling the prediction module to determine a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map. According to the method and the device, the prediction network model based on deep learning is adopted to directly extract rich image features carried by the MR image to be predicted, so that the problem that the prediction performance of the model is influenced due to subjectivity in image feature analysis can be avoided. Meanwhile, the CK19 expression features are extracted through the first feature extraction module, the shared features of CK19 expression and MVI are extracted through the second feature extraction module, the MVI features are extracted through the third feature extraction module, task parameters of a shared task and task parameters of an independent task are separated, and the problem that prediction results of a prediction network model are unstable due to complex parameter sharing can be solved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without any inventive work.
Fig. 1 is a flowchart of a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning provided in the present application.
Fig. 2 is a schematic diagram illustrating a training process and a testing process of a prediction network model in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 3 is a schematic structural diagram of a first feature extraction module in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 4 is a schematic structural diagram of a convolution block a in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 5 is a schematic structural diagram of a first residual unit in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 6 is a schematic structural diagram of one implementation of a prediction network model in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 7 is a schematic structural diagram of a convolution unit in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 8 is a schematic structural diagram of a convolution block c in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 9 is a schematic structural diagram of a spatial transform unit in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 10 is a schematic structural diagram of one implementation of a prediction network model in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 11 is a schematic structural diagram of a relationship inference unit in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 12 is a schematic structural diagram of one implementation of a prediction network model in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.
Fig. 13 is a schematic structural diagram of a hepatocellular carcinoma CK19 and MVI prediction device based on multitask learning according to the present application.
Fig. 14 is a schematic structural diagram of a terminal device provided in the present application.
Detailed Description
The present application provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, and in order to make the objects, technical solutions and effects of the present application clearer and clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
It should be understood that, the sequence numbers and sizes of the steps in this embodiment do not mean the execution sequence, and the execution sequence of each process is determined by its function and inherent logic, and should not constitute any limitation on the implementation process of this embodiment.
The inventor finds that the primary liver cancer is the sixth most common cancer worldwide, and the incidence rate of the primary liver cancer is on the rise worldwide. Hepatocellular carcinoma is the most common one of primary liver cancers, accounting for 90% of all primary liver malignancies. Hepatocellular Carcinoma (HCC) has high mortality and strong heterogeneity, and has significant global differences in morbidity and mortality. Surgical resection is currently the primary treatment of choice for HCC, however, most HCC patients treated by surgical resection have a recurrence of neogenetic tumors within 2 years. Among the many factors associated with the early recurrence of HCC post-surgery, positive expression of Cytokeratin19 (cytokine 19, CK19) and Microvascular Invasion (MVI) are important predictors of early recurrence and poor prognosis of HCC, and these two factors have a correlation, with HCC positive expression of CK19 being more prone to MVI.
HCC positively expressed by CK19 is more likely to have postoperative recurrent metastasis than HCC negatively expressed by CK19, and CK19 positive expression indicates that HCC is strongly invasive and is one of independent factors of HCC poor prognosis. When HCC patients are found to be at a high risk of poor prognosis, physicians need to reduce the postoperative recurrence rate by expanding the resection range of the tumor. Therefore, the preoperative prediction of CK19 expression of liver cancer is important for the formulation of treatment strategies and the judgment of prognosis of patients. According to literature reports, the risk of postoperative recurrence of HCC patients with MVI is high, and the MVI is an important prediction factor of HCC poor prognosis. Therefore, the preoperative diagnosis of the liver cancer MVI has important guiding function in the aspects of selection of treatment modes, prognosis and the like of liver cancer patients.
At present, CK19 expression is mainly diagnosed clinically through pathological examination of preoperative invasive puncture biopsy or operative excision of focus, MVI is also mainly diagnosed by depending on postoperative pathological examination, and the results of the two examinations serve as diagnostic gold standards of CK19 expression and MVI. The Imaging assessment method before the operation of liver cancer patients, which is commonly used in clinical practice, mainly includes ultrasound, electron computed tomography and Magnetic Resonance Imaging (MRI). Wherein, contrast agent enhanced MRI of gadolinate disodium (Gadolinium ethybenzyl diethylstillnamide Pentaacetic Acid, Gd-EOB-DTPA) is an important examination method for HCC preoperative diagnosis. The kit can accurately reflect the pathological condition of liver cancer tissue, provides rich information for evaluation and diagnosis of HCC, and can reflect the biological behavior and gene expression of HCC due to the imaging characteristics, thereby being beneficial to predicting CK19 expression and preoperative MVI. However, at present, doctors generally analyze imaging characteristics of MRI manually to predict the expression and MVI of the liver cancer CK19, so that the prediction results of the expression and MVI of the liver cancer CK19 are often influenced by clinical experience of doctors, and the prediction results may be different among different doctors, thereby causing inaccuracy of analysis and prediction results.
In order to solve the problems, a statistical analysis method and a traditional machine learning method are generally adopted to predict the CK19 expression and MVI of the liver cancer at present, wherein the statistical analysis method is to use the characteristic information of a medical image to perform statistical analysis to predict, and the statistical analysis method has measurement errors in part of characteristic critical value indexes when the MR image is analyzed, so that the result of the statistical analysis may be inaccurate; meanwhile, the analysis of most image characteristics has subjectivity, and the accuracy of the analysis prediction result cannot be ensured. In addition, the statistical learning method is easy to have an over-fitting problem, so that the model prediction performance is general, and under the condition that the feature data volume is large, the statistical learning method may not find the correlation among the features, so that the model prediction performance is poor. The machine learning method weakens the convergence problem, and the model prediction capability is strong under the condition of large characteristic data quantity. The accuracy of a prediction model established based on a traditional machine learning method is high, however, the traditional machine learning needs artificial design of features, the artificial design of the features often has the problems of strong data dependence and poor feature generalization, and overfitting is easy to occur on a small sample.
Based on this, in the embodiment of the application, the first feature extraction module, the second feature extraction module and the third feature extraction module are controlled to determine a first feature map, a second feature map and a third feature map based on an MR image to be predicted; and controlling the prediction module to determine a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map. According to the method and the device, the prediction network model based on deep learning is adopted to directly extract rich image features carried by the MR image to be predicted, so that the problem that the prediction performance of the model is influenced due to subjectivity in image feature analysis can be avoided. Meanwhile, the CK19 expression features are extracted through the first feature extraction module, the shared features of CK19 expression and MVI are extracted through the second feature extraction module, the MVI features are extracted through the third feature extraction module, task parameters of a shared task and task parameters of an independent task are separated, and the problem that prediction results of a prediction network model are unstable due to complex parameter sharing can be solved.
The following further describes the content of the application by describing the embodiments with reference to the attached drawings.
The embodiment provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the prediction method is applied to a trained prediction network model, the prediction network model is a neural network model based on deep learning, and feature extraction can be directly performed on an MR (Magnetic Resonance) image to be predicted so as to extract rich image features carried by the MR image. The prediction network model comprises a first feature extraction module for extracting CK19 (cytokine 19, Cytokeratin 19) expression features, a second feature extraction module for extracting shared features of CK19 expression and MVI (Microvascular Invasion), a third feature extraction module for extracting MVI features and a prediction module; the first feature extraction module, the second feature extraction module and the third feature extraction module are parallel and are connected with the prediction module, input items of the first feature extraction module, the second feature extraction module and the third feature extraction module are all MR images to be predicted, and input items of the prediction module are CK19 expression categories and MVI categories corresponding to the MR images to be predicted.
The embodiment provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, as shown in fig. 1 and 2, the method includes:
and S10, controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted.
Specifically, the first feature extraction module, the second feature extraction module and the third feature extraction module are parallel, the first feature extraction module is used for extracting a CK19 expression feature in the MR image to be predicted, the second feature extraction module is used for extracting a CK19 expression and an MVI shared feature in the MR image to be predicted, so that a shared learning effect is achieved, and the third feature extraction module is used for extracting an MVI feature in the MR image to be predicted. The input items of the first feature extraction module, the second feature extraction module and the third feature extraction module are MR images to be predicted, the output item of the first feature extraction module is a first feature map carrying image features for reflecting CK19 expression, the output item of the second feature extraction module is a second feature map carrying shared image features for reflecting CK19 expression and MVI, and the output item of the third feature extraction module is a third feature map carrying image features for reflecting MVI. In the embodiment, the CK19 expression characteristics, the CK19 expression and MVI shared characteristics and the MVI characteristics are independently obtained through the first feature extraction module, the second feature extraction module and the third feature extraction module, and each feature extraction module is configured with independent parameters, so that the CK19 expression feature extraction task is only influenced by the CK19 expression feature extraction task, the MVI feature extraction task is only influenced by the MVI feature extraction task, and the CK19 expression and MVI shared feature extraction task is influenced by the CK19 expression feature extraction task and the MVI feature extraction task, therefore, the feature extraction modules formed by the first feature extraction module, the second feature extraction module and the third feature extraction module can learn not only the feature knowledge of the CK19 expression feature extraction task and the MVI feature extraction task, but also the shared knowledge between the CK19 expression feature extraction task and the MVI feature extraction task, the method is beneficial to avoiding the situation that the network is biased to fit a CK19 expression feature extraction task or an MVI feature extraction task due to complex parameter sharing, so that negative migration and/or performance of one task is improved and performance of the other task is reduced in a prediction network model, wherein when the performance improvement task is the CK19 expression feature extraction task, the task performance reduction task is the MVI feature extraction task, and conversely, when the performance improvement task is the MVI feature extraction task, the task performance reduction task is the CK19 expression feature extraction task.
Therefore, in the embodiment, the CK19 expression feature extraction task, the MVI feature extraction task, the CK19 expression and MVI shared feature extraction task are independent from each other, and the task parameter parameters of each task are independent, so that the problems that the negative migration phenomenon exists in the existing multi-task learning and the performance of each task cannot be improved at the same time can be solved. The method is that the existing multi-task learning shares one feature extraction module, and when related association or conflict exists among the multi-tasks and the feature sharing is directly carried out through the shared feature extraction module, the shared task parameters in the shared feature extraction module are complex and low in effectiveness, so that the model performance of the prediction network model is reduced. Meanwhile, in the learning process, the shared feature extraction module is led or influenced by a certain task, so that the prediction network model is more prone to fitting the certain task, the effect of other tasks is possibly negatively influenced, and the performance effect of other tasks is poor. In the embodiment, the CK19 expression feature extraction task, the MVI feature extraction task, and the CK19 expression and MVI shared feature extraction task are respectively provided with the feature extraction modules, and the model parameters of each feature extraction module are independently configured, so that the influence caused by the sharing of the task parameters of each task can be avoided, and meanwhile, each task is not influenced by other tasks, so that the model performance of the prediction network model can be improved.
In addition, in the embodiment, in addition to independently setting the feature extraction module for the CK19 expression feature extraction task and the MVI feature extraction task, a shared feature extraction module (i.e., a second feature extraction module) for extracting the CK19 expression and MVI shared features is also provided, and the shared feature extraction module realizes flexible balance between the CK19 expression feature extraction task and the MVI feature extraction task. In the feature extraction modules of the CK19 expression feature extraction task and the MVI feature extraction task, the feature extraction module corresponding to each task and the shared feature extraction module responsible for shared learning provide feature knowledge and shared feature knowledge of each task, so that the predictive network model can improve the prediction performance and generalization performance of the CK19 expression feature extraction task and the MVI feature extraction task by utilizing the correlation of the CK19 expression feature extraction task and the MVI feature extraction task.
In an implementation manner of this embodiment, the network structures of the first feature extraction module, the second feature extraction module, and the third feature extraction module are the same. The first feature extraction module is described as an example. As shown in fig. 3, the first feature extraction module includes a convolution block a, a maximum pooling layer, a first residual unit, and a second residual unit, which are sequentially cascaded, where as shown in fig. 4, the convolution block a includes a convolution layer, a Group Normalization layer GN), and an active linear unit layer. In one implementation, the convolution kernel of the convolution layer in convolution block a may be a 3 x 3 convolution kernel.
The network structure of the first residual error unit and the second residual error unit is the same. As shown in fig. 5, the first residual unit includes a convolution block b, a convolution block a, a residual block, a convolution block b, a global average pooling layer, and a convolution layer, which are cascaded in sequence, wherein the convolution block b includes the same network layer as the convolution block a, and the difference is that the convolution kernel of the convolution layer in the convolution block b is 1 × 1 convolution kernel. The residual block comprises a first sub-input layer, a second sub-input layer, a first adder, an adaptive pooling layer, a convolution block b, a first convolution layer, a second convolution layer, a softmax layer, a first multiplier, a second multiplier and a second adder, wherein the first sub-input layer is respectively connected with the first adder and the first multiplier, the second sub-input layer is respectively connected with the first adder and the second multiplier, the first adder is sequentially connected with the adaptive pooling layer and the convolution block b, the convolution block b is respectively connected with the first convolution layer and the second convolution layer, the first convolution layer is connected with the first multiplier, the second convolution layer is connected with the second multiplier, the first multiplier and the second multiplier are both connected with the second adder, wherein the input items of the first sub-input layer and the second sub-input layer are both output items of the convolution block a, and the first sub-input layer and the second sub-input layer have input and output functions, i.e. the output items of the first sub-input layer and the second sub-input layer are both output items of the volume block a.
S20, controlling the prediction module to determine a CK19 expression class and an MVI class corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map.
Specifically, the prediction module is used for predicting a CK19 expression category and an MVI category corresponding to the MR image to be predicted, wherein the CK19 expression category comprises CK19 expression negativity or CK19 expression positivity, and the MVI category comprises MVI or does not comprise MVI; the MR image to be predicted is a magnetic resonance image formed by Gd-EOB-DTPA enhanced MRI examination of a liver cancer patient. The prediction module is used for absorbing image features extracted by the first feature extraction module based on a CK19 expression task, extracting image features extracted by the third feature extraction module based on an MVI task, extracting shared image features extracted by the second feature extraction module based on a CK19 expression task and the MVI task, and determining a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on all the extracted image features.
In an implementation manner of this embodiment, as shown in fig. 6, the prediction module includes a first prediction branch and a second prediction branch, and the first prediction branch and the second prediction branch both include a convolution unit, a spatial transform unit, and a prediction unit that are sequentially cascaded. Correspondingly, the controlling the prediction module to determine the CK19 expression category and the MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map specifically includes:
controlling the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the first feature map and the second feature map;
and controlling the second prediction branch to determine the MVI category corresponding to the MR image to be predicted based on the second characteristic diagram and the third characteristic diagram.
Specifically, the first prediction branch is used for predicting a CK19 expression class corresponding to an MR image to be predicted, and the second prediction branch is used for predicting an MVI class corresponding to the MR image to be predicted, where an input item of a convolution unit in the first prediction branch is a fusion graph of a first feature map and a second feature map, and an input item of a convolution unit in the second prediction branch is a fusion graph of a second feature map and a third feature map, where the fusion graph is formed based on channel splicing. That is, the fused graph of the first feature map and the second feature map is obtained by splicing the first feature map and the second feature map in the channel direction, and the fused graph of the second feature map and the third feature map is obtained by splicing the third feature map and the second feature map in the channel direction.
As shown in fig. 7, the convolution unit includes a convolution block c, a third residual unit, and a fourth residual unit, which are cascaded in sequence; as shown in fig. 8, the convolution block c includes a convolution layer, a Batch Normalization layer (BN), and a Leaky Linear Unit (Leaky ReLU) layer, where the convolution layer is connected to the Batch Normalization layer, the Batch Normalization layer is connected to the Leaky Linear Unit layer, and a convolution kernel of the convolution layer is 3 × 3 convolution kernels. The network configuration of the third residual unit and the fourth residual unit is the same as that of the first residual unit, and therefore, not specifically described here, the network configuration of the first residual unit may be referred to. The prediction unit may include a max pooling layer and a full connectivity layer through which CK19 expression categories or MVI categories are output.
The spatial transformation unit is used for carrying out affine transformation on the feature map output by the convolution unit so as to remove noise in the feature map, so that the model performance of the prediction network model can be improved, meanwhile, the spatial transformation unit can enhance the generalization capability of the prediction network module and improve the learning capability of the prediction network model. In one implementation, the affine transformation performed on the feature map by the spatial transformation unit may be represented as:
Figure BDA0003552370330000141
wherein the content of the first and second substances,
Figure BDA0003552370330000142
the pixels of the image are represented by pixels of the image,
Figure BDA0003552370330000143
representing pixels of the image after affine transformation, A θ Representing an affine transform coefficient matrix. The affine transform coefficient matrix may include one or more of a translation coefficient, a scaling coefficient, a flipping coefficient, a rotation coefficient, and a shearing coefficient. In the space transformation unit, an affine transformation coefficient matrix is learned through the training process of the prediction network model, the image is transformed into a next expected form, meanwhile, the interested area of the prediction network model for the characteristic diagram can be automatically selected in the training process, and therefore the prediction accuracy of the prediction network model is improved.
As shown in fig. 9, the spatial transform unit may include a positioning network block connected to the network generator, a network generator connected to the sampler, the positioning network block configured to generate an affine transformation coefficient matrix, the network generator configured to generate a sampling grid based on the affine transformation coefficient matrix, and a sampler configured to perform position mapping on the feature map input to the spatial transform unit based on the sampling network to obtain an affine feature map.
In one implementation, as shown in fig. 9, the positioning network block includes a first convolutional layer, a second convolutional layer, a first leaky linear activation unit layer, a third convolutional layer, a second leaky linear activation unit layer, a first fully-connected layer, and a second fully-connected layer, which are sequentially cascaded, where a convolution kernel of the first convolutional layer is 1 × 1 convolution kernel, a convolution kernel of the second convolutional layer is 3 × 3 convolution kernel, and a convolution kernel of the third convolutional layer is 5 × 5 convolution kernel. The network generator generates a grid according to the image size of the affine characteristic image determined by the space transformation unit, and then determines the adopted network T corresponding to the output item of the space transformation unit based on the affine transformation coefficient matrix and the generated network θ (G) In that respect The sampler may include an adder that performs position mapping on the feature map of the input spatial transform unit using the layer pair to obtain an affine transformThe sampler can also use an interpolation method to complete interpolation of the non-value areas in the affine feature map.
In the embodiment, a spatial transformation unit is added into a prediction network model, the spatial transformation unit absorbs image features extracted by a first feature extraction module based on CK19 expression tasks, a third feature extraction module based on MVI tasks, and shared image features extracted by a second feature extraction module based on CK19 expression tasks and MVI tasks, and affine transforms effective image features in a fused graph of a first feature graph and a second feature graph input into the spatial transformation unit or in a fused graph of the second feature graph and the third feature graph through the spatial transformation unit, so that the prediction unit can perform CK19 expression prediction and MVI prediction based on affine feature graphs obtained by affine transformation, thus noise information of the feature graphs and CK19 expression prediction tasks or MVI prediction tasks can be reduced, and the input features can be transformed into a form expected by a next layer of the network, the generalization of a network model is promoted, and meanwhile the prediction accuracy of a CK19 expression prediction task and an MVI prediction task is promoted. Of course, in practical application, the first prediction branch and the second prediction branch may not include a spatial variation unit, that is, the first prediction branch and the second prediction branch may only include a convolution unit and a prediction unit that are cascaded in sequence.
In an implementation manner of this embodiment, as shown in fig. 10, the prediction module includes a first prediction branch and a second prediction branch, where the first prediction branch and the second prediction branch both include a feature transformation unit, a relationship inference unit, and a prediction unit in sequence, and the feature transformation unit includes a convolution unit and a spatial transformation unit that are cascaded in sequence; the controlling the prediction module to determine, based on the first feature map, the second feature map and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted specifically includes:
controlling a feature transformation unit in the first prediction branch to determine a first affine feature map based on the first feature map and the second feature map;
controlling a feature transformation unit in the second prediction branch to determine a second affine feature map based on the second feature map and the third feature map;
controlling a relation inference unit in the first prediction branch to determine a first relation feature map based on the first affine feature map and the second affine feature map, and controlling a prediction unit in the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the first relation feature map;
and controlling a relation inference unit in the second prediction branch to determine a second relation feature map based on the first affine feature map and the second affine feature map, and controlling a prediction unit in the second prediction branch to determine the corresponding MVI type of the MR image to be predicted based on the second relation feature map.
Specifically, the convolution unit and the spatial transform unit in the prediction unit and the feature transform unit are all the same as those in the foregoing implementation, and specific reference may be made to the description of the foregoing implementation, which is not repeated here. The description of the relational inference unit is focused here. The relation reasoning unit in the first prediction branch is respectively connected with the relation reasoning unit in the first prediction branch and the feature transformation unit in the second prediction branch, and the relation reasoning unit in the second prediction branch is respectively connected with the feature transformation unit in the first prediction branch and the feature transformation unit in the second prediction branch. That is to say, the input items of the relationship inference unit in the first prediction branch are the first affine feature map determined by the relationship inference unit in the first prediction branch and the second affine feature map determined by the relationship inference unit in the second prediction branch, and the input items of the relationship inference unit in the second prediction branch are the first affine feature map determined by the relationship inference unit in the first prediction branch and the second affine feature map determined by the relationship inference unit in the second prediction branch.
The relation reasoning unit is used for determining the correlation between the first affine feature map and the second affine feature map, namely the relation reasoning unit is used for extracting the correlation between the CK19 expression prediction task and the MVI prediction task, and the prediction unit is made to know the correlation, so that the prediction accuracy of the prediction network model is improved. In one implementation, the relational inference unit may be represented as a relational function, where the expression of the relational function may be:
Figure BDA0003552370330000161
wherein, LR (A) l ，A m ) Representing a relational feature graph, A l Representing a first affine feature map, A m A second affine feature map is represented which,
Figure BDA0003552370330000162
and
Figure BDA0003552370330000163
is a function for reflecting the pairwise relationship between tasks.
In a specific implementation manner, the relational inference unit comprises a volume block, a global pooling layer and a multi-layer perceptron; the convolution block is used for acquiring a feature relation graph of two affine feature graphs input into the associated reasoning unit; the global pooling layer is used for converting the characteristic relation graph into a characteristic relation vector; the multilayer perceptron is configured to generate a relational feature map based on the feature relationship vector. That is to say that the position of the first electrode,
Figure BDA0003552370330000171
by means of a rolling block, a global average pooling layer and a multi-layer perceptron,
Figure BDA0003552370330000172
is realized by a multilayer perceptron. As shown in fig. 11, the convolution block includes a convolution layer, a group normalization layer, an activated linear unit layer with leakage, four convolution blocks c, an adaptive pooling layer, a full-link layer, an activated linear unit layer, a full-link layer, and an activated linear unit layer, which are sequentially cascaded. The network structure of the convolution block c is the same as that of the convolution block c in the convolution unit, hereAnd is not specifically described. In addition, the convolution kernel of the convolution layer is a 1 × 1 convolution kernel.
In this embodiment, both the global average pooling layer and the multilayer perceptron in the relational inference unit have model parameters that can be trained, and the model parameters are obtained by training in the training process of the prediction network model, that is, end-to-end learning enables the relational inference unit to learn the relationship between the CK19 expression prediction task and the MVI prediction task. Meanwhile, the relational reasoning unit enables the relational reasoning unit to automatically learn the mutual relation between the CK19 expression prediction task and the MVI prediction task in a data-driven mode without inputting any prior knowledge about task relations into the prediction network model.
In the embodiment, by arranging the relationship inference unit in the prediction network model, on a branch of a CK19 expressing a prediction task, a feature map corresponding to a CK19 expressing the prediction task and a feature map corresponding to an MVI prediction task are connected in series, the feature maps after being connected in series are input into the relationship inference unit, the feature maps obtained after being connected in series are analyzed by the relationship inference unit, then the prediction unit performs CK19 expressing prediction based on the relationship feature map obtained by the relationship inference unit, and the prediction is performed on the branch of the MVI prediction task in the same way, so that the relationship inference unit can influence the feature learning of a network layer at the upstream of a module, thereby generating an implicit feature capable of performing relationship inference, and being beneficial to improving the performance of the prediction network model in the CK19 expressing the prediction task and the MVI prediction task.
In an implementation manner of this embodiment, as shown in fig. 12, in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the prediction module includes a first prediction branch and a second prediction branch, where the first prediction branch and the second prediction branch each include a convolution unit, a relationship inference unit, and a prediction unit in sequence; the controlling the prediction module to determine, based on the first feature map, the second feature map and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted specifically includes:
controlling a convolution unit in the first prediction branch to determine a third affine feature map based on the first feature map and the second feature map;
controlling a convolution unit in the second prediction branch to determine a fourth affine feature map based on the second feature map and the third feature map;
controlling a relation inference unit in the first prediction branch to determine a third relation feature map based on the third affine feature map and the fourth affine feature map, and controlling a prediction unit in the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the third relation feature map;
and controlling a relation inference unit in the second prediction branch to determine a fourth relation feature map based on the third affine feature map and the fourth affine feature map, and controlling a prediction unit in the second prediction branch to determine the corresponding MVI type of the MR image to be predicted based on the fourth relation feature map.
Specifically, the prediction network model does not include a spatial variation unit, the relationship inference unit is directly connected to the convolution unit, and an output item of the convolution unit is used as an input item of the convolution unit and performs relationship inference to obtain a relationship characteristic diagram, wherein network structures and functions of the convolution unit, the relationship inference unit and the prediction unit are the same as those of the convolution unit, the relationship inference unit and the prediction network, and are not repeated here.
In an implementation manner of this embodiment, before predicting a CK19 expression class and an MVI class corresponding to an MR image to be predicted based on the prediction network model, the prediction network model is trained in advance, wherein in a training process of the prediction network model, a training sample set adopted by the prediction network model includes a plurality of training MR images, and each of the plurality of training MR images is an MR image obtained by Gd-EOB-DTPA enhanced MRI examination of a liver cancer patient. In a specific implementation manner, each training MR image is obtained by performing Gd-EOB-DTPA enhanced MRI examination within one month before a liver cancer operation, a liver cancer tumor carried by a liver cancer patient corresponding to each training MR image is a single-shot lesion, each training MR image carries a CK19 expression category label and an MVI category label, and before a liver cancer surgical resection treatment, other treatments for HCC are not performed, wherein the CK19 expression category label can be determined by immunohistochemical analysis after the liver cancer operation, and the MVI category label is determined by pathological confirmation after the liver cancer operation.
After the training sample set is obtained, the training sample set may be preprocessed, where the preprocessing may include one or more of resampling, image cropping, image size unification, and data amplification. The resampling is specifically to count the resolution of the training MR images in the received training sample set, and determine the resolution (e.g., 1.19 × 1.19 × 2 mm) corresponding to the training MR image with the most resolution 3 ) As the target resolution, the resolutions of all the training MR images in the training sample set are resampled to the target resolution, so that the resampling operation on excessive data can be avoided, and meanwhile, the generalization of the prediction network model can be improved.
The image cutting is to reduce interference of background information of a training MR image on a network model, wherein the image cutting is to cut each layer of image in the training MR image based on a focus area, wherein the preset focus area can be formed by drawing three layers of gold standards on the cross section of the MR image of a patient based on ITK software by a radiologist, the three layers of gold standards are respectively the top layer, the maximum layer and the bottom layer of a tumor, and finally, a three-dimensional frame containing the whole focus is formed based on the three layers of gold standards.
The image size is uniformly used for adjusting the image size of the cropped training MR images, so that the image sizes of all the adjusted training MR images are the same, wherein the adjusted image size may be the maximum image size of all the training MR images. For example, the image size of each slice image in the adjusted training MR image is 224 × 224, which can be achieved by performing a zero padding operation around the image of each slice image of the training MR image.
The data amplification aims to improve the diversity of the training sample set and improve the generalization capability of the prediction network model. Wherein the data augmentation may include (1) image flipping: turning over the original image in the horizontal direction or the vertical direction; (2) image random cropping: image clipping is carried out on the original image, and the clipping amplitude is 0-10%; (3) image zooming: randomly zooming to 70% -110% of the original image; (4) image translation: translating 0-10% in the horizontal direction and the vertical direction; (5) image rotation: rotating the image by a rotation angle within the range of-20 degrees; (6) shearing transformation: and (4) cutting and transforming the original image with the amplitude of-16 degrees.
In the training process of the prediction network model, an internal cross validation result is obtained by using a 10-fold cross validation method: the Area (AUC) of a Receiver Operating Characteristic Curve (ROC) of a CK19 expression prediction task Under the ROC can reach 0.87, and the accuracy rate reaches 0.83; the AUC of the MVI prediction task of the liver cancer can reach 0.88, and the accuracy rate reaches 0.85. Then, in order to evaluate the generalization ability of the algorithm, an external independent test set is adopted to carry out external independent test on the prediction network model obtained by training, so as to obtain an external independent verification result: the AUC of the CK19 expression prediction task can reach 0.80, and the accuracy rate reaches 0.84; the AUC of the MVI prediction task of the liver cancer can reach 1.00, and the accuracy rate reaches 0.89.
In summary, the present embodiment provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, and the method includes: controlling a first feature extraction module, a second feature extraction module and a third feature extraction module to determine a first feature map, a second feature and a third feature map based on the MR image to be predicted; the control prediction module determines a CK19 expression type and an MVI type of the MR image based on the first feature map, the second feature map and the third feature map. According to the method and the device, the abundant image characteristics carried by the MR image to be predicted are directly extracted through the prediction network model, and the problem that the prediction performance of the model is influenced due to subjectivity in image characteristic analysis can be avoided. Meanwhile, the CK19 expression features are extracted through the first feature extraction module, the shared features of CK19 expression and MVI are extracted through the second feature extraction module, the MVI features are extracted through the third feature extraction module, task parameters of a shared task and task parameters of an independent task are separated, and the problem that prediction results are unstable due to complex parameter sharing can be solved.
Based on the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the embodiment provides a hepatocellular carcinoma CK19 and MVI prediction device based on multitask learning, the prediction device is configured with a trained prediction network model, the prediction network model comprises a first feature extraction module for extracting CK19 expression features, a second feature extraction module for extracting CK19 expression and MVI shared features, a third feature extraction module for extracting MVI features, and a prediction module, as shown in fig. 13, the prediction device comprises:
a first control module 100, configured to control the first feature extraction module, the second feature extraction module, and the third feature extraction module to determine a first feature map, a second feature map, and a third feature map based on an MR image to be predicted;
a second control module 200, configured to control the prediction module to determine, based on the first feature map, the second feature map, and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted.
Based on the above-mentioned hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the present embodiment provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning as described in the above embodiments.
Based on the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the present application further provides a terminal device, as shown in fig. 14, including at least one processor (processor) 20; a display screen 21; and a memory (memory)22, and may further include a communication Interface (Communications Interface)23 and a bus 24. The processor 20, the display 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a user guidance interface preset in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may call logic instructions in the memory 22 to perform the methods in the embodiments described above.
Furthermore, the logic instructions in the memory 22 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product.
The memory 22, which is a computer-readable storage medium, may be configured to store a software program, a computer-executable program, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the functional application and data processing, i.e. implements the method in the above-described embodiments, by executing the software program, instructions or modules stored in the memory 22.
The memory 22 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. Further, the memory 22 may include a high speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.
In addition, the specific processes loaded and executed by the storage medium and the instruction processors in the terminal device are described in detail in the method, and are not stated herein.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims (10)

1. The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the method is applied to a trained prediction network model, and the prediction network model comprises a first feature extraction module, a second feature extraction module, a third feature extraction module and a prediction module, wherein the first feature extraction module is used for extracting CK19 expression features, the second feature extraction module is used for extracting shared features of CK19 expression and MVI, and the third feature extraction module is used for extracting MVI features; the method comprises the following steps:
controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted;
and controlling the prediction module to determine a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map.
2. The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning of claim 1, wherein the prediction module comprises a first prediction branch and a second prediction branch, and the first prediction branch and the second prediction branch respectively comprise a convolution unit, a spatial transformation unit and a prediction unit which are cascaded in sequence; the controlling the prediction module to determine, based on the first feature map, the second feature map and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted specifically includes:
controlling the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the first feature map and the second feature map;
and controlling the second prediction branch to determine the MVI category corresponding to the MR image to be predicted based on the second characteristic diagram and the third characteristic diagram.
3. The method for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning according to claim 1, wherein the prediction module comprises a first prediction branch and a second prediction branch, the first prediction branch and the second prediction branch each comprise a feature transformation unit, a relation inference unit and a prediction unit in sequence, and the feature transformation unit comprises a convolution unit and a spatial transformation unit in sequence; the controlling the prediction module to determine, based on the first feature map, the second feature map and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted specifically includes:
controlling a feature transformation unit in the first prediction branch to determine a first affine feature map based on the first feature map and the second feature map;
controlling a feature transformation unit in the second prediction branch to determine a second affine feature map based on the second feature map and the third feature map;
controlling a relation inference unit in the first prediction branch to determine a first relation feature map based on the first affine feature map and the second affine feature map, and controlling a prediction unit in the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the first relation feature map;
and controlling a relation inference unit in the second prediction branch to determine a second relation feature map based on the first affine feature map and the second affine feature map, and controlling a prediction unit in the second prediction branch to determine the corresponding MVI type of the MR image to be predicted based on the second relation feature map.
4. The method according to claim 2 or 3, wherein the spatial transform unit comprises a positioning network block, a network generator and a sampler, the positioning network block is connected with the network generator, the network generator is connected with the sampler, the positioning network block is used for generating an affine transform coefficient matrix, the network generator is used for generating a sampling grid based on the affine transform coefficient matrix, and the sampler is used for mapping the position of the feature map input to the spatial transform unit based on the sampling network to obtain an affine feature map.
5. The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning as claimed in claim 1, wherein the prediction module comprises a first prediction branch and a second prediction branch, and the first prediction branch and the second prediction branch each comprise a convolution unit, a relation inference unit and a prediction unit in sequence; the controlling the prediction module to determine, based on the first feature map, the second feature map and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted specifically includes:
controlling a convolution unit in the first prediction branch to determine a third affine feature map based on the first feature map and the second feature map;
controlling a convolution unit in the second prediction branch to determine a fourth affine feature map based on the second feature map and the third feature map;
controlling a relation inference unit in the first prediction branch to determine a third relation feature map based on the third affine feature map and the fourth affine feature map, and controlling a prediction unit in the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the third relation feature map;
and controlling a relation inference unit in the second prediction branch to determine a fourth relation feature map based on the third affine feature map and the fourth affine feature map, and controlling a prediction unit in the second prediction branch to determine the corresponding MVI type of the MR image to be predicted based on the fourth relation feature map.
6. The method for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning according to claim 3 or 5, wherein the relational inference unit comprises a volume block, a global pooling layer and a multi-layer perceptron; the convolution block is used for acquiring a feature relation graph of the two affine feature graphs input into the associated reasoning unit; the global pooling layer is used for converting the characteristic relation graph into a characteristic relation vector; the multilayer perceptron is configured to generate a relational feature map based on the feature relationship vector.
7. The method for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning according to claim 1, wherein the CK19 expression classes comprise CK19 expression negative or CK19 expression positive; the MVI category includes MVI or no MVI.
8. A hepatocellular carcinoma CK19 and MVI prediction device based on multitask learning is characterized in that the prediction device is configured with a trained prediction network model, the prediction network model comprises a first feature extraction module for extracting CK19 expression features, a second feature extraction module for extracting shared features of CK19 expression and MVI, a third feature extraction module for extracting MVI features and a prediction module, and the prediction device comprises:
the first control module is used for controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted;
and the second control module is used for controlling the prediction module to determine a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map.
9. A computer readable storage medium, storing one or more programs, which are executable by one or more processors, for performing the steps of the method for hepatocellular carcinoma CK19 and MVI prediction based on multitask learning according to any one of claims 1-7.
10. A terminal device, comprising: a processor, a memory, and a communication bus; the memory has stored thereon a computer readable program executable by the processor;
the communication bus realizes connection communication between the processor and the memory;
the processor, when executing the computer readable program, implements the steps in the method of multitask learning based hepatocellular carcinoma CK19 and MVI prediction according to any one of claims 1-7.
CN202210265257.4A 2022-03-17 2022-03-17 Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning Active CN114882996B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210265257.4A CN114882996B (en) 2022-03-17 2022-03-17 Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210265257.4A CN114882996B (en) 2022-03-17 2022-03-17 Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning

Publications (2)

Publication Number Publication Date
CN114882996A true CN114882996A (en) 2022-08-09
CN114882996B CN114882996B (en) 2023-04-07

Family

ID=82668289

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210265257.4A Active CN114882996B (en) 2022-03-17 2022-03-17 Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning

Country Status (1)

Country Link
CN (1) CN114882996B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117558414A (en) * 2023-11-23 2024-02-13 之江实验室 System, electronic device and medium for predicting early recurrence of multi-tasking hepatocellular carcinoma
CN117786536A (en) * 2024-02-23 2024-03-29 北京壹永科技有限公司 Training method, device, equipment and medium for large language model

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170103287A1 (en) * 2015-10-13 2017-04-13 Impac Medical Systems, Inc. Pseudo-ct generation from mr data using tissue parameter estimation
CN107180154A (en) * 2017-02-08 2017-09-19 冯德昭 The method and system of patients with orthotopic liver transplantation prognosis situation packet is carried out to the patients with hepatocellular carcinoma of single tumour
CN107428821A (en) * 2014-12-08 2017-12-01 博格有限责任公司 Purposes of the mark including tenuin A in diagnosing and treating prostate cancer
CN112233112A (en) * 2020-12-07 2021-01-15 深圳大学 Prediction method of CK19 of hepatocellular carcinoma based on deep learning analysis
CN113129316A (en) * 2021-04-15 2021-07-16 重庆邮电大学 Heart MRI image multi-task segmentation method based on multi-mode complementary information exploration
CN113345576A (en) * 2021-06-04 2021-09-03 江南大学 Rectal cancer lymph node metastasis diagnosis method based on deep learning multi-modal CT
CN113571194A (en) * 2021-07-09 2021-10-29 清华大学 Modeling method and device for hepatocellular carcinoma long-term prognosis prediction
CN114068001A (en) * 2020-08-07 2022-02-18 四川医枢科技股份有限公司 Auxiliary support system and method for clinical decision, teaching and scientific research of head and neck tumors
CN114141366A (en) * 2021-12-31 2022-03-04 杭州电子科技大学 Cerebral apoplexy rehabilitation assessment auxiliary analysis method based on voice multitask learning

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107428821A (en) * 2014-12-08 2017-12-01 博格有限责任公司 Purposes of the mark including tenuin A in diagnosing and treating prostate cancer
US20170103287A1 (en) * 2015-10-13 2017-04-13 Impac Medical Systems, Inc. Pseudo-ct generation from mr data using tissue parameter estimation
CN107180154A (en) * 2017-02-08 2017-09-19 冯德昭 The method and system of patients with orthotopic liver transplantation prognosis situation packet is carried out to the patients with hepatocellular carcinoma of single tumour
CN114068001A (en) * 2020-08-07 2022-02-18 四川医枢科技股份有限公司 Auxiliary support system and method for clinical decision, teaching and scientific research of head and neck tumors
CN112233112A (en) * 2020-12-07 2021-01-15 深圳大学 Prediction method of CK19 of hepatocellular carcinoma based on deep learning analysis
CN113129316A (en) * 2021-04-15 2021-07-16 重庆邮电大学 Heart MRI image multi-task segmentation method based on multi-mode complementary information exploration
CN113345576A (en) * 2021-06-04 2021-09-03 江南大学 Rectal cancer lymph node metastasis diagnosis method based on deep learning multi-modal CT
CN113571194A (en) * 2021-07-09 2021-10-29 清华大学 Modeling method and device for hepatocellular carcinoma long-term prognosis prediction
CN114141366A (en) * 2021-12-31 2022-03-04 杭州电子科技大学 Cerebral apoplexy rehabilitation assessment auxiliary analysis method based on voice multitask learning

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
MIRA VALKONEN: "Cytokeratin-Supervised Deep Learning for Automatic Recognition of Epithelial Cells in Breast Cancers Stained for ER, PR, and Ki-67", 《IEEE TRANSACTIONS ON MEDICAL IMAGING》 *
段亚阳等: "基于非增强MRI的影像组学术前预测肝细胞癌微血管浸润的研究", 《磁共振成像》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117558414A (en) * 2023-11-23 2024-02-13 之江实验室 System, electronic device and medium for predicting early recurrence of multi-tasking hepatocellular carcinoma
CN117558414B (en) * 2023-11-23 2024-05-24 之江实验室 System, electronic device and medium for predicting early recurrence of multi-tasking hepatocellular carcinoma
CN117786536A (en) * 2024-02-23 2024-03-29 北京壹永科技有限公司 Training method, device, equipment and medium for large language model
CN117786536B (en) * 2024-02-23 2024-05-10 北京壹永科技有限公司 Training method, device, equipment and medium for large language model

Also Published As

Publication number Publication date
CN114882996B (en) 2023-04-07

Similar Documents

Publication Publication Date Title
US11610308B2 (en) Localization and classification of abnormalities in medical images
CN111127466B (en) Medical image detection method, device, equipment and storage medium
CN114882996B (en) Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning
EP3806035A1 (en) Reducing false positive detections of malignant lesions using multi-parametric magnetic resonance imaging
US20230410301A1 (en) Machine learning techniques for tumor identification, classification, and grading
US11645753B2 (en) Deep learning-based multi-site, multi-primitive segmentation for nephropathology using renal biopsy whole slide images
JP2023540910A (en) Connected Machine Learning Model with Collaborative Training for Lesion Detection
US11587231B2 (en) Comprehensive detection device and method for cancerous region
Lo et al. Cycle-consistent GAN-based stain translation of renal pathology images with glomerulus detection application
CN117078692B (en) Medical ultrasonic image segmentation method and system based on self-adaptive feature fusion
CN114693671B (en) Lung nodule semi-automatic segmentation method, device, equipment and medium based on deep learning
US20210035694A1 (en) Population-specific prediction of prostate cancer recurrence based on stromal morphology features
Yamaguchi et al. Glomerular classification using convolutional neural networks based on defined annotation criteria and concordance evaluation among clinicians
Aslam et al. Liver-tumor detection using CNN ResUNet
CN113764101B (en) Novel auxiliary chemotherapy multi-mode ultrasonic diagnosis system for breast cancer based on CNN
CN111528918B (en) Tumor volume change trend graph generation device after ablation, equipment and storage medium
Khaledyan et al. Enhancing breast ultrasound segmentation through fine-tuning and optimization techniques: sharp attention UNet
CN116109856A (en) Multi-mode mammary gland energy spectrum image classification method
Madhukar et al. Multi-scale convolution based breast cancer image segmentation with attention mechanism in conjunction with war search optimization
CN110570417B (en) Pulmonary nodule classification device and image processing equipment
Wang et al. MSAMS-Net: accurate lung lesion segmentation from COVID-19 CT images
Sahlin Detection of breast cancer in pocket ultrasound images using deep learning
CN116051467B (en) Bladder cancer myolayer invasion prediction method based on multitask learning and related device
Rani et al. Accurate artificial intelligence method for abnormality detection of CT liver images
CN115330949B (en) Liver cell cancer invasion edge prediction method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant