CN115330733A

CN115330733A - Disease intelligent identification method and system based on fine-grained domain knowledge

Info

Publication number: CN115330733A
Application number: CN202210990977.7A
Authority: CN
Inventors: 陈超; 王铭宇; 徐埌; 黄凌云; 肖京
Original assignee: Ping An Technology Shenzhen Co Ltd
Current assignee: Ping An Technology Shenzhen Co Ltd
Priority date: 2022-08-18
Filing date: 2022-08-18
Publication date: 2022-11-11

Abstract

The invention provides a disease intelligent identification method and system based on fine-grained domain knowledge, belonging to the technical field of intelligent medical treatment, wherein a dual-coordinate disease identification model is used for carrying out feature extraction on a focus image of a patient to obtain focus region features and focus boundary features; the dual-coordinate disease identification model comprises a first coordinate model for dividing a focus area and a second coordinate model for recognizing focus boundary characteristics; acquiring focus aggregation characteristics according to focus region characteristics and focus boundary characteristics through a self-attention network; classifying and identifying the focus aggregation characteristics to obtain a disease identification result of a patient; making fine grained information of the disease interpretable. The system is convenient for assisting doctors to carry out qualitative and even quantitative analysis on the focus and other interested areas, thereby greatly improving the accuracy and reliability of medical diagnosis; the system can also play an important auxiliary role in medical teaching, operation planning, operation simulation and various medical researches.

Description

Disease intelligent identification method and system based on fine-grained domain knowledge

Technical Field

The invention belongs to the technical field of intelligent medical treatment, and particularly relates to a disease intelligent identification method and system based on fine-grained domain knowledge, electronic equipment and a storage medium.

Background

In the prior art, the ultrasonic imaging examination is the most common tool for diagnosing cancer because of its advantages of innocuity, non-invasiveness, low cost, fast imaging, etc., and among them, the B-mode is the most common ultrasonic imaging mode in cancer examination because of its sufficient sensitivity. Based on an ultrasonic image of a disease region, a Computer Aided Design-CAD (Computer Aided Design-CAD) technology of efficient scalable learning (DNN) of a deep neural network is excellent in intelligent diagnosis of diseases, migration learning is mostly used for overcoming the problem of limited data sets in medical image processing, and strong feature representation capability of a pre-trained backbone model is used.

For Thyroid disease as an example, the Thyroid imaging reporting and data system (TI-RADS) ranks the malignancy characteristics of five Thyroid nodules according to: 1. solid nodules; 2. hypo-or ultra-low echo; 3. lobulation or irregular edges; 4. calcification of gravel sample; 5. the aspect ratio is more than or equal to 1. The existing thyroid imaging report and data system guidance have strong subjectivity, and thyroid disease diagnosis has high requirements on experience of radiologists. On the basis, the method for automatically diagnosing the thyroid diseases by fully utilizing a data-driven method becomes a feasible scheme. In the intelligent diagnosis process of thyroid diseases, although the problem of limited data set in medical image processing is solved by using transfer learning and some assistance is provided for intelligent diagnosis of thyroid by using the strong feature representation capability of a pre-trained backbone model, the following disadvantages still exist: 1) The thyroid diagnosis task is used as a conventional binary classification in the method, and only two classification labels of benign and malignant are embodied. However, benign and malignant are related to a plurality of factors, and only modeling with a benign tag and a malignant tag is relatively lack of sufficient interpretability.

In order to improve the interpretability, a multi-task learning (MTL) method is applied to intelligent diagnosis of thyroid diseases, and Domain Knowledge (DK) formed by doctor experience is taken as supervision information and is incorporated into a modeling process; although partial TI-RADS information is utilized, there are still only two classification tags, benign and malignant, and the interpretability of fine-grained information of classification variables is not sufficient.

Therefore, a disease intelligent identification method based on fine-grained domain knowledge is needed.

Disclosure of Invention

The invention provides a disease intelligent identification method, a disease intelligent identification system, electronic equipment and a storage medium based on fine-grained domain knowledge, which are used for overcoming at least one technical problem in the prior art.

In order to achieve the aim, the invention provides an intelligent disease identification method based on fine-grained domain knowledge,

acquiring a focus image of a patient to be identified;

performing feature extraction on a focus image of a patient by using a double-coordinate disease identification model to obtain focus region features and focus boundary features; wherein the dual-coordinate disease recognition model comprises a first coordinate model for lesion region division and a second coordinate model for lesion boundary feature recognition;

acquiring focus aggregation characteristics according to focus region characteristics and focus boundary characteristics through a self-attention network;

classifying and identifying focus aggregation characteristics to obtain a disease identification result of a patient; wherein the disease recognition result comprises benign/malignant class determination of the lesion and a lesion location.

Further, preferably, the method for obtaining the focus aggregation characteristics according to the focus area characteristics and the focus boundary characteristics through the self-attention network comprises the following steps,

fusing the focus region characteristics and the focus boundary characteristics to obtain original fusion characteristics;

after the original fusion features are subjected to convolution layers and activation functions, significance weight features corresponding to input features are obtained;

and multiplying the significance weight characteristic corresponding to the input characteristic with the original characteristic to obtain an aggregation characteristic.

performing feature de-entanglement on the focus boundary features by using a split merging module to obtain a plurality of focus boundary feature blocks;

marking a plurality of focus boundary characteristic blocks and focus region characteristic blocks, and embedding a category marking characteristic block to obtain a marked focus characteristic block;

and acquiring aggregation characteristics by using the marked focus characteristic block by using a multi-head self-attention module.

Further, preferably, the method for training the two-coordinate disease recognition model comprises,

preprocessing a sample containing the marking information of the focus area to obtain a mask of the focus area and a mask of a normal area; obtaining a trained first coordinate model based on the lesion area mask and the normal area mask; preprocessing a sample containing the lesion dot matrix marking information to obtain vectorized representation of a lesion dot matrix; obtaining a trained second coordinate model based on vectorization representation of the focus lattice; wherein the first coordinate model is a Cartesian coordinate model; the second coordinate model is a polar coordinate model;

integrating the trained first coordinate model and the trained second coordinate model based on a multi-head self-attention module to obtain a double-coordinate disease recognition model;

and training the dual-coordinate disease recognition model by using a gradient back propagation algorithm based on the loss function until convergence.

Further, preferably, the method for training the two-coordinate disease recognition model by using the gradient back propagation algorithm based on the loss function until convergence comprises,

obtaining the grading distribution of disease benign and malignant evaluation and the grading distribution of lesion position evaluation of an input sample containing lesion region marking information through a double-coordinate disease identification model;

obtaining a loss function between the score distribution of the disease benign and malignant assessment and the score distribution of the original disease benign and malignant assessment corresponding to the sample image data set, and a loss function between the score distribution of the lesion position assessment and the score distribution of the original disease lesion position assessment corresponding to the sample image data set;

and updating the network parameters of the dual-coordinate disease identification model according to the loss function until the mean square deviation of the grading distribution of the disease benign and malignant evaluation and the mean square deviation of the grading distribution of the lesion position evaluation both belong to a preset standard range.

Further, preferably, the lesion dot matrix marking information includes a benign characteristic block mark, a sharp edge characteristic block mark, a burr characteristic block mark, an edge angulation characteristic block mark, an edge smoothness characteristic block mark and an ultrasound image characteristic block mark.

Further, preferably, the method for training the two-coordinate disease recognition model further comprises,

establishing a disease scoring system by utilizing binary logistic regression analysis;

screening for independent variables having a correlation with a benign or malignant assessment of a disease using a stepwise regression method based on the disease scoring system;

determining the correlation between the predicted value of the two-coordinate disease recognition model and the malignancy and well of the disease using the independent variable correlated with the malignancy and well of the disease

In order to solve the above problems, the present invention further provides a disease intelligent recognition system based on fine-grained domain knowledge, comprising:

the characteristic extraction unit is used for acquiring a focus image of a patient to be identified; performing feature extraction on a focus image of a patient by using a double-coordinate disease recognition model to obtain focus region features and focus boundary features; wherein the two-coordinate disease identification model comprises a first coordinate model for dividing a lesion region and a second coordinate model for identifying lesion boundary characteristics;

the characteristic aggregation unit is used for acquiring focus aggregation characteristics according to the focus region characteristics and the focus boundary characteristics through a self-attention network;

the identification unit is used for carrying out classification identification on the focus aggregation characteristics to obtain a disease identification result of the patient; wherein the disease recognition result comprises benign/malignant class determination of the lesion and a lesion location.

In order to solve the above problem, the present invention also provides an electronic device, including:

a memory storing at least one instruction; and

and the processor executes the instructions stored in the memory to realize the steps in the disease intelligent identification method based on the fine-grained domain knowledge.

In order to solve the above problem, the present invention further provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the above method for intelligently identifying a disease based on fine-grained domain knowledge.

The invention relates to a disease intelligent identification method, a system, electronic equipment and a storage medium based on fine-grained domain knowledge, which are characterized in that a focus image of a patient to be identified is obtained; performing feature extraction on a focus image of a patient by using a double-coordinate disease identification model to obtain focus region features and focus boundary features; wherein the dual-coordinate disease recognition model comprises a first coordinate model for lesion region division and a second coordinate model for lesion boundary feature recognition; acquiring focus aggregation characteristics according to focus region characteristics and focus boundary characteristics through a self-attention network; classifying and identifying the focus aggregation characteristics to obtain a disease identification result of a patient; wherein the disease identification result comprises benign/malignant classification judgment and lesion position of a lesion; making fine-grained information of disease classification variables interpretable; the qualitative and even quantitative analysis of disease focus and other interested areas by assisting doctors is realized, and the accuracy and reliability of medical diagnosis are greatly improved; the system can also play an important auxiliary role in medical teaching, operation planning, operation simulation and various medical researches.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a schematic flow chart of a disease intelligent identification method based on fine-grained domain knowledge according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a schematic framework of a disease intelligent identification method based on fine-grained domain knowledge according to an embodiment of the present invention;

fig. 3 is a schematic diagram of another principle of the disease intelligent identification method based on fine-grained domain knowledge according to an embodiment of the present invention;

fig. 4 is a schematic network structure diagram of a disease intelligent identification method based on fine-grained domain knowledge according to an embodiment of the present invention;

fig. 5 is a schematic block diagram of an intelligent disease recognition system based on fine-grained domain knowledge according to an embodiment of the present invention;

fig. 6 is a schematic diagram of an internal structure of an electronic device for implementing a disease intelligent identification method based on fine-grained domain knowledge according to an embodiment of the present invention;

the implementation, functional features and advantages of the present invention will be further described with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Fig. 1 is a schematic flow chart of a disease intelligent identification method based on fine-grained domain knowledge according to an embodiment of the present invention. The method may be performed by a system, which may be implemented by software and/or hardware.

The disease intelligent identification method based on fine-grained domain knowledge is mainly suitable for an artificial intelligent diagnosis scene of thyroid nodules. In the prior art, the thyroid nodule is diagnosed by observing the ultrasonic image of the thyroid nodule, which is often determined by the experience of a doctor, so that the time consumption is more, and the accuracy of the determination result can be influenced by the subjective factors of the doctor.

Artificial Intelligence (AI): a theory, method, technique and application system for simulating, extending and expanding human intelligence, sensing environment, acquiring knowledge and using knowledge to obtain optimal results by using a digital computer or a machine controlled by a digital computer. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the implementation method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making. The artificial intelligence technology is a comprehensive subject, and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence base technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision (Computer Vision, CV): the method is a science for researching how to make a machine look, and particularly relates to a method for replacing human eyes with a camera and a computer to perform machine vision such as identification, tracking, measurement and the like on a target, and further performing image processing to make the computer processed into an image more suitable for human eyes to observe or transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. The computer vision technology generally includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning, map construction and the like, and also includes common biometric technologies such as face recognition, fingerprint recognition and the like.

Based on computer vision and artificial intelligence technology, the invention relates to a disease intelligent identification method, a system, electronic equipment and a storage medium based on fine-grained domain knowledge, which comprises the steps of obtaining a focus image of a patient to be identified; performing feature extraction on a focus image of a patient by using a double-coordinate disease recognition model to obtain focus region features and focus boundary features; wherein the two-coordinate disease identification model comprises a first coordinate model for dividing a lesion region and a second coordinate model for identifying lesion boundary characteristics; acquiring focus aggregation characteristics according to focus region characteristics and focus boundary characteristics through a self-attention network; classifying and identifying the focus aggregation characteristics to obtain a disease identification result of a patient; wherein the disease identification result comprises benign/malignant class determination and focus position of the focus, so that fine-grained information of thyroid nodule classification variables has interpretability; the qualitative and even quantitative analysis of thyroid lesions and other interested areas by assisting doctors is realized, and the accuracy and reliability of medical diagnosis are greatly improved; the system can also play an important auxiliary role in medical teaching, operation planning, operation simulation and various medical researches.

As shown in fig. 1, in this example, the present invention will be specifically described by taking thyroid diseases as an example. The intelligent disease identification method based on fine-grained domain knowledge comprises the steps of S110-S130.

And S110, acquiring a focus image of the patient to be identified.

The widely used medical image types mainly include Computed Tomography (CT), magnetic Resonance Imaging (MRI), nuclear Medicine Imaging (NMI), and Ultrasonic Imaging (UI). After the medical image (i.e., lesion image) is obtained, the image is pre-processed, including, but not limited to, techniques and processes for segmenting into specific regions with unique properties and extracting the object of interest. From a mathematical point of view, image segmentation is the division of a digital image into a plurality of mutually disjoint image sub-regions (sets of pixels), which is also a labeling process, i.e. pixels belonging to the same region (having equal visual characteristics) are assigned the same label.

S120, extracting features of the focus image of the patient by using a double-coordinate disease recognition model to obtain focus region features and focus boundary features; wherein the two-coordinate disease identification model comprises a first coordinate model used for dividing a lesion region and a second coordinate model used for identifying lesion boundary characteristics.

Specifically, the training method of the two-coordinate disease recognition model comprises steps S121-S123.

S121, preprocessing a sample containing the marking information of the focus area to obtain a lesion area mask and a normal area mask; obtaining a trained first coordinate model based on the diseased region mask and the normal region mask; preprocessing a sample containing the lesion dot matrix marking information to obtain vectorization representation of the lesion dot matrix; obtaining a trained second coordinate model based on vectorization representation of the focus lattice; wherein the first coordinate model is a Cartesian coordinate model; the second coordinate model is a polar coordinate model.

Namely, respectively performing feature extraction on the preprocessed focus images of the patient by utilizing a polar coordinate feature model and a Cartesian coordinate feature model; extracting a regional characteristic vector of a focus image through a polar coordinate characteristic model; and extracting an edge feature vector of the focus image through the Cartesian coordinate feature model.

Fig. 2-4 collectively illustrate the principles of the two-coordinate disease recognition model. Fig. 2 is a schematic diagram of a principle framework of an intelligent disease identification method based on fine-grained domain knowledge according to an embodiment of the present invention; fig. 3 is a schematic diagram of another principle of the disease intelligent identification method based on fine-grained domain knowledge according to an embodiment of the present invention.

As shown in fig. 2 and fig. 3, in order to utilize domain Knowledge as much as possible and make the Thyroid nodule classification method based on smart medicine more interpretable and accurate, the present invention proposes an MTL framework based on Knowledge dual Image Coordinates, namely Two Image Coordinates for Knowledge Embedding in Thyroid nodule due (tick), for diagnosing Thyroid nodule benign and malignant, embedding Two common domain Knowledge of coarse granularity and fine granularity. In the Domain Knowledge Representation (Domain Knowledge Representation) stage, all Knowledge is divided into region-based (region-based) and edge-based (margin-based) categories. After the field knowledge representation stage, integrating the first coordinate model and the second coordinate model to obtain a Bi-coordinate model, namely a thyroid classification identification model; the thyroid classification recognition model is used to learn the main tasks of learning and predicting the two branches of these knowledge, and a relationship module to further fuse the characteristics of the two branches to further improve performance. By learning this knowledge and using a progressive learning strategy, the performance of the main benign and malignant classification task of each MTL model is steadily improved.

For the Cartesian feature extraction model, weighting summation and summation are carried out on the identified results, averaging processing is carried out, and binary classification is carried out on the fusion result; performing Cartesian coordinate system conversion on the recognition result of each focus image subjected to binary classification by adopting a bilinear interpolation method, and marking a connected region on the converted result; in the mark communication region, the region is determined as a thyroid nodule focal region and is determined as a malignant or benign nodule focal region.

In a specific implementation process, for the second coordinate model, the original thyroid nodule focus image is first converted from a cartesian product coordinate system to a polar coordinate system, and the thyroid nodule focus image under the polar coordinate system is preprocessed, including removing interference information such as an interfering catheter at the top and the like, and removing noise. Each column of the pre-processed thyroid nodule lesion image is then taken as a sample reconstruction dataset. The specific implementation manner may be, but is not limited to, randomly and respectively selecting 10 ten thousand positive and negative samples from a data set to form a training set, randomly selecting 5 times, respectively establishing learning models by using stacked self-coding, to obtain 5 learning models in total, then fusing the 5 learning models, and the fusion rule is to perform weighted summation and averaging on a plurality of obtained recognition results, and perform binary classification on the fusion result.

In prior art thyroid ultrasound examinations, when nodules are found, the radiologist typically locates some important features according to authoritative TI-RADS (e.g., ACR TI-RADS) to assess the risk of the nodule. In the present invention, as shown in FIG. 2, the region-based image patch x is given in a Cartesian coordinate system, and the radiologist labels the corresponding region-based knowledge for segmentation, which is represented as a binary mask. Boundary-based is, in ACR TI-RADS, based on the shape, edges and features of the echogenic lesion. As shown in fig. 2, in consideration of labeling cost and importance of features, the following boundary-based features are selected for labeling and model learning: clear, burred, angled, smooth, sound shadow. Among them, thyroid TI-RADS grading criteria published by ACR (american society of radiology) in 2017, 5 scores mentioned in ACR criteria (nodule composition, sound, morphology, margin, calcified hyperechoic).

And S122, integrating the trained first coordinate model and the trained second coordinate model based on a multi-head self-attention module to obtain a double-coordinate disease recognition model.

And S123, training the double-coordinate disease recognition model by using a gradient back propagation algorithm based on the loss function until convergence.

Specifically, thyroid ultrasound diagnosis field knowledge is divided into two main categories, namely region-based knowledge and boundary-based knowledge, and different attributes of regions involved in the two categories of knowledge are represented in the form of masks and vectors respectively. Representing the region feature vector of the focus image extracted by the polar coordinate feature model by using a mask mode; extracting an edge feature vector of the lesion image through the Cartesian coordinate feature model is represented in the form of a plurality of vectors. In general, the process of using the disease intelligent identification method based on fine-grained domain knowledge to perform thyroid nodule classification identification includes: acquiring a medical image to be identified; respectively passing the obtained medical image through a polar coordinate characteristic model and a Cartesian coordinate characteristic model; and then, acquiring a medical image content identification result and a characteristic map corresponding to the medical image to be identified through a classifier. And respectively and independently training the polar coordinate feature model and the Cartesian coordinate feature model based on a reverse gradient propagation algorithm. After the two coordinate system models are trained, the two models are combined to form the bi-coordinate model of the present invention. And a gradient back propagation algorithm is used for training the combined model, and a cosine annealing learning rate strategy with warmup is used for ensuring the stability of the training process and the rapid convergence. In terms of loss functions, focal loss is used for classification variables and vector regression tasks, and Dice loss is used for segmentation tasks. That is, it is desirable to extract features consistent with the labels, the function of the network is feature extraction, and the final goal is that the feature extraction result of the network is consistent with the labeled features. And finally, inputting the image of the thyroid lesion to be recognized for the trained bi-coordinate disease recognition model, wherein the bi-coordinate disease recognition model can respectively output the type and the corresponding position of the characteristic. Specifically, the overall decision results include a benign and malignant decision, a lesion location, and lesion performance, wherein the lesion performance includes whether edges are sharp, whether burrs are present, whether edges are angled, whether edges are smooth, and whether an ultrasound contrast performance is present.

The first coordinate model and the second coordinate model are any one of a residual network (ResNet), a Visual Geometry Group network (VGGNet), and a compact-and-excitation network (SENet).

Specifically, the first coordinate model and the second coordinate model are integrated based on a multi-head self-attention structure to obtain the final classification features of the thyroid nodules; wherein the first and second coordinate models have the same encoder structure, mutually independent decoder and fully connected layer structure; training a thyroid nodule classifier by using the final classification characteristic of the thyroid nodule to obtain a dual-coordinate disease identification model; the two-coordinate disease recognition model is trained and constrained through a loss function.

As shown in fig. 3, the first coordinate model and the second coordinate model both use a multi-task learning approach to learn and predict domain knowledge through a common encoder-decoder architecture. In particular, it is desirable to map the potential feature space encoded by the encoder to various fine-grained domain knowledge, such as segmentation results and localization results. On the other hand, information from the encoder is also used to predict coarse-grained qualitative domain knowledge. The network architecture of the two coordinate systems of the first coordinate model and the second coordinate model is shown in fig. 3, and a hard parameter sharing structure is mainly used in the cartesian coordinate system because the corresponding auxiliary multitask has higher correlation. In the polar coordinate system, a channel separation structure is used, because the edge information has different positions and weak correlation. Wherein, hard parameter sharing means that the parameters of the feature extraction layers of the two networks are the same, namely an encoder; and the task channel splitting refers to that independent channels are used for each task to perform individual prediction, and not all the channels are used together to perform single prediction. In addition, the two structures have the same encoder structure, but different decoder structures for cartesian coordinate systems and polar coordinate feature models. And a splitting and combining module for feature de-entanglement is arranged between the decoder and the encoder of the second coordinate model. The main difference is that TCS adds an extra Split and Merge (SM) block for feature de-entanglement and uses separate decoders and fully connected layers to predict edge-based knowledge of different granularity. The feature de-entanglement comprises a scrolling operation on a local gradient corresponding to the feature map.

And S130, acquiring focus aggregation characteristics according to the focus region characteristics and the focus boundary characteristics through a self-attention network.

Fig. 4 is a schematic network structure diagram of the disease intelligent identification method based on fine-grained domain knowledge according to an embodiment of the present invention. As shown in fig. 4, after two coordinate system models, namely the first coordinate model and the second coordinate model, are trained, the two models, namely the first coordinate model and the second coordinate model, are combined to form the bi-coordinate model of the present invention. In a specific implementation, the characteristics of the two models, the first coordinate model and the second coordinate model, can be combined through two different attention mechanisms. And the integration may be two ways, one is to use a channel weighting method in the sense (CW-SE), and the other is to use a Self-Attention module (SA-Trans) in the transform decoder. That is, an attention-based feature integration approach, i.e., two different attention mechanisms to weigh the features of two coordinate models that have been adept at the primary task, rather than a simple Direct Concatenation (DC), was chosen.

The method for acquiring focus aggregation characteristics according to focus area characteristics and focus boundary characteristics through a self-attention network comprises the following steps: s1311, fusing the focus region characteristics and the focus boundary characteristics to obtain original fusion characteristics; s1312, after the original fusion features are subjected to convolution layer and activation function, obtaining significance weight features corresponding to input features; s1313, multiplying the saliency weight features corresponding to the input features and the original features to obtain aggregated features.

As shown in the upper part of fig. 4, in the SENet (Squeeze-and-Excitation Networks) algorithm model, a channel weighting method is used; i.e. different channels are given different weights. The preset weighting setting mode may include, but is not limited to: in this case, when the bicoordinate disease recognition model performs this step, the plurality of sampling feature information may be weighted to obtain a plurality of weighted values, and then the average value of the plurality of weighted values may be used as the detection score of the image in the candidate frame of the lesion area. The weighted value corresponding to each sampling characteristic information can be preset in the dual-coordinate disease recognition model by a user in advance or obtained by training through a certain training method.

In a specific embodiment, the method for acquiring the focus aggregation feature according to the focus area feature and the focus boundary feature by the self-attention network comprises the following steps: s1321, performing feature de-entanglement on the focus boundary features by using a split merge module to obtain a plurality of focus boundary feature blocks; s1322, marking a plurality of focus boundary characteristic blocks and focus region characteristic blocks, and embedding category marking characteristic blocks to obtain marked focus characteristic blocks; and S1323, acquiring aggregation characteristics by using the marked focus characteristic block by using a multi-head self-attention module.

Specifically, as shown in the lower half of fig. 4, in the integration process, two stacked self-attention blocks are employed, and a multi-head mechanism is used therein. Multi-headed Self-attention structure, a standard algorithmic structure. FIG. 4 corresponds to the relationship module in the right hand side of FIG. 2. The outputs of the two coordinate models are combined to output the final B/M (benign/malignant).

In the self-attention structure, by labeling five feature blocks for a fixed subtask in the polar feature model (tokenize), an edge-sharp feature block label, a burr feature block label, an edge-angulation feature block label, an edge-smooth feature block label, and an ultrasound image feature block label. In addition, all features in the encoder are compressed and labeled in the cartesian coordinate system model, and an additional label is introduced to represent the category information. The lesion dot matrix marking information of the second coordinate model comprises benign and malignant classification marks, edge clear feature block marks, burr feature block marks, edge angulation feature block marks, edge smooth feature block marks and ultrasonic image feature block marks. The method comprises the following steps of marking an edge clear feature block, a burr feature block, an edge angled feature block, an edge smooth feature block and an ultrasonic image feature block, wherein the edge clear feature block mark, the burr feature block mark, the edge angled feature block mark, the edge smooth feature block mark and the ultrasonic image feature block mark are marks of five original feature blocks. Newly added is a classification mark of benign and malignant.

S140, classifying and identifying the focus aggregation characteristics to obtain a disease identification result of the patient; wherein the disease identification result comprises benign/malignant classification judgment and lesion position of the lesion. Lesion location may also include whether the lesion representation includes sharp edges, the presence of burrs, angled edges, smooth edges, and ultrasound contrast representation.

Training a thyroid nodule classifier by using the final classification characteristic of the thyroid nodule to obtain a dual-coordinate disease identification model; the bi-coordinate disease recognition model is trained and constrained through a loss function. Specifically, the method comprises the steps of obtaining the grading distribution of disease benign and malignant evaluation and the grading distribution of lesion position evaluation of an input sample containing lesion region marking information through a two-coordinate disease identification model; obtaining a loss function between the score distribution of the disease benign and malignant assessment and the score distribution of the original disease benign and malignant assessment corresponding to the sample image data set, and a loss function between the score distribution of the lesion position assessment and the score distribution of the original disease lesion position assessment corresponding to the sample image data set; and updating the network parameters of the dual-coordinate disease identification model according to the loss function until the mean square deviation of the grading distribution of disease benign and malignant evaluation and the mean square deviation of the grading distribution of lesion position evaluation both belong to a preset standard range.

In the embodiment, the coincidence degree of the prediction result of the double-coordinate disease identification model and the label is evaluated through a Dice loss function and a mean square error loss function; the cross-entropy loss function is optimized using the Adam algorithm, and the network parameters of the bi-coordinate disease recognition model are updated using the Focal loss function.

The training method of the double-coordinate disease recognition model further comprises the steps of establishing a disease scoring system by utilizing binary logistic regression analysis; screening for independent variables having a correlation with a good or bad assessment of a disease using a stepwise regression method based on the disease scoring system; and judging the correlation between the predicted value of the two-coordinate disease recognition model and the benign or malignant degree of the disease by using the independent variable correlated with the benign or malignant degree evaluation of the disease. That is, the statistical independent variables are analyzed by binary logistic regression to determine whether the predicted value is related to the benign or malignant disease, and the result shows that the predicted value of the two-coordinate disease recognition model is closely related to the classification task of benign and malignant diseases, which reflects that the method must be interpretable.

Specifically, a combined model (namely a two-coordinate disease recognition model) is trained by using a gradient back propagation algorithm, and a cosine annealing learning rate strategy with warmup is used to ensure that the training process is stable and can be converged quickly, wherein the strategy is specifically set to warmup 5epoch, cosine annealing 200 epoch, maximum learning rate 0.001 and minimum learning rate 0.0000001. In the training process, models of two branches, namely a polar coordinate feature model and a Cartesian coordinate feature model, are trained firstly, then the weight is fixed after 100 epochs, and an integrated module based on an attention mechanism, namely a two-coordinate disease recognition model, is trained. In the aspect of loss function, focal loss is used for classification variable and vector regression tasks, and dice loss is used for segmentation tasks.

In addition, the performance evaluation index selects two modes of quantitative analysis and qualitative analysis. For the main task, the area under the working curve (AUC), accuracy ACC, sensitivity SEN, specificity SPC of the common pair subjects were used. For the vector regression task, 1D Dice Similarity Coefficient (DSC) and mean square error MSE were used as evaluation indexes, and for the segmentation task, 2D DSC was used as an evaluation index.

In a specific embodiment, in the thyroid nodule data set used for training the two-coordinate disease recognition model, the labeling conditions of the training set and the verification set of the data set are shown in table 1, and the sample numbers of the training set, the verification set and the test set are shown in table 2.

Table 1: labeling conditions of training set and verification set

Table 2: number of samples in training set, validation set, and test set

Under different coordinate systems, the two-coordinate disease recognition model obtained by the above data set training and the general CNN network model are subjected to performance testing by using a test set, and the test results are shown in table 3.

Table 3: performance situation of the invention and a general CNN network model in a test set

As can be seen from the observation of the table 3, the general CNN network model has lower performance under different coordinate systems, and the two-coordinate disease identification model of the invention has the characteristics of optimal performance and obvious statistics after combining the domain knowledge and the fusion module.

The two-coordinate disease recognition model obtained by the above data set training is applied to benign and malignant evaluation scenes of thyroid nodules, and compared with evaluation results of doctors with different annual capital, and the comparison results are shown in table 4.

Table 4: the invention compares the evaluation results with those of doctors with different annual capital

As can be seen by observing Table 4, the evaluation performance score of the present invention is higher than the general physician evaluation performance score.

Due to the cost of annotation, there is little verification data for fine annotations, and further confirmation is needed as to whether the assisting task really learned a more meaningful representation of the feature. Therefore, in the embodiment, the authoritative TI-RADS is simulated, and a benign scoring system is established by using the reasoning result of the auxiliary task, so that the accuracy and the importance of the auxiliary task prediction are indirectly proved.

If the prediction result of the auxiliary task is obviously related to the benign and malignant conditions, the method is equivalent to not only predicting the main task but also providing more evidence which is considered to be understandable, thereby enhancing the interpretability. Logistic regression is performed by using qualitative and quantitative inferences from the helper task on the data in the non-fine labeled training set. The test set was then tested for benign and malignant classification using optimized logistic regression as a scoring method. The test results are shown in table 5.

As can be seen from the observation of the table 5, the auxiliary task prediction result of the disease intelligent identification method based on the fine-grained domain knowledge is indeed related to the main label. Therefore, the disease intelligent identification method based on fine-grained domain knowledge is proved to be interpretable.

In summary, the disease intelligent identification method based on fine-grained domain knowledge of the invention establishes a polar coordinate characteristic model and a Cartesian coordinate characteristic model by using the regional thyroid ultrasound diagnosis domain knowledge and the boundary thyroid ultrasound diagnosis domain knowledge, and performs characteristic extraction under two different coordinate systems to further obtain the judgment result of the thyroid nodule; the judgment result not only comprises good and malignant judgment, but also comprises various fine-grained information such as focus position, focus expression and the like, so that the fine-grained information of the thyroid nodule classification variable has interpretability. The method is convenient for assisting doctors to carry out qualitative and even quantitative analysis on the thyroid lesion and other interested areas, thereby greatly improving the accuracy and reliability of medical diagnosis; the system can also play an important auxiliary role in medical teaching, operation planning, operation simulation and various medical researches.

As shown in fig. 5, the present invention provides an intelligent disease identification system 500 based on fine-grained domain knowledge, and the present invention can be installed in an electronic device. According to the implemented functions, the disease intelligent recognition system 500 based on fine-grained domain knowledge may include a feature extraction unit 510, a feature aggregation unit 520, and a recognition unit 530. The units of the invention, which may also be referred to as modules, are a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function and that are stored in a memory of the electronic device.

In the present embodiment, the functions of the respective modules/units are as follows:

a feature extraction unit 510, configured to obtain a focus image of a patient to be identified; performing feature extraction on a focus image of a patient by using a double-coordinate disease identification model to obtain focus region features and focus boundary features; wherein the two-coordinate disease identification model comprises a first coordinate model for dividing a lesion region and a second coordinate model for identifying lesion boundary characteristics;

a feature aggregation unit 520, configured to obtain a focus aggregation feature according to the focus area feature and the focus boundary feature through a self-attention network;

an identifying unit 530, configured to perform classification and identification on the focus aggregation features, and obtain a disease identification result of the patient; wherein the disease recognition result comprises benign/malignant class determination of the lesion and a lesion location.

The disease intelligent recognition system 500 based on fine-grained domain knowledge obtains the focus image of a patient to be recognized; performing feature extraction on a focus image of a patient by using a double-coordinate disease identification model to obtain focus region features and focus boundary features; wherein the dual-coordinate disease recognition model comprises a first coordinate model for lesion region division and a second coordinate model for lesion boundary feature recognition; acquiring focus aggregation characteristics according to focus region characteristics and focus boundary characteristics through a self-attention network; classifying and identifying the focus aggregation characteristics to obtain a disease identification result of a patient; wherein the disease identification result comprises benign/malignant class determination and lesion position of a lesion, so that fine-grained information of thyroid nodule classification variables has interpretability; the qualitative and even quantitative analysis of thyroid lesions and other interested areas by assisting doctors is realized, and the accuracy and reliability of medical diagnosis are greatly improved; the system can also play an important auxiliary role in medical teaching, operation planning, operation simulation and various medical researches.

As shown in fig. 6, the present invention provides an electronic device 6 for a disease intelligent recognition method based on fine-grained domain knowledge.

The electronic device 6 may comprise a processor 60, a memory 61 and a bus, and may further comprise a computer program stored in the memory 61 and executable on said processor 60, such as a disease intelligent recognition program 62 based on fine-grained domain knowledge. The memory 61 may also include both an internal storage unit and an external storage device of the disease intelligent recognition system based on fine-grained domain knowledge. The memory 61 may be used not only to store the code of a disease smart recognition program or the like based on fine-grained domain knowledge, which is installed in application software, and various types of data, but also to temporarily store data that has been output or is to be output.

The memory 61 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 61 may in some embodiments be an internal storage unit of the electronic device 6, for example a removable hard disk of the electronic device 6. The memory 61 may also be an external storage device of the electronic device 6 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the electronic device 6. Further, the memory 61 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 61 may be used not only to store application software installed in the electronic device 6 and various types of data, such as codes of a disease intelligent recognition program based on fine-grained domain knowledge, etc., but also to temporarily store data that has been output or is to be output.

The processor 60 may be formed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be formed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 60 is a Control Unit (Control Unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions of the electronic device 6 and processes data by running or executing programs or modules (for example, a disease intelligent recognition program based on fine-grained domain knowledge, etc.) stored in the memory 61 and calling data stored in the memory 61.

The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 61 and at least one processor 60 or the like.

Fig. 6 shows only an electronic device with components, and it will be understood by those skilled in the art that the structure shown in fig. 6 does not constitute a limitation of the electronic device 6, and may comprise fewer or more components than those shown, or some components may be combined, or a different arrangement of components.

For example, although not shown, the electronic device 6 may further include a power source (such as a battery) for supplying power to various components, and preferably, the power source may be logically connected to the at least one processor 60 through a power management system, so that functions such as charge management, discharge management, and power consumption management are implemented through the power management system. The power supply may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 6 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.

Further, the electronic device 6 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used to establish a communication connection between the electronic device 6 and other electronic devices.

Optionally, the electronic device 6 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 6 and for displaying a visualized user interface.

It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.

The disease intelligent recognition program 62 based on fine-grained domain knowledge stored in the memory 61 of the electronic device 6 is a combination of instructions that, when executed in the processor 60, enable: acquiring a focus image of a patient to be identified; performing feature extraction on a focus image of a patient by using a double-coordinate disease identification model to obtain focus region features and focus boundary features; wherein the dual-coordinate disease recognition model comprises a first coordinate model for lesion region division and a second coordinate model for lesion boundary feature recognition; acquiring focus aggregation characteristics according to focus region characteristics and focus boundary characteristics through a self-attention network; classifying and identifying the focus aggregation characteristics to obtain a disease identification result of a patient; wherein the disease recognition result comprises benign/malignant class determination of the lesion and a lesion location.

Specifically, the processor 60 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, and details thereof are not repeated herein. It should be emphasized that, in order to further ensure the privacy and security of the disease intelligent identification program based on fine-grained domain knowledge, the database high-available processing data is stored in the node of the block chain where the server cluster is located.

Further, the integrated modules/units of the electronic device 6 may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as separate products. The computer-readable medium may include: any entity or system capable of carrying said computer program code, a recording medium, a usb-disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).

An embodiment of the present invention further provides a computer-readable storage medium, where the storage medium may be non-volatile or volatile, and the storage medium stores a computer program, and when executed by a processor, the computer program implements: acquiring a focus image of a patient to be identified; performing feature extraction on a focus image of a patient by using a double-coordinate disease identification model to obtain focus region features and focus boundary features; wherein the two-coordinate disease identification model comprises a first coordinate model for dividing a lesion region and a second coordinate model for identifying lesion boundary characteristics; acquiring focus aggregation characteristics according to focus region characteristics and focus boundary characteristics through a self-attention network; classifying and identifying focus aggregation characteristics to obtain a disease identification result of a patient; wherein the disease identification result comprises benign/malignant classification judgment and lesion position of the lesion.

Specifically, the specific implementation method of the computer program when being executed by the processor may refer to the description of the relevant steps in the intelligent disease identification method based on fine-grained domain knowledge in the embodiment, which is not described herein again.

In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus, system, and method may be implemented in other ways. For example, the system embodiments described above are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.

The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.

The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a string of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, which is used for verifying the validity (anti-counterfeiting) of the information and generating a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.

Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or systems recited in the system claims may also be implemented by one unit or system in software or hardware. The terms second, etc. are used to denote names, but not to denote any particular order.

Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A disease intelligent identification method based on fine-grained domain knowledge is characterized by comprising the following steps:

acquiring a focus image of a patient to be identified;

classifying and identifying the focus aggregation characteristics to obtain a disease identification result of a patient; wherein the disease recognition result comprises benign/malignant class determination of the lesion and a lesion location.

2. The disease intelligent identification method based on fine-grained domain knowledge according to claim 1, wherein the method for obtaining focus aggregation characteristics according to focus area characteristics and focus boundary characteristics through a self-attention network comprises,

3. The disease intelligent recognition method based on fine grained domain knowledge as claimed in claim 1, wherein the method for obtaining focus aggregate features from focus area features and focus boundary features through a self-attention network comprises,

performing feature disentanglement on the focus boundary features by using a splitting and merging module to obtain a plurality of focus boundary feature blocks;

4. The disease intelligent recognition method based on fine-grained domain knowledge as claimed in claim 3, wherein the training method of the two-coordinate disease recognition model comprises,

preprocessing a sample containing the marking information of the focus area to obtain a mask of the focus area and a mask of a normal area; obtaining a trained first coordinate model based on the lesion area mask and the normal area mask; preprocessing a sample containing the lesion dot matrix marking information to obtain vectorization representation of the lesion dot matrix; obtaining a trained second coordinate model based on vectorization representation of the focus lattice; wherein the first coordinate model is a cartesian coordinate model; the second coordinate model is a polar coordinate model;

5. The intelligent disease identification method based on fine-grained domain knowledge as recited in claim 4, wherein the method for training a two-coordinate disease identification model by using a gradient back propagation algorithm until convergence based on a loss function comprises,

and updating the network parameters of the dual-coordinate disease identification model according to the loss function until the mean square deviation of the grading distribution of disease benign and malignant evaluation and the mean square deviation of the grading distribution of lesion position evaluation both belong to a preset standard range.

6. The disease intelligent identification method based on fine-grained domain knowledge according to claim 4,

the focus dot matrix marking information comprises benign and malignant characteristic block marks, clear edge characteristic block marks, burr characteristic block marks, angled edge characteristic block marks, smooth edge characteristic block marks and ultrasonic image characteristic block marks.

7. The disease intelligent recognition method based on fine-grained domain knowledge as claimed in claim 4, wherein the training method of the two-coordinate disease recognition model further comprises,

screening for independent variables having a correlation with a good or bad assessment of a disease using a stepwise regression method based on the disease scoring system;

and determining the correlation between the predicted value of the two-coordinate disease recognition model and the malignancy of the disease by using the independent variable correlated with the malignancy evaluation of the disease.

8. An intelligent disease identification system based on fine-grained domain knowledge, comprising:

the characteristic extraction unit is used for acquiring a focus image of a patient to be identified; performing feature extraction on a focus image of a patient by using a double-coordinate disease identification model to obtain focus region features and focus boundary features; wherein the dual-coordinate disease recognition model comprises a first coordinate model for lesion region division and a second coordinate model for lesion boundary feature recognition;

9. An electronic device, characterized in that the electronic device comprises:

at least one processor; and the number of the first and second groups,

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the steps of the fine-grained domain knowledge based intelligent identification method of diseases as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, storing a computer program, wherein the computer program, when executed by a processor, implements the fine-grained domain knowledge-based intelligent disease identification method according to any one of claims 1 to 7.