CN117235670A - Visual solution method for medical imaging problems based on fine-grained cross-attention - Google Patents

Abstract

The application relates to a medical image problem vision solving method based on fine grain cross attention. The method comprises the following steps: acquiring a radioactive medical image and text data of symptom questions corresponding to the radioactive medical image, carrying out local feature extraction on the radioactive medical image by adopting a fine-granularity visual feature extraction module to obtain local image features, carrying out feature extraction on the text data by adopting a text feature extraction module to obtain text features, inputting multi-mode features consisting of the local image features and the text features into a cross-mode encoder module to carry out multi-mode feature fusion to obtain fused features, inputting the fused features into an answer prediction module to carry out answer prediction to obtain an answer prediction result, and solving the symptom questions according to the answer prediction result, thereby improving the accuracy of medical visual question solutions.

Description

Visual solution method for medical imaging problems based on fine-grained cross-attention

Technical field

The present application relates to the field of image processing technology, and in particular to a visual solution method for medical imaging problems based on fine-grained cross-attention.

Background technique

Medical Vision Question Answering is designed to accurately answer clinical questions presented in medical images. Despite its huge potential in the healthcare industry and services, the technology is still in its infancy and is far from practical use. Medical visual question answering tasks are very challenging due to the huge diversity of clinical problems across different types of questions and the differences in required visual reasoning skills. Medical visual question answering is a domain-specific visual question answering question that requires interpretation of medically relevant visual concepts by considering image and verbal information. Specifically, the medical visual question answering system aims to take a medical image and a clinical question about the image as natural language input and output the correct answer. Medical vision question answering can help patients get timely feedback on inquiries and make more informed decisions. It can reduce pressure on medical facilities and save valuable medical resources for those in urgent need. It could also help doctors get a second opinion on diagnosis and reduce the high cost of training medical professionals.

Among related technologies, visual and linguistic reasoning require understanding of visual concepts and linguistic semantics, and most importantly, the alignment and relationship between these two modalities. Traditional visual question answering methods require large amounts of labeled data for training. Such large-scale data are generally not available in the medical field. Images in the medical field are fundamentally different from images in the general field. Therefore, it is not feasible to directly adopt a general domain visual question answering model in the medical field. Furthermore, medical image annotation is an expensive and time-consuming process. The application of visual question answering in the medical field has significantly influenced traditional medical research methods. A mature medical visual question answering system is of great help to patient diagnosis. Due to the complex diversity of clinical problems and the difficulty of multi-modal reasoning, general-domain visual question answering models are not attractive enough for feature alignment in medical images and text semantics, and their accuracy when applied to medical visual question answering is low.

Contents of the invention

Based on this, it is necessary to address the above technical problems and provide a visual solution method for medical imaging questions based on fine-grained cross-attention that can accurately answer medical visual questions.

A visual solution method for medical imaging problems based on fine-grained cross-attention. The method includes:

Obtain radioactive medical images and text data of symptom questions corresponding to the radioactive medical images;

The fine-grained visual feature extraction module of the medical visual question and answer model is used to extract local features of the radioactive medical image to obtain local image features;

Using the text feature extraction module of the medical visual question and answer model to perform feature extraction on the text data to obtain text features;

Perform multi-modal feature fusion on the cross-modal encoder module input to the medical visual question answering model using multi-modal features composed of local image features and text features to obtain fused features;

Input the fused features into the answer prediction module of the medical visual question answering model for answer prediction, and obtain answer prediction results;

The symptom question is answered based on the answer prediction result.

In one embodiment, the fine-grained visual feature extraction module using the medical visual question and answer model performs local feature extraction on the radioactive medical image to obtain local image features, including:

Input the radioactive medical image into the feature extraction unit of the fine-grained visual feature extraction module for feature extraction to obtain preliminary image features;

Input the preliminary image features to the full convolution unit of the fine-grained visual feature extraction module for processing, and obtain processed image features;

The processed image features are input to the fine-grained visual feature extraction unit of the fine-grained visual feature extraction module for feature extraction to obtain local image features.

In one embodiment, the cross-modal encoder module of the medical visual question answering model includes N cross-modal coding layers and a feature pooling layer connected in sequence;

The input of the first cross-modal coding layer is the output of the fine-grained visual feature extraction module and the output of the text feature extraction module. The output of the first cross-modal coding layer is the input of the second cross-modal coding layer. By analogy, the output of the last cross-modal coding layer is the input of the feature pooling layer, and the output of the feature pooling layer is the input of the answer prediction module.

In one embodiment, the cross-modal coding layer includes a first self-attention layer, a second self-attention layer, a first cross-attention layer, a second cross-attention layer, a first feedforward sub-layer and The second feedforward sublayer;

The output of the first self-attention layer is input to the first cross-attention layer and the second cross-attention layer, and the output of the second self-attention layer is input to the first cross-attention layer. layer and the second cross-attention layer, the output of the first cross-attention layer is input to the first feed-forward sub-layer and the second feed-forward sub-layer, and the second cross-attention layer The output of the layer is input into the first feed-forward sub-layer and the second feed-forward sub-layer.

In one embodiment, the processing expression of the first self-attention layer is:

;

The processing expression of the second self-attention layer is:

;

Among them, Attention ( Qi _, Ki _, Vi ₎ is the output of the first self-attention layer, Attention ( Qt , _Kt , _{Vt )} is the output of the second self-attention layer, and Vi is the output of the first self - attention _layer . The input features of the attention layer, Q _i is the input feature query of the first self-attention layer, K _i is the input feature key of the first self-attention layer, V _t is the input feature of the second self-attention layer, Q _t is the input feature query of the second self-attention layer, K _t is the input feature key of the second self-attention layer, d _k is the number of features, T is the transpose, and softmax (•) is the softmax function.

In one embodiment, the processing expression of the first cross-attention layer is:

;

The processing expression of the second cross-attention layer is:

;

in, is the output of the first cross-attention layer, /> For cross-modal attention operations from text to vision to capture the relationship between text and images,/> is the hidden state of the i -th position in the n -1th layer of the text feature extraction module,/> is the image representation of the i -th position in the previous layer of the n-th layer of the first cross-attention layer, is the image representation of the last position in the previous layer of the n- th layer of the first cross-attention layer, /> is the output of the second cross-attention layer, /> For cross-modal attention operations from vision to text to capture the relationship between text and images,/> is the hidden state of the j- th position in the n -1th layer of the text feature extraction module,/> is the image representation of the j-th position in the previous layer of the n-th layer of the second cross-attention layer, /> is the image representation of the last position in the previous layer of the nth layer of the second cross-attention layer.

In one embodiment, the answer prediction module includes: a first fully connected layer, a Relu function, a weight normalization layer and a second fully connected layer;

The first fully connected layer, the Relu function, the weight normalization layer and the second fully connected layer are connected in sequence.

In one embodiment, the loss function of the medical visual question answering model is:

;

Among them, L is the cross-entropy loss function, M is the number of samples, X is the real label, y is the prediction result, and p is the amount of information.

The above-mentioned visual answering method for medical imaging questions based on fine-grained cross-attention obtains radioactive medical images and text data of symptom questions corresponding to the radioactive medical images, and then uses the fine-grained visual feature extraction module of the medical visual question answering model to analyze the radioactive medical images. Local features are extracted from medical images to obtain more localized image features and suppress useless noise. The text feature extraction module of the medical visual question and answer model is used to extract features from the text data to obtain text features, which are then extracted from the local images. Multi-modal feature fusion composed of features and text features. Multi-modal feature fusion is performed on the cross-modal encoder module input to the medical visual question answering model, which can capture all potential local features through the mutual attention mechanism between regions and words. Alignment to achieve selective aggregation of context to obtain fused features, and then input the fused features into the answer prediction module of the medical visual question answering model for answer prediction, and obtain the answer prediction results to predict the answer based on the answer The prediction results answer the symptom questions, thereby improving the accuracy of medical vision question answering.

Description of drawings

Figure 1 is a schematic flowchart of a visual solution method for medical imaging problems based on fine-grained cross-attention in one embodiment;

Figure 2 is a schematic structural diagram of a cross-modal coding layer in an embodiment;

Figure 3 is a schematic diagram of the processing flow of the cross-modal coding layer in one embodiment;

Figure 4 is a schematic structural diagram of a medical visual question and answer model in one embodiment;

Figure 5 is a schematic structural diagram of the cross-modal encoder module and answer prediction module of the medical visual question and answer model in one embodiment;

Figure 6 is a schematic diagram of the multi-modal feature extraction process in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

In one embodiment, as shown in Figure 1, a visual solution method for medical imaging questions based on fine-grained cross-attention is provided. The application of this method to a terminal is used as an example to illustrate, including the following steps:

Step S220: Obtain radioactive medical images and text data of symptom questions corresponding to the radioactive medical images.

Among them, radioactive medical images can be the most commonly used radiology data, and the content of the images is the head, chest, abdomen, etc. of the body.

Among them, symptom questions can include: "abnormal situation", "attribute", "color", "quantity", "morphology", "organ type" and "section" and other questions.

Step S240: Use the fine-grained visual feature extraction module of the medical visual question and answer model to extract local features of the radioactive medical image to obtain local image features.

Among them, local image features can be the process of feature extraction of local areas in radioactive medical images, and are mainly used to describe the characteristics of local details or key points.

Among them, using the fine-grained visual feature extraction module of the medical visual question answering model to extract fine-grained visual features from the features of radioactive medical images can make the obtained fine-grained visual features more localized and make it easier to utilize the features in radioactive medical images. , thereby improving the accuracy of subsequent medical vision question answering.

It should be understood that the visual features extracted by the fine-grained visual feature extraction module can suppress useless noise in the key information of the fusion representation. The fine-grained visual feature extraction module uses fully convolutional layers instead of linear layers to reduce the number of parameters, uses fine-grained visual feature extraction layers to focus on the extraction and recognition of local features, and extracts distinguishable sums while retaining contextual information. complementary characteristics.

In one embodiment, the fine-grained visual feature extraction module of the medical visual question and answer model is used to extract local features of radioactive medical images to obtain local image features, including: inputting the radioactive medical images into the feature extraction unit of the fine-grained visual feature extraction module. Feature extraction to obtain preliminary image features; input the preliminary image features to the full convolution unit of the fine-grained visual feature extraction module for processing, and obtain the processed image features; input the processed image features to the fine-grained visual feature extraction module The fine-grained visual feature extraction unit performs feature extraction to obtain local image features.

In one embodiment, the feature extraction unit may be a neural network constructed using the ResNet-152 algorithm.

Among them, the ResNet-152 algorithm is used to extract regional features of medical images. In the image feature extraction process, previous studies extracted image features from the five convolutional layers of ResNet-152. After each convolutional layer that extracts features, they use a global average pooling layer. However, this ignores the fine-grained local information of the image. Instead of using a global average pooling layer, image features are extracted directly from the fifth convolutional layer in ResNet-152. First, the image features of the 3×224×224 image are extracted through the ResNet-152 model, and then the local visual features of the image are extracted through the full convolution unit and the fine-grained visual feature extraction unit.

Among them, the fine-grained visual feature extraction module of this application uses fully convolutional units instead of linear layers to reduce the number of parameters, and aligns the feature dimensions from the original 2048×7×7 to 768×7×7, and then fine-grained The visual feature module converts the 7×7 feature map into a sequence of length 49, and the final image feature dimension is 49×768, thus achieving the result of aligned feature dimensions.

Among them, the fine-grained visual feature extraction unit is used, dominated by the ROI pooling layer, to map the region of interest (ROI) to the corresponding position of the feature map according to the input image features. The mapped area is then divided into parts. The fine-grained visual feature extraction unit converts the feature map into a sequence of length 49. Finally, the image features obtained by the fine-grained visual feature extraction unit.

Step S260: Use the text feature extraction module of the medical visual question and answer model to extract features from the text data to obtain text features.

Among them, the text feature extraction module can use the BERT-base model, and the text feature extraction module can also use the BERT model.

It should be understood that by directly inputting the word sequence in the text data into the pre-trained text feature extraction module, the output obtained is the features of the words and sentences.

Step S280: perform multi-modal feature fusion on the cross-modal encoder module input to the medical visual question answering model using multi-modal features composed of local image features and text features to obtain fused features.

Among them, the self-attention mechanism of the cross-modal encoder module is used to pay attention to some details and perform analysis based on the part. The core is to determine the part to focus on based on the target, and further analyze after finding this part of the details; use cross-modal coding The cross-attention mechanism of the processor module allows the model to better understand and model the correlation between different modal data, thereby improving the performance of multi-modal tasks, and can utilize the modalities between image areas and sentence words. relationships to complement and enhance the matching of images and sentences.

Among them, the cross-modal encoder module takes image regions and sentence words as input. Each cross-modal layer in the cross-modal encoder module includes two self-attention sub-layers, two bidirectional cross-attention sub-layers, and two feed-forward sub-layers. To achieve robust cross-modal matching, a cross-modal encoder module is used, which can capture all potential local alignments through a mutual attention mechanism between regions and words.

In one embodiment, the cross-modal encoder module of the medical visual question answering model includes N cross-modal coding layers connected in sequence and a feature pooling layer; the input of the first cross-modal coding layer is the medical visual question answering model The output of the fine-grained visual feature extraction module and the output of the text feature extraction module, the output of the first cross-modal coding layer is the input of the second cross-modal coding layer, and so on, the last cross-modal coding layer The output of is the input of the feature pooling layer, and the output of the feature pooling layer is the input of the answer prediction module.

In one embodiment, the cross-modal coding layer includes a first self-attention layer, a second self-attention layer, a first cross-attention layer, a second cross-attention layer, a first feed-forward sub-layer and a second feed-forward sub-layer. Feeder layer; The output of the first self-attention layer is input to the first cross-attention layer and the second cross-attention layer, and the output of the second self-attention layer is input to the first cross-attention layer and the second cross-attention layer. In the force layer, the output of the first cross-attention layer is input to the first feed-forward sub-layer and the second feed-forward sub-layer, and the output of the second cross-attention layer is input to the first feed-forward sub-layer and the second feed-forward sub-layer. in the sub-layer.

Among them, the feed-forward sub-layer of the cross-modal coding layer enhances the representation ability of the captured visual features and problem description features. Considering that the attention mechanism may not fit the complex process well enough, the capabilities of the model can be enhanced by adding feedforward sublayers.

It should be understood that the input to the cross-modal encoder module is local image features and text features. Therefore, the first self-attention layer and the second self-attention layer of the first cross-modal encoding layer, one input local image feature, The other input is text features. After the cross-processing of the first cross-modal coding layer, the first feedforward sub-layer of the first cross-modal coding layer outputs the fusion of local image features and text features. Features, the second feed-forward sub-layer of the first cross-modal coding layer also outputs features after the fusion of local image features and text features, the first feed-forward sub-layer and the second feed-forward sub-layer of the first cross-modal coding layer The output of the feedforward sub-layer is input in parallel to the first self-attention layer and the second self-attention layer of the next cross-modal coding layer, without cross-input, and so on, until the first self-attention layer of the final cross-modal coding layer The features output by the feedforward sublayer and the second feedforward sublayer are input to the feature pooling layer.

Among them, as shown in Figure 2, in the cross-modal coding layer implementation, stacking (that is, using the output of the cross-modal coding layer of the nth layer as the input of the cross-modal coding layer of the (n+1) layer). In the nth layer, a bidirectional cross-attention layer is first applied, which contains two unidirectional cross-attention layers: one from language to vision and the other from vision to language.

Among them, the cross-modal coding layer can obtain a weight matrix or affinity matrix with scaled dot product attention. This application uses a self-attention layer to align entities between the two modalities before the cross-attention layer. The output of the self-attention layer is the input to the cross-attention layer to enhance the input of the (n+1)th layer of information. At the nth layer, a bidirectional cross-attention layer is adopted, which contains two cross-attention layers: one from vision to language and the other from language to vision. As shown in Figure 3, the result of the text feature query Q and key K after passing the softmax function is the attention coefficient matrix, which is multiplied by the value V of the visual feature to obtain the final attention output. Similarly, the result of visual feature query Q and key K after passing the softmax function is the attention coefficient matrix, which is multiplied by the value V of the text feature to obtain the final attention output. The two attention outputs are then fused as the output result. The cross-attention fusion mechanism has global learning capabilities and good parallelism. It can also complement and enhance image and sentence matching through modal relationships between image regions and sentence words. It selectively aggregates context based on spatial attention maps. These feature representations can realize mutual benefits. The feed-forward sub-layer of the cross-modal coding layer enhances the representation ability of the captured visual features and problem description features. Considering that the attention mechanism may not fit the complex process well enough, the capabilities of the medical visual question answering model are enhanced by adding feedforward sublayers.

In one embodiment, the processing expression of the first self-attention layer is:

;

In one embodiment, the processing expression of the second self-attention layer is:

;

Among them, Attention ( Qi _, Ki _, Vi ₎ is the output of the first self-attention layer, Attention ( Qt , _Kt , _{Vt )} is the output of the second self-attention layer, and Vi is the output of the first self - attention _layer . The input features of the attention layer, Q _i is the input feature query of the first self-attention layer, K _i is the input feature key of the first self-attention layer, V _t is the input feature of the second self-attention layer, Q _t is the input feature query of the second self-attention layer, K _t is the input feature key of the second self-attention layer, d _k is the number of features, T is the transpose, and softmax (•) is the softmax function.

In one embodiment, the processing expression of the first cross-attention layer is:

;

In one embodiment, the processing expression of the second cross-attention layer is:

;

in, is the output of the first cross-attention layer, /> For cross-modal attention operations from text to vision to capture the relationship between text and images,/> is the hidden state of the i -th position in the n -1th layer of the text feature extraction module,/> is the image representation of the i -th position in the previous layer of the n-th layer of the first cross-attention layer, is the image representation of the last position in the previous layer of the n- th layer of the first cross-attention layer, /> is the output of the second cross-attention layer, /> For cross-modal attention operations from vision to text to capture the relationship between text and images,/> is the hidden state of the j- th position in the n -1th layer of the text feature extraction module,/> is the image representation of the j-th position in the previous layer of the n-th layer of the second cross-attention layer, /> is the image representation of the last position in the previous layer of the nth layer of the second cross-attention layer.

Step S300: Input the fused features into the answer prediction module of the medical visual question answering model to predict the answer, and obtain the answer prediction result.

Among them, the answer prediction module can use the VQA classifier.

In one embodiment, the answer prediction module includes: a first fully connected layer, a Relu function, a weight normalization layer and a second fully connected layer; a first fully connected layer, a Relu function, a weight normalization layer and a second fully connected layer. The connection layers are connected in turn.

Step S320: Answer the symptom questions based on the answer prediction results.

The above-mentioned visual answering method for medical imaging questions based on fine-grained cross-attention obtains text data of radioactive medical images and symptom questions corresponding to radioactive medical images, and then uses the fine-grained visual feature extraction module of the medical visual question answering model to perform local analysis on the radioactive medical images. Feature extraction, the fine-grained visual features obtained are more localized, and useless noise is suppressed to obtain image features. The text feature extraction module of the medical visual question and answer model is used to extract features from the text data to obtain text features, which are then combined with local image features and Multi-modal features composed of text features perform multi-modal feature fusion on the cross-modal encoder module input to the medical visual question answering model. All potential local alignments can be captured through the mutual attention mechanism between regions and words to achieve selection. The context is comprehensively aggregated to obtain fused features, and then the fused features are input into the answer prediction module of the medical visual question answering model for answer prediction, and the answer prediction results are obtained to answer symptom questions based on the answer prediction results, thereby improving Improved the accuracy of medical vision question answering.

In one embodiment, a visual solution method for medical imaging problems based on fine-grained cross-attention is provided. As shown in Figures 4 and 5, a 3×224×224 radioactive medical image is first extracted from the local part through the ResNet-152 model. The image features are then extracted from the local visual features of the image (ie, local image features) through the full convolution layer (ie, full convolution unit) and the fine-grained visual feature extraction layer (ie, fine-grained visual feature extraction unit). Multi-modal feature fusion is performed through the local visual features and text features through the cross-modal encoder (i.e., cross-modal encoder module), and finally the predicted answer is generated through the VQA classifier. For example, the answer is: the opacity of the left signal area Increased sex.

In one embodiment, in training the medical visual question answering model, the VQA-RAD data set is used to train the medical visual question answering model. The VQA-RAD data set is currently the most commonly used radiology data set. , contains 315 pictures and 3515 question and answer pairs, each picture corresponds to at least one question and answer pair. The types of questions include 11 categories: "Abnormal situation", "Attribute", "Color", "Quantity", "Shape", "Organ type", "Others", "Aspect" and other questions. 58% of the questions were in the form of closed-ended questions and the rest were open-ended questions. The content of the image is the head, chest, abdomen, etc. of the body. It is necessary to manually divide the training set and test set. This data set is a high-quality data set for human-annotated question and answer pairs.

Among them, regarding the text feature extraction module in the medical visual question answering model, as shown in the dotted line in the middle of Figure 6, pad filling is used to align the sequence lengths. The text feature padding length is 49, and the local image feature padding length is 21. In order to keep the length of all sentences in the training set consistent, a "maximum length" is specified. For sentences that are shorter than this maximum length, they are completed; for sentences that exceed this maximum length, they are trimmed. As shown in Figure 6, after using [PAD] as padding, an indicator flag is needed to indicate that [PAD] is only used for padding and does not mean anything. A list called attention mask is used here to represent it, and its length is equal to the maximum length. For each sentence, if the word at the corresponding position is [PAD], the value of the element at this position of the attention mask is 0, otherwise it is 1. In the BERT model used in the text feature extraction module, in the pre-trained BERT model, token IDs are obtained by converting the input text into tokens in the corresponding vocabulary. The input text of the BERT model is usually composed of words or subwords (wordpieces), and each word or subword is mapped to a corresponding unique tag ID. The specific steps are: word segmentation, sub-word division, mapping to tag ID, adding special tags, filling and truncation. The mark serial number and attention_mask are input into the pre-trained BERT model, and the word embedding representation of each word is obtained. After being processed by the BERT model, the embedding representation of each word is obtained (this embedding representation contains the context of the entire sentence). Assuming that the BERT-base model is used, the dimension of each word embedding is 768.

Among them, in the training of the medical visual question answering model, the cross-entropy loss function is used to calculate the loss between the answer after passing through the answer prediction module based on the fusion features and the standard answer of the question description. The last layer of the answer prediction module of the medical visual question answering model obtains the score of each category; the score is passed through the Relu function to obtain the probability output; the category probability output predicted by the medical visual question answering model is calculated with the one hot form of the real category to calculate the cross entropy loss function .

In one embodiment, the loss function of the medical visual question answering model is:

;

Among them, L is the cross-entropy loss function, M is the number of samples, X is the real label, y is the prediction result, and p is the amount of information.

Among them, when training the medical visual question answering model, the cross-entropy loss function is used to calculate the loss between the answer after passing through the answer prediction module based on the fusion features and the standard answer of the question description. According to the multi-modal cross-entropy loss function, the medical The visual question answering model updates the gradient, calculates the loss, and adjusts parameters to obtain the parameters of the best medical visual question answering model, which improves the accuracy of medical visual question answering.

Among them, the cross-entropy loss function is used to guide the weight adjustment of the medical visual question and answer model. Through this function, you can know how to improve the weight coefficient. Calculate the loss between the answer after passing through the answer prediction module based on the fused features and the standard answer described in the question. Update the gradient of the medical visual question and answer model based on the multi-modal cross entropy loss function, calculate the loss, adjust the parameters, and obtain the parameters of the best model; finally, by calculating the loss, the visual question and answer model is trained according to the loss value, and the visual question and answer is optimized. parameters of the model. The total number of training rounds is set to 120 rounds, and the parameters of the medical visual question answering model with the highest accuracy in the verification set are the parameters of the final medical visual question answering model. The experimental results are shown in Table 1, and the displayed results are the results on the VQA-RAD data set validation set.

Table 1. Effect display of the visual solution method for medical imaging problems based on fine-grained cross-attention

;

According to Table 1, it can be concluded that the overall accuracy of the medical visual question answering model that adds fine-grained cross-attention is greatly improved compared with closed questions, and it can better capture the contextual correlation of medical images and language features, and improve the medical visual question answering model. Model performance and robustness.

It should be understood that although various steps in the flowchart of FIG. 1 are shown in sequence as indicated by arrows, these steps are not necessarily executed in the order indicated by arrows. Unless explicitly stated in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Figure 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution of these sub-steps or stages The sequence is not necessarily sequential, but may be performed in turn or alternately with other steps or sub-steps of other steps or at least part of the stages.

The technical features of the above embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, all possible combinations should be used. It is considered to be within the scope of this manual.

The above-described embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all fall within the protection scope of the present application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims (8)

1. A method for visual resolution of medical imaging problems based on fine-grained cross-attention, the method comprising:

acquiring a radioactive medical image and text data of symptom problems corresponding to the radioactive medical image;

carrying out local feature extraction on the radioactive medical image by adopting a fine-granularity visual feature extraction module of a medical visual question-answering model to obtain local image features;

the text feature extraction module of the medical visual question-answering model is adopted to perform feature extraction on the text data, so that text features are obtained;

the multi-modal feature pair consisting of the local image feature and the text feature is input into a cross-modal encoder module of the medical visual question-answering model to perform multi-modal feature fusion, so that the fused feature is obtained;

inputting the fused features into an answer prediction module of the medical visual question-answering model to perform answer prediction, so as to obtain an answer prediction result;

and solving the symptom problem according to the answer prediction result.

2. The fine-grained cross-attention-based medical image question visual solution method according to claim 1, wherein the fine-grained visual feature extraction module using a medical visual question-answering model performs local feature extraction on the radiological medical image to obtain local image features, comprising:

inputting the radioactive medical image into a feature extraction unit of the fine-granularity visual feature extraction module to perform feature extraction to obtain a primary image feature;

inputting the preliminary image features to a full convolution unit of the fine-granularity visual feature extraction module for processing to obtain processed image features;

and inputting the processed image features to a fine-granularity visual feature extraction unit of the fine-granularity visual feature extraction module for feature extraction to obtain local image features.

3. The fine grain cross-attention based medical imaging problem visual solution method of claim 1, wherein the cross-modality encoder module of the medical visual question model comprises N cross-modality encoding layers and one feature pooling layer connected in sequence;

the input of the first cross mode coding layer is the output of the fine-grained visual feature extraction module and the output of the text feature extraction module, the output of the first cross mode coding layer is the input of the second cross mode coding layer, and the like, the output of the last cross mode coding layer is the input of the feature pooling layer, and the output of the feature pooling layer is the input of the answer prediction module.

4. The fine granularity cross-attention based medical image problem visual solution of claim 3, wherein the cross-modality encoding layer comprises a first self-attention layer, a second self-attention layer, a first cross-attention layer, a second cross-attention layer, a first feed-forward sub-layer, and a second feed-forward sub-layer;

the output of the first self-attention layer is input into the first and second cross-attention layers, the output of the second self-attention layer is input into the first and second cross-attention layers, the output of the first cross-attention layer is input into the first and second feedforward sublayers, and the output of the second cross-attention layer is input into the first and second feedforward sublayers.

5. The fine-grained cross-attention based medical image problem visual solution method of claim 4, wherein the first self-attention layer is processed by the following expression:

；

the process expression of the second self-attention layer is:

；

wherein,Attention(Q _i ，K _i ，V _i ) Is the firstThe output from the attention layer(s),Attention(Q _t ，K _t ，V _t ) For the output of the second self-attention layer,V _i as an input feature for the first self-attention layer,Q _i for the input feature query of the first self-attention layer,K _i for the input feature key of the first self-attention layer,V _t as an input feature for the second self-attention layer,Q _t for the input feature query of the second self-attention layer,K _t is an input feature key of the second self-attention layer, d _k as a function of the number of features,Tfor the purpose of the transposition,softmax(.) is a softmax function.

6. The fine-grained cross-attention based medical image problem visual solution method of claim 4, wherein the first cross-attention layer is processed by the following expression:

；

the processing expression of the second cross-attention layer is:

；

wherein,for the output of the first cross-attention layer, < >>For a cross-modal attention manipulation from text to vision, capturing the relation between text and image,/->To the first text feature extraction modulen-layer 1 of the first layeriHidden status of individual location,/>First cross-attention layernThe first layer of the layersiImage representation of individual positions +.>First cross-attention layernThe image representation of the last position in the layer preceding the layer, is->For the output of the second cross-attention layer, < >>For a cross-modal attention manipulation from visual to text, capturing the relation between text and image,/->To the first text feature extraction modulen-layer 1 of the first layerjHidden status of individual location,/>Is the second cross-attention layernThe first layer of the layersjImage representation of individual positions +.>Is the second cross-attention layernImage representation of the last position in the previous layer of the layer.

7. The fine grain cross-attention based medical imaging problem visual solution method of claim 1, wherein the answer prediction module comprises: a first fully connected layer, a Relu function, a weight normalization layer, and a second fully connected layer;

the first full-connection layer, the Relu function, the weight normalization layer and the second full-connection layer are sequentially connected.

8. The fine grain cross-attention based medical imaging problem visual solution method of claim 7, wherein the loss function of the medical visual question answering model is:

；

wherein,Lin order to cross-entropy loss function,Mfor the number of samples to be taken,Xas a real tag it is possible to provide a real tag,yin order to predict the outcome of the result,pis the information quantity.