WO2022057309A1 - Lung feature recognition method and apparatus, computer device, and storage medium - Google Patents

Abstract

A lung feature recognition method and apparatus, a computer device, and a storage medium, which relate to the technical field of artificial intelligence. The method comprises: acquiring data to be recognized that comprises a lung image to be recognized and a lung text description to be recognized; extracting lung image features by means of a lung image recognition model and generating a lung image feature vector and an image recognition result, while extracting lung text features by means of a lung text recognition model and generating a lung text feature vector and a text recognition result; fusing the lung image feature vector and the lung text feature vector by means a lung fusion recognition model and by using using an attention mechanism, extracting image text fusion features for recognition, and obtaining a fusion recognition result; and obtaining a lung feature recognition result by means of voting. The described method achieves the accurate recognition of lung features and improves the accuracy and reliability of recognition. The described method is applicable to the fields of smart medical treatment and so on and can further promote the construction of smart cities.

Description

Lung feature identification method, device, computer equipment and storage medium

This application claims the priority of the Chinese patent application filed on September 21, 2020 with the application number 202010991495.4 and the invention titled "Method, Device, Computer Equipment and Storage Medium for Pulmonary Feature Recognition", the entire contents of which are approved by Reference is incorporated in this application.

technical field

The present application relates to the field of image classification of artificial intelligence, and in particular, to a method, device, computer equipment and storage medium for identifying lung features.

Background technique

In the current medical system, the identification of lung features mainly relies on medical personnel to manually identify lung image information based on their own experience. Because the movement of lung tissue is uneven and complex, the identification process not only costs medical personnel At the same time, in the process of identification, only the lung image information is often identified, and the main complaint information (text description for the lung image information) of the lung image information is not combined for identification. It is easy to lose the information of lung tissue movement, resulting in low accuracy and low efficiency.

SUMMARY OF THE INVENTION

The present application provides a lung feature identification method, device, computer equipment and storage medium, which realizes a lung feature identification model including a lung image identification model, a lung text identification model, and a lung fusion identification model, and uses attention It realizes the automatic, rapid and accurate identification of lung features, improves the accuracy and reliability of identification, and improves the efficiency of identification. This application is applicable to fields such as smart medical care, and can further promote the construction of smart cities.

A lung feature recognition method, comprising:

Acquiring data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified;

Inputting the data to be identified into a lung feature identification model, the lung feature identification model includes a lung image identification model, a lung text identification model and a lung fusion identification model;

Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result. The text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;

Using the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract and identify the fused features to obtain a fusion recognition result;

The image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.

A lung feature identification device, comprising:

a receiving module for acquiring data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified;

an input module for inputting the data to be identified into a lung feature identification model, the lung feature identification model comprising a lung image identification model, a lung text identification model and a lung fusion identification model;

The first recognition module is used for performing lung image feature extraction on the to-be-recognized lung image through the lung image recognition model, generating a lung image feature vector and an image recognition result, and simultaneously through the lung text recognition model performing lung text feature extraction on the description of the to-be-recognized lung text to generate a lung text feature vector and a text recognition result;

The second recognition module is used to fuse the lung image feature vector and the lung text feature vector using the attention mechanism through the lung fusion recognition model, and extract and recognize the fused features to obtain a fusion recognition result;

a voting module, configured to vote on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; The lung feature identification result indicates the lung feature category of the data to be identified.

A computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, the processor implementing the following steps when executing the computer-readable instructions:

Acquire data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;

Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result. The text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;

Using the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract and identify the fused features to obtain a fusion recognition result;

The image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.

One or more readable storage media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps:

Acquire data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;

Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result. The text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;

Using the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract and identify the fused features to obtain a fusion recognition result;

The image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.

The lung feature identification method, device, computer equipment and storage medium provided by the present application obtain the data to be identified; the data to be identified includes the image of the lung to be identified and the text description of the lung to be identified; the data to be identified is input to a lung feature recognition model including a lung image recognition model, a lung text recognition model and a lung fusion recognition model; the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, Generate lung image feature vectors and image recognition results, and perform lung text feature extraction on the description of the to-be-recognized lung text through the lung text recognition model to generate lung text feature vectors and text recognition results; The lung fusion recognition model uses the attention mechanism to fuse the lung image feature vector and the lung text feature vector, and extracts and recognizes the fused features to obtain a fusion recognition result; through the lung feature recognition model The image recognition result, the text recognition result and the fusion recognition result are voted, and the lung feature recognition result corresponding to the data to be recognized is obtained. In this way, the recognition to be recognized by the lung image recognition model is realized. Lung image, get the image recognition result, identify the text description of the lung to be recognized through the lung text recognition model, get the text recognition result, and then combine the lung image to be recognized and the text description to be recognized, use the attention mechanism, through the lung fusion The recognition model extracts the image and text fusion features for recognition, and obtains the fusion recognition result. Finally, according to the image recognition result, the text recognition result and the fusion recognition result, voting is carried out, and the lung feature recognition result is obtained. Recognize lung images and text descriptions of the lungs to be recognized, and automatically, quickly and accurately identify lung features through the multimodal model-based lung feature recognition model, improve the recognition accuracy and reliability, and improve the recognition effectiveness.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below, and other features and advantages of the application will become apparent from the description, drawings, and claims.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. , for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

1 is a schematic diagram of an application environment of a lung feature identification method in an embodiment of the present application;

2 is a flowchart of a lung feature identification method in an embodiment of the present application;

3 is a flowchart of step S30 of the lung feature identification method in an embodiment of the present application;

4 is a flowchart of step S30 of the lung feature identification method in another embodiment of the present application;

5 is a flowchart of step S40 of the lung feature identification method in an embodiment of the present application;

6 is a flowchart of step S50 of the lung feature identification method in an embodiment of the present application;

7 is a schematic block diagram of a lung feature identification device in an embodiment of the present application;

FIG. 8 is a schematic diagram of a computer device in an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The lung feature identification method provided by the present application can be applied in the application environment as shown in FIG. 1 , wherein the client (computer device) communicates with the server through the network. Among them, the client (computer equipment) includes but is not limited to various personal computers, notebook computers, smart phones, tablet computers, cameras and portable wearable devices. The server can be implemented as an independent server or a server cluster composed of multiple servers.

In one embodiment, as shown in FIG. 2, a method for identifying lung features is provided, and its technical solution mainly includes the following steps S10-S50:

S10: Acquire data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified.

Understandably, the lung image to be identified is an image collected by a lung imaging device, and the lung imaging device can be selected according to requirements, for example, the lung imaging device is a CT device, an X-ray machine, or a three-dimensional projection device, etc. etc., the lung text description is a description of the lung features in the to-be-recognized lung image, that is, the lung text is described as the main complaint information for the to-be-recognized lung image, the lung features The features reflected by the movement of lung tissue, such as lung features including pleural concave features, air bronchus features, lung vacuole features, lung spur features, lung ground glass-like features, etc., after collecting the to-be-identified lung image, And after inputting the text description of the lung to be recognized for the image of the lung to be recognized, the image of the lung to be recognized and the text description of the lung to be recognized are determined as the data to be recognized, and a recognition request is triggered. The identification request is a request for performing lung feature identification on the data to be identified, the identification request is received, and the data to be identified in the identification request is acquired.

S20, input the data to be recognized into a lung feature recognition model, where the lung feature recognition model includes a lung image recognition model, a lung text recognition model, and a lung fusion recognition model.

Understandably, the lung feature recognition model is a multimodal model that has been trained, and the lung feature recognition model can recognize the lung features of the data to be identified, and the lung feature recognition model includes lungs. part image recognition model, lung text recognition model and lung fusion recognition model, the lung image recognition model is to obtain the image recognition result by extracting the lung image features in the lung image to be recognized, and performing image recognition, And generate a lung image feature vector for the lung fusion recognition model, the lung image feature is the feature of the image space embodied by the movement of the lung tissue, and the network structure of the lung image recognition model can be based on the needs of image recognition. Setting, for example, the network structure of the lung image recognition model is VGG16, VGG19, GoogleNet or ResNet, etc. As an option, the network structure of the lung image recognition model is the network structure of VGG19; the lung text recognition model is By extracting the lung text features in the text description of the lungs to be identified, and performing text recognition, the text recognition results are obtained, and a lung text feature vector for the lung fusion recognition model is generated, where the lung text features are lungs The characteristics of the text space reflected by the movement of the external tissue, the network structure of the lung text recognition model can be set according to the needs of language recognition, for example, the network structure of the lung text recognition model is TextCNN, LSTM or BERT, etc., as a preference, The network structure of the described lung text recognition model selects the network structure of TextCNN; the described lung fusion recognition model is to use the attention mechanism to fuse the described lung image feature vector and the described lung text feature vector, and extract the fused The image-text fusion feature in the lung image feature vector and the lung text feature vector, and the fusion recognition result is identified, and the image-text fusion feature is the lung image feature vector and the lung text feature The implicit feature associated between the vectors, that is, the global similarity feature between the lung image feature vector and the lung text feature vector, the network structure of the lung fusion recognition model can be set according to requirements, For example, the network structure of the lung fusion recognition model is DenseNet, Deep LearningNet or LeNet, etc. Preferably, the network structure of the lung fusion recognition model is the network structure of DenseNet.

In one embodiment, before the step S20, that is, before the input of the data to be identified into the lung feature identification model, the method includes:

S201. Obtain a lung sample set, where the lung sample set includes a plurality of lung samples, the lung samples include a lung image and a lung text description associated with the lung image, and the lung samples are associated with the lung image. A lung feature class label association.

Understandably, the lung sample set is a collection of the lung samples, and the historical collection of the lung samples includes a lung image and a lung text description associated with the lung image. The lung sample is associated with a lung feature class label, the lung feature class label is a label related to the lung feature class marked on the lung sample, and the lung image is a historically collected through the lung The image picture of the lung collected by the photographing device, the lung text description is a description of the lung feature in the lung image associated therewith, and the lung feature category is a classification of the lung feature, For example, the lung feature categories include a pleural indentation feature class corresponding to a pleural indentation feature, an air bronchus feature class corresponding to an air bronchus feature, a lung vacuolar feature class corresponding to a pulmonary vacuole feature, and a lung burr feature corresponding to a pleural indentation feature class. Lung spur feature class and lung ground glass features corresponding to lung ground glass features.

S202, input the lung sample into a multimodal model containing initial parameters; the multimodal model includes a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model.

Understandably, the multimodal model is to match the similarity between images and texts, that is, to measure the similarity between an image and a piece of text (the global similarity between the image and the text), and identify the implicit relationship between the image and the text. The characteristics of the relationship are determined, and the classification result of the fusion of an image and a text is determined. The multimodal model includes the initial parameters, and the initial parameters include the lung sample image recognition model and the lung sample text recognition model. The parameters of the model and the lung sample fusion recognition model can be transferred directly from the parameters in the multimodal recognition models in other fields to the initial parameters in the multimodal model by means of transfer learning, simplifying The training process shortens the training time and improves the training efficiency. The multimodal model includes a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model. The lung image recognition model The lung sample image recognition model that has been trained, the lung text recognition model is the lung sample text recognition model that has been trained, and the lung fusion recognition model is the lung sample fusion that has been trained. Identify the model.

S203, performing the lung image feature extraction on the lung image by using the lung sample image recognition model to generate a lung sample image feature vector and an image sample recognition result, and simultaneously using the lung sample text recognition model to The lung text description performs the lung text feature extraction to generate a lung sample text feature vector and a text sample recognition result.

Understandably, the lung image feature is the feature of the image space embodied by the movement of lung tissue, the lung sample image feature vector is a vector matrix with the lung image feature, and the image sample recognition result is all The lung sample image recognition model identifies the results of the lung features in the lung image by the similarity of the image space based on the extracted lung image features, and the lung sample text feature vector is a feature vector with the lung image. A vector matrix of image features, and the text sample recognition result is that the lung sample text recognition model performs text space similarity according to the extracted lung text features to identify the lung features in the lung text description. result.

S204, using the attention mechanism to fuse the image feature vector of the lung sample and the text feature vector of the lung sample through the lung sample fusion recognition model, and learn to extract the image and text fusion features and recognition, to obtain a fusion sample recognition result.

Understandably, the image feature vector of the lung sample and the text feature vector of the lung sample are fused through the attention mechanism, and the learning to extract the image-text fusion feature is to capture the implicit similarity between the image and the text. Feature extraction, and local similarity measurement and extraction.

S205, voting on the image sample recognition result, the text sample recognition result, and the fusion sample recognition result, to obtain a sample recognition result.

S206: Determine a loss value according to the sample identification result and the lung feature category label.

Understandably, the sample identification result and the lung feature category label are input into the loss function of the multimodal model, and the loss value is calculated through the loss function.

S207, when the loss value does not reach a preset convergence condition, iteratively update the initial parameters of the multimodal model, until the loss value reaches the preset convergence condition, update the multimodal model after convergence Modal models were recorded as lung feature recognition models.

Understandably, the convergence condition may be a condition that the loss value is small and will not decrease after 6,000 calculations, that is, the loss value is small and will not decrease after 6,000 calculations. When descending again, stop training, and record the multimodal model after convergence as a lung feature recognition model; the convergence condition can also be the condition that the loss value is less than the set threshold, that is, when the loss value is less than the set threshold. When it is less than the set threshold, the training is stopped, and the multimodal model after convergence is recorded as a lung feature recognition model. In this way, when the loss value does not reach the preset convergence condition, the multimodal model is continuously adjusted. The initial parameters in the model are triggered, and the lung image feature extraction is performed on the lung image through the lung sample image recognition model to generate a lung sample image feature vector and an image sample recognition result. Part of the sample text recognition model extracts the lung text features from the lung text description, and generates the lung sample text feature vector and the text sample recognition results, which can continuously move closer to the accurate results, so that the accuracy of the recognition is higher. Come higher. In this way, the lung feature recognition of the multimodal model can be optimized, and the accuracy and reliability of the lung feature recognition are improved.

S30, performing lung image feature extraction on the to-be-recognized lung image by the lung image recognition model, generating a lung image feature vector and an image recognition result, and simultaneously using the lung text recognition model to perform the lung image feature extraction on the to-be-recognized lung image Lung text description performs lung text feature extraction to generate lung text feature vectors and text recognition results.

Understandably, the lung image recognition model performs channel splitting and convolution on the to-be-recognized lung image, thereby extracting the lung image features, and the lung image features are images embodied by the movement of lung tissue. Spatial features, the lung image recognition model includes a plurality of convolutional layers, the convolutional layers of the lung image recognition model can be marked as image convolutional layers, and through each of the lung image recognition model The image convolution layer convolves the to-be-identified lung image according to different convolution kernels, and generates the lung image feature vector corresponding to each image convolution layer, and the lung image feature vector has the lung image The dimension of each described lung image feature vector is different according to the difference of each image convolution layer, and the image recognition result is the lung image feature extracted by the lung image recognition model according to the The result of identifying the lung features by the similarity of the image space, the lung text recognition model performs word vector conversion on the description of the lung text to be identified, and then performs convolution to extract the lung text features. The lung text feature is the feature of the text space embodied by the movement of lung tissue, and the lung text recognition model includes a plurality of convolution layers, and the convolution layer of the lung text recognition model can be marked as text convolution Layer, through each text convolution layer in the lung image recognition model, the description of the lung text to be recognized is convolved according to different convolution kernels, and the lung text features corresponding to each text convolution layer are generated. vector, the lung text feature vector is a vector matrix with the lung text feature, the dimension of each described lung text feature vector is different according to the difference of each text convolution layer, and the text recognition result is the The lung text recognition model identifies the result of the lung features by performing text space similarity based on the extracted lung text features.

In one embodiment, as shown in FIG. 3 , in the step S30, the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, and a lung image feature vector is generated. and image recognition results, including:

S301, splitting the to-be-identified lung image into a red channel image, a green channel image and a blue channel image by using the lung image recognition model; the lung image recognition model is a network model constructed based on VGG19.

Understandably, the lung image to be identified is an image of three channels: red channel, green channel, and blue channel, that is, the lung image to be identified includes the red channel image corresponding to the red channel, and the green channel image corresponding to the red channel. The green channel image corresponding to the channel and the blue channel image corresponding to the blue channel, through channel splitting, the to-be-identified lung image is split into the red channel image, the green channel image and the The blue channel image, the red channel image is an image that reflects the degree of redness of each pixel through pixel values in the range of 0 to 255, and the green channel image is an image that reflects the redness of each pixel through pixel values in the range of 0 to 255. An image with a green level, and the blue channel image is an image in which the blue level of each pixel is represented by pixel values ranging from 0 to 255.

The lung image recognition model is a network model constructed based on VGG19, and the convolution depth of the lung image can be set to 19, that is, a network model with 19-level convolution layers.

S302, performing convolution extraction on the red channel image, the green channel image and the blue channel image respectively by using the lung image recognition model to obtain a red feature vector corresponding to the red channel image, The green feature vector corresponding to the green channel image and the blue feature vector corresponding to the blue channel image.

Understandably, the red channel image is convolved by the lung image recognition model to obtain the red feature vector, and the red feature vector is the vector embodied in the red space extracted from the lung image features, and is obtained by The lung image recognition model convolves the green channel image to obtain the green feature vector. The recognition model convolves the blue channel image to obtain the blue feature vector, where the blue feature vector is the vector embodied in the blue space extracted from the lung image features, and the red feature vector, The green feature vector and the blue feature vector are determined as the lung image feature vector.

S303: Perform image recognition on the lung image feature vector by using the lung image recognition model to obtain the image recognition result.

Understandably, image recognition is performed on the lung image feature vector by the lung image recognition model, and the image recognition is to perform fully connected classification according to the extracted lung image feature vector to obtain each lung feature category. The probability distribution of , so as to output the recognized image recognition result.

The present application realizes that the lung image to be recognized is split into a red channel image, a green channel image and a blue channel image through the lung image recognition model; the lung image recognition model is a network model constructed based on VGG19 ; Carry out convolution extraction on the red channel image, the green channel image and the blue channel image respectively by the lung image recognition model to obtain the lung image feature vector; by the lung image recognition model to The lung image feature vector is used for image recognition to obtain the image recognition result. In this way, it is realized by dividing the lung image to be recognized into a red channel image, a green channel image and a blue channel image, and based on VGG19 The constructed network model performs convolution on each channel image to extract the lung image features, obtains the lung image feature vector, and outputs the image recognition result according to the lung image feature vector, which can extract the lung image in the lung image to be identified. The lung feature categories are identified through the extracted lung image features, which provides a data basis for subsequent identification and improves the accuracy and reliability of identification.

In one embodiment, as shown in FIG. 4 , in the step S30, that is, the lung text feature extraction is performed on the description of the lung text to be identified by the lung text recognition model, and the lung text feature is generated. Vector and text recognition results, including:

S304, performing word segmentation on the description of the lung text to be recognized by the lung text recognition model, and constructing a text word vector corresponding to the description of the lung text to be recognized, and the lung text recognition model is constructed based on TextCNN network model.

Understandably, the word segmentation is to use a word dictionary to split the description of the lung text to be identified into individual words, and the word dictionary contains word vectors corresponding to all medical terms and words related to the lungs. , and then convert the split words into their corresponding word vectors, which can be converted by the conversion method of word2vec or Glove, and then splicing the converted word vectors to form the text word vector.

Wherein, the pulmonary text recognition model is a network model constructed based on TextCNN, that is, the pulmonary text recognition model has the network structure of TextCNN, and the convolution depth of the pulmonary text recognition model is set to 19, that is, it has 19 The network model of the hierarchical convolution layer, the convolution depth of the lung text recognition model is the same as the convolution depth of the lung image recognition model, so as to facilitate the recognition of the subsequent lung fusion recognition model.

S305, channel-expanding the text word vector to generate a first text word vector, a second text word vector, and a third text word vector.

Understandably, the channel expansion is a process of expanding the text word vector of a single channel to a vector matrix of a preset dimension and copying the vector matrix until the number of preset channels, that is, expanding the text word vector to the same size as the text word vector. The vector matrix of the same dimension of the lung image feature vector, the expansion method can be set according to the needs, and the vector matrix is copied into a vector matrix with the same number of channels as the lung image feature vector, so as to obtain the same vector matrix as the vector matrix. Describe the first text word vector, the second text word vector and the third text word vector.

S306, perform convolution extraction on the first text word vector, the second text word vector, and the third text word vector respectively by using the lung text recognition model, to obtain a correspondence with the first text word vector The first text feature vector of , the second text feature vector corresponding to the second text word vector, and the third text feature vector corresponding to the third text word vector.

Understandably, the first text word vector is convolved by the lung text recognition model to obtain the first text feature vector, and the second text word vector is processed by the lung text recognition model. convolving to obtain the second text feature vector, and convolving the third text word vector through the lung text recognition model to obtain the third text feature vector, wherein the first text word The convolution kernel for convolving the vector, the convolution kernel for convolving the second text word vector, and the convolution kernel for convolving the third text word vector may be different, that is, from different text spaces. The dimension of lung text feature vector is extracted, and the first text word vector, the second text word vector and the third text word vector are determined as the lung text feature vector.

S307, performing text recognition on the lung text feature vector by using the lung text recognition model to obtain the text recognition result.

Understandably, text recognition is performed on the lung text feature vector by the lung text recognition model, and the text recognition is to perform a fully connected classification according to the extracted lung text feature vector to obtain each lung feature category. The probability distribution of , so as to output the recognized text recognition result. The present application implements word segmentation for the description of the lung text to be identified through the lung text recognition model, and constructs a text word vector corresponding to the description of the lung text to be identified; the lung text recognition model is based on A network model constructed by TextCNN; channel expansion of the text word vector to generate a first text word vector, a second text word vector and a third text word vector; The word vector, the second text word vector, and the third text word vector are extracted by convolution to obtain a lung text feature vector, and text recognition is performed on the lung text feature vector by the lung text recognition model, Obtaining the text recognition result, in this way, it is realized that the first text word vector, the second text word vector and the third text word vector are generated by segmenting the description of the lung text to be identified and constructing a text word vector, and channel expansion. , Extracting the lung text features through the network model constructed based on TextCNN, obtaining the lung text feature vector, and outputting the text recognition result according to the lung text feature vector, the lung text features in the description of the lung text to be identified can be extracted, The lung feature category is identified through the extracted lung text features, which provides a data basis for subsequent identification and improves the accuracy and reliability of identification.

S40, using the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract and recognize the fused features to obtain a fusion recognition result.

Understandably, the attention mechanism is a mechanism learned by an additional feedforward neural network in neural network learning and recognition through the attention weight, and the lung image feature vector and the lung image feature vector and The implicit relationship between the lung text feature vectors, that is, the lung image feature vector and the lung text feature vector are carried out according to the weight parameters corresponding to each convolution layer learned through the attention mechanism. Weighted fusion, so as to obtain the fusion feature vector corresponding to each convolution layer, convolve all the fusion feature vectors, and extract the image-text fusion features, that is, extract the fused features, the image The text fusion feature is an implicit feature associated between the lung image feature vector and the lung text feature vector, that is, the global similarity between the lung image feature vector and the lung text feature vector. The feature is identified according to the extracted image-text fusion features, that is, the probability distribution of each lung feature category is classified by full connection, so as to output the fusion recognition result.

In one embodiment, as shown in FIG. 5 , in the step S40, the lung image feature vector and the lung text feature vector are fused by using the attention mechanism through the lung fusion recognition model. , and extract image-text fusion features for recognition, and obtain fusion recognition results, including:

S401, using the attention mechanism technology, through weighting parameters corresponding to each convolution layer in the lung fusion recognition model, weighted fusion of the lung image feature vector and the lung text feature vector to obtain a The fused feature vector corresponding to each convolutional layer.

Understandably, the attention mechanism technology is to enhance the useful information in the feature vector, that is, the weight parameters corresponding to each convolutional layer according to the useful vector of the lung image feature vector and the lung text feature vector. A weighted average is performed and fused to generate a fused feature vector corresponding to each convolutional layer.

Wherein, the convolution depth in the lung fusion recognition model is the same as the convolution depth of the lung image recognition model or the lung text recognition model, and the convolution depth in the lung fusion recognition model is preferably 19 levels.

In one embodiment, in step S401, the lung image feature vector includes a red feature vector, a green feature vector and a blue feature vector; the lung text feature vector includes a first text feature vector, a second text feature vector feature vector and third text feature vector;

The lung image recognition model, the lung text recognition model and the lung fusion recognition model all have the same convolution level, and each of the three models is provided with a convolution layer corresponding to each convolution level. ;

The described lung image feature vector and the described lung text feature vector are weighted and fused by the weight parameters corresponding to each convolutional layer in the described lung fusion recognition model to obtain a fusion corresponding to each convolutional layer. eigenvectors, including:

S4011, fuse the red feature vector corresponding to the same convolution level and the first text feature vector according to the first weight parameter corresponding to the convolution level to obtain a first fusion feature vector.

Understandably, the red feature vector and the first text feature vector corresponding to the same convolution level are weighted according to the first weight parameter of the convolution level, that is, the red feature vector and the first text feature Each vector value in the vector is weighted and averaged according to the first weight parameter to obtain the first fusion feature vector, and the red feature vector, the first text feature vector and the first fusion feature vector have the same dimensions .

S4012, fuse the green feature vector and the second text feature vector corresponding to the same convolution level according to the second weight parameter corresponding to the convolution level to obtain a second fusion feature vector.

Understandably, the green feature vector and the second text feature vector corresponding to the same convolution level are weighted according to the second weight parameter of the convolution level, that is, the green feature vector and the second text feature Each vector value in the vector is weighted and averaged according to the second weight parameter to obtain the second fusion feature vector, and the green feature vector, the second text feature vector and the second fusion feature vector have the same dimensions .

S4013, fuse the blue feature vector and the third text feature vector corresponding to the same convolution level according to the third weight parameter corresponding to the convolution level to obtain a third fusion feature vector.

Understandably, the blue feature vector and the third text feature vector corresponding to the same convolution level are weighted according to the third weight parameter of the convolution level, that is, the blue feature vector and the third Each vector value in the text feature vector is weighted and averaged according to the third weight parameter to obtain the third fusion feature vector, the blue feature vector, the third text feature vector and the third fusion feature vector dimensions are the same.

The execution order of the steps S4011, S4012 and S4013 is not limited, and may be executed in series or in parallel, and the first weight parameter, the second weight parameter and the third weight parameter may be the same, or can all be different.

S4014. Perform a weighted average of the first fusion feature vector, the second fusion feature vector, and the third fusion feature vector corresponding to the same convolution level to obtain the fusion feature vector.

Understandably, the weighted average is to average the first fusion feature vector, the second fusion feature vector and the third fusion feature vector after weighting, and the The first fusion feature vector, the second fusion feature vector and the third fusion feature vector are weighted and averaged to obtain the fusion feature vector corresponding to each convolution layer.

S402, extracting the image-text fusion feature on the fusion feature vector by using the lung fusion recognition model.

Understandably, the extraction process of the image-text fusion feature may be to convolve the fusion feature vector of the convolutional layer of the first layer, and then perform the convolution with the fusion feature vector of the convolutional layer of the next layer of the convolutional layer. The transfer feature vector is obtained by superposition, and then the transfer feature vector is convolved, and the transfer feature vector is continuously superimposed with the fusion feature vector of the convolution layer of the next layer to obtain the transfer feature vector, and the superimposed transfer feature vector is convolved until a one-dimensional The feature vector extraction process of .

S403 , performing recognition by the lung fusion recognition model according to the extracted image-text fusion feature, to obtain the fusion recognition result.

Understandably, the identification is performed according to the extracted image-text fusion feature by the lung fusion recognition model, and the identification is to obtain the probability distribution of each lung feature category according to the extracted image-text fusion feature, thereby outputting The identified fusion identification result.

The present application realizes the weighted fusion of the lung image feature vector and the lung text feature vector by using the attention mechanism technology and the weight parameters corresponding to each convolution layer in the lung fusion recognition model. , obtain the fusion feature vector corresponding to each convolution layer; perform the image-text fusion feature extraction on the fusion feature vector by the lung fusion recognition model; The image and text fusion features are used for recognition, and the fusion recognition result is obtained. In this way, the attention mechanism can be used to enhance the useful information in the image and the text, and the global similarity between the image and the text can be better captured. The lung image feature vector and the lung text feature vector are weighted and fused, and the image and text fusion features are extracted for identification, which can improve the accuracy and reliability of lung feature identification.

S50, voting on the image recognition result, the text recognition result and the fusion recognition result by using the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized; the lung The partial feature identification result indicates the lung feature category of the data to be identified.

Understandably, the voting is to perform a weighted average of the probability values corresponding to the same lung feature category in the image recognition result, the text recognition result and the fusion recognition result, and finally determine that the probability value is the highest. and the lung feature category with the highest probability value is used as the lung feature identification result, and the lung feature identification result includes the identified lung feature category and the probability value corresponding to the category, so The lung feature recognition result indicates the lung feature category of the data to be identified, and the lung feature is the feature embodied by the movement of lung tissue. feature, lung burr feature, lung ground glass-like feature, etc., the lung feature category is a classification of the lung feature, for example, the lung feature category includes the pleural depression feature class corresponding to the pleural depression feature, The air bronchus feature class corresponding to the air bronchus feature, the lung vacuole feature class corresponding to the lung vacuole feature, the lung spur feature class corresponding to the lung spur feature, and the lung ground glass feature corresponding to the lung ground glass feature.

The present application realizes by acquiring the data to be recognized in the recognition request; the data to be recognized includes the lung image to be recognized and the text description of the lung to be recognized; the data to be recognized is input into a recognition model containing a lung image , the lung feature recognition model of the lung text recognition model and the lung fusion recognition model; the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, and the lung image feature vector and the image are generated. Recognition results, while performing lung text feature extraction on the description of the to-be-recognized lung text by the lung text recognition model, to generate a lung text feature vector and a text recognition result; using the attention through the lung fusion recognition model The mechanism fuses the lung image feature vector and the lung text feature vector, and extracts image-text fusion features for recognition, and obtains a fusion recognition result; through the lung feature recognition model, the image recognition result, the text The identification results and the fusion identification results are voted for, and the lung feature identification results corresponding to the data to be identified are obtained. In this way, the identification of the lung images to be identified through the lung image identification model is realized, and the image identification results are obtained. The lung text recognition model recognizes the text description of the lungs to be recognized, and obtains the text recognition result, and then combines the lung image to be recognized and the text description to be recognized, and uses the attention mechanism to extract the image and text fusion features through the lung fusion recognition model for recognition. Obtain the fusion recognition result, and finally vote according to the image recognition result, the text recognition result and the fusion recognition result, and obtain the lung feature recognition result, which realizes the combination of the lung image to be recognized and the lung text to be recognized. Described, through the lung feature recognition model based on the multimodal model, the lung features are automatically, quickly and accurately identified, the recognition accuracy and reliability are improved, and the recognition efficiency is improved.

In one embodiment, as shown in FIG. 6 , in step S50, the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model. , to obtain the lung feature identification results corresponding to the data to be identified, including:

S501: Obtain weight parameters corresponding to the last layer of the convolutional layer in the lung fusion identification model.

Understandably, the weight parameters corresponding to the last layer of the convolutional layer in the lung fusion recognition model include the image weights and all of the image weights provided to the lung image feature vector corresponding to the last layer of the convolutional layer. Describe the text weights of the lung text feature vector.

S502: Determine voting parameters according to the obtained weight parameters.

Understandably, the obtained image weight and the text weight are kept unchanged, the image weight is used as the voting parameter of the image recognition result, and the text weight is used as the vote for the document recognition result. Voting parameter, a value of one is used as the voting parameter of the fusion identification result.

S503, according to the voting parameters, perform the voting on the image recognition result, the text recognition result and the fusion recognition result, and obtain the lung feature recognition result.

Understandably, according to the voting parameters of the image recognition results, the image recognition results, the voting parameters of the document recognition results, the text recognition results, the voting parameters of the fusion recognition results, and the In the fusion identification, the final probability distribution of each lung feature category is obtained through weighted average, and the lung feature category with the highest probability value is determined as the lung feature identification result.

In the present application, the weight parameters corresponding to the last layer of the convolutional layer in the lung fusion recognition model are obtained; voting parameters are determined according to the obtained weight parameters; according to the voting parameters, Carry out the voting on the image recognition result, the text recognition result and the fusion recognition result to obtain the lung feature recognition result. The above fusion recognition results are objectively voted, and the lung feature categories are finally identified, which improves the accuracy and reliability of lung feature recognition.

In one embodiment, a device for identifying lung features is provided, and the device for identifying lung features corresponds to the method for identifying lung features in the above embodiments. As shown in FIG. 7 , the lung feature identification device includes a receiving module 11 , an input module 12 , a first identification module 13 , a second identification module 14 and a voting module 15 . The detailed description of each functional module is as follows:

The receiving module 11 is configured to receive the identification request and obtain the data to be identified in the identification request; the data to be identified includes the lung image to be identified and the text description of the lung to be identified; the description of the lung text to be identified is A description of the lung features in the to-be-identified lung image;

The input module 12 is used to input the data to be recognized into the lung feature recognition model; the lung feature recognition model includes a lung image recognition model, a lung text recognition model and a lung fusion recognition model;

The first recognition module 13 is used for performing lung image feature extraction on the to-be-recognized lung image by the lung image recognition model, generating a lung image feature vector and an image recognition result, and simultaneously identifying the lung text through the lung image. The model performs lung text feature extraction on the description of the to-be-recognized lung text, and generates a lung text feature vector and a text recognition result;

The second recognition module 14 is configured to use the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract the image text fusion feature for recognition, and obtain a fusion recognition result ;

The voting module 15 is used for voting on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model, and obtains the lung feature recognition result corresponding to the data to be recognized ; The lung feature identification result indicates the lung feature category of the data to be identified.

For the specific limitation of the lung feature identification device, reference may be made to the definition of the lung feature identification method above, which will not be repeated here. Each module in the above-mentioned lung feature identification device may be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 8 . The computer device includes a processor, memory, a network interface and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium, an internal memory. The readable storage medium stores an operating system, computer readable instructions and a database. The internal memory provides an environment for the execution of the operating system and computer-readable instructions in the readable storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer-readable instructions, when executed by a processor, implement a lung feature identification method. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored on the memory and executable on the processor, and the processor implements the lungs in the above embodiments when the computer-readable instructions are executed. feature recognition method.

In one embodiment, one or more readable storage media storing computer-readable instructions are provided, and the readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media medium; computer-readable instructions are stored on the readable storage medium, and when the computer-readable instructions are executed by one or more processors, cause the one or more processors to implement the lung feature identification method in the foregoing embodiment.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile computer. In a readable storage medium or a volatile readable storage medium, the computer-readable instructions, when executed, may include the processes of the foregoing method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided in this application may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above-mentioned embodiments, those of ordinary skill in the art should understand that: it can still be used for the above-mentioned implementations. The technical solutions described in the examples are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the application, and should be included in the within the scope of protection of this application.

Claims (20)

1. A lung feature identification method, comprising:

   Acquire data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;

   Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result. The text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;

   Using the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract and identify the fused features to obtain a fusion recognition result;

   The image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
2. The lung feature recognition method according to claim 1, wherein the lung image feature extraction is performed on the to-be-recognized lung image by using the lung image recognition model to generate a lung image feature vector and an image recognition result ,include:

   The lung image to be identified is split into a red channel image, a green channel image and a blue channel image by the lung image recognition model, which is a network model constructed based on VGG19;

   The red channel image, the green channel image and the blue channel image are respectively extracted by convolution through the lung image recognition model to obtain a red feature vector corresponding to the red channel image, the green feature vector corresponding to the channel image and the blue feature vector corresponding to the blue channel image;

   Perform image recognition on the red feature vector, the green feature vector and the blue feature vector by using the lung image recognition model to obtain the image recognition result.
3. The lung feature identification method according to claim 1, wherein the lung text feature extraction is performed on the to-be-recognized lung text description by the lung text identification model to generate a lung text feature vector and text identification Results, including:

   The lung text description to be recognized is segmented by the lung text recognition model, and a text word vector corresponding to the lung text description to be recognized is constructed, and the lung text recognition model is a network constructed based on TextCNN Model;

   Perform channel expansion on the text word vector to generate a first text word vector, a second text word vector and a third text word vector;

   The first text word vector, the second text word vector and the third text word vector are respectively extracted by convolution through the lung text recognition model to obtain the first text word vector corresponding to the first text word vector. a text feature vector, a second text feature vector corresponding to the second text word vector, and a third text feature vector corresponding to the third text word vector;

   Perform text recognition on the first text feature vector, the second text feature vector and the third text feature vector by using the lung text recognition model to obtain the text recognition result.
4. The lung feature identification method according to claim 1, wherein the lung image feature vector and the lung text feature vector are fused by using an attention mechanism through the lung fusion identification model, and the fusion After extracting and identifying the features, the fusion identification results are obtained, including:

   Using the attention mechanism technology, the lung image feature vector and the lung text feature vector are weighted and fused through the weight parameters corresponding to each convolutional layer in the lung fusion recognition model, and the results are obtained with each volume. The fusion feature vector corresponding to the product layer;

   Extracting the image-text fusion feature on the fusion feature vector through the lung fusion recognition model;

   The fusion recognition result is obtained by performing recognition by the lung fusion recognition model according to the extracted image-text fusion features.
5. The lung feature identification method according to claim 4, wherein the lung image feature vector includes a red feature vector, a green feature vector and a blue feature vector; the lung text feature vector includes the first text feature vector, the second text feature vector and the third text feature vector;

   The lung image recognition model, the lung text recognition model and the lung fusion recognition model all have the same convolution level, and each of the three models is provided with a convolution layer corresponding to each convolution level. ;

   The described lung image feature vector and the described lung text feature vector are weighted and fused by the weight parameters corresponding to each convolutional layer in the described lung fusion recognition model to obtain a fusion corresponding to each convolutional layer. eigenvectors, including:

   The red feature vector corresponding to the same convolution level and the first text feature vector are fused according to the first weight parameter corresponding to the convolution level to obtain the first fusion feature vector;

   The green feature vector corresponding to the same convolution level and the second text feature vector are fused according to the second weight parameter corresponding to the convolution level to obtain the second fusion feature vector;

   The blue feature vector corresponding to the same convolution level and the third text feature vector are fused according to the third weight parameter corresponding to the convolution level to obtain the third fusion feature vector;

   The first fusion feature vector, the second fusion feature vector and the third fusion feature vector corresponding to the same convolution level are weighted and averaged to obtain the fusion feature vector.
6. The lung feature recognition method according to claim 4, wherein the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model, and a Describe the lung feature identification results corresponding to the data to be identified, including:

   Obtain the weight parameter corresponding to the last layer of the convolutional layer in the lung fusion recognition model;

   According to the obtained weight parameters, the voting parameters are determined;

   According to the voting parameters, the voting is performed on the image recognition result, the text recognition result and the fusion recognition result to obtain the lung feature recognition result.
7. The lung feature identification method according to claim 1, wherein before the inputting the data to be identified into the lung feature identification model, the method comprises:

   obtaining a lung sample set, the lung sample set including a plurality of lung samples, the lung samples including a lung image and a lung text description associated with the lung image, the lung samples being associated with a lung External feature category label association;

   Inputting the lung sample into a multimodal model containing initial parameters; the multimodal model includes a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model;

   The lung image feature extraction is performed on the lung image by the lung sample image recognition model to generate a lung sample image feature vector and an image sample recognition result. The lung text description performs the lung text feature extraction, and generates a lung sample text feature vector and a text sample recognition result;

   Using the attention mechanism to fuse the image feature vector of the lung sample and the text feature vector of the lung sample through the lung sample fusion recognition model, and learn to extract the image text fusion feature and recognition, to obtain a fusion sample recognition result;

   voting on the image sample recognition result, the text sample recognition result and the fusion sample recognition result to obtain a sample recognition result;

   Determine the loss value according to the sample identification result and the lung feature category label;

   When the loss value does not reach the preset convergence condition, the initial parameters of the multimodal model are iteratively updated, and when the loss value reaches the preset convergence condition, the multimodal model after convergence is updated. The model was recorded as a lung feature recognition model.
8. A lung feature identification device, comprising:

   A receiving module for acquiring data to be identified, wherein the data to be identified includes a lung image to be identified and a text description of the lung to be identified;

   an input module for inputting the data to be identified into a lung feature identification model, the lung feature identification model comprising a lung image identification model, a lung text identification model and a lung fusion identification model;

   The first recognition module is used for performing lung image feature extraction on the to-be-recognized lung image through the lung image recognition model, generating a lung image feature vector and an image recognition result, and simultaneously through the lung text recognition model performing lung text feature extraction on the description of the to-be-recognized lung text to generate a lung text feature vector and a text recognition result;

   The second recognition module is used to fuse the lung image feature vector and the lung text feature vector using the attention mechanism through the lung fusion recognition model, and extract and recognize the fused features to obtain a fusion recognition result;

   a voting module, configured to vote on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; The lung feature identification result indicates the lung feature category of the data to be identified.
9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer-readable instructions:

   Acquire data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;

   Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result. The text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;

   Using the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract and identify the fused features to obtain a fusion recognition result;

   The image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
10. The computer device according to claim 9, wherein the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and an image recognition result, comprising:

    The lung image to be identified is split into a red channel image, a green channel image and a blue channel image by the lung image recognition model, which is a network model constructed based on VGG19;

    The red channel image, the green channel image and the blue channel image are respectively extracted by convolution through the lung image recognition model to obtain a red feature vector corresponding to the red channel image, the green feature vector corresponding to the channel image and the blue feature vector corresponding to the blue channel image;

    Perform image recognition on the red feature vector, the green feature vector and the blue feature vector by using the lung image recognition model to obtain the image recognition result.
11. The computer device according to claim 9, wherein the lung text feature extraction is performed on the to-be-recognized lung text description by the lung text recognition model to generate a lung text feature vector and a text recognition result, comprising: :

    The lung text description to be recognized is segmented by the lung text recognition model, and a text word vector corresponding to the lung text description to be recognized is constructed, and the lung text recognition model is a network constructed based on TextCNN Model;

    Perform channel expansion on the text word vector to generate a first text word vector, a second text word vector and a third text word vector;

    The first text word vector, the second text word vector and the third text word vector are respectively extracted by convolution through the lung text recognition model to obtain the first text word vector corresponding to the first text word vector. a text feature vector, a second text feature vector corresponding to the second text word vector, and a third text feature vector corresponding to the third text word vector;

    Perform text recognition on the first text feature vector, the second text feature vector and the third text feature vector by using the lung text recognition model to obtain the text recognition result.
12. The computer device according to claim 9, wherein the lung image feature vector and the lung text feature vector are fused by the lung fusion recognition model using an attention mechanism, and the fused feature Extract and identify to obtain fusion identification results, including:

    Using the attention mechanism technology, the lung image feature vector and the lung text feature vector are weighted and fused through the weight parameters corresponding to each convolutional layer in the lung fusion recognition model, and the results are obtained with each volume. The fusion feature vector corresponding to the product layer;

    Extracting the image-text fusion feature on the fusion feature vector through the lung fusion recognition model;

    The fusion recognition result is obtained by performing recognition by the lung fusion recognition model according to the extracted image-text fusion features.
13. The computer device of claim 11, wherein the lung image feature vector includes a red feature vector, a green feature vector and a blue feature vector; the lung text feature vector includes a first text feature vector, a second text feature vector feature vector and third text feature vector;

    The lung image recognition model, the lung text recognition model and the lung fusion recognition model all have the same convolution level, and each of the three models is provided with a convolution layer corresponding to each convolution level. ;

    The described lung image feature vector and the described lung text feature vector are weighted and fused by the weight parameters corresponding to each convolutional layer in the described lung fusion recognition model to obtain a fusion corresponding to each convolutional layer. eigenvectors, including:

    The red feature vector corresponding to the same convolution level and the first text feature vector are fused according to the first weight parameter corresponding to the convolution level to obtain the first fusion feature vector;

    The green feature vector corresponding to the same convolution level and the second text feature vector are fused according to the second weight parameter corresponding to the convolution level to obtain the second fusion feature vector;

    The blue feature vector corresponding to the same convolution level and the third text feature vector are fused according to the third weight parameter corresponding to the convolution level to obtain the third fusion feature vector;

    The first fusion feature vector, the second fusion feature vector and the third fusion feature vector corresponding to the same convolution level are weighted and averaged to obtain the fusion feature vector.
14. The computer device according to claim 11, wherein the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model, and a The lung feature identification results corresponding to the data, including:

    Obtain the weight parameter corresponding to the last layer of the convolutional layer in the lung fusion recognition model;

    According to the obtained weight parameters, the voting parameters are determined;

    According to the voting parameters, the voting is performed on the image recognition result, the text recognition result and the fusion recognition result to obtain the lung feature recognition result.
15. One or more readable storage media storing computer-readable instructions, wherein the computer-readable instructions, when executed by one or more processors, cause the one or more processors to perform the following steps:

    Acquire data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;

    Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result. The text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;

    Using the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract and identify the fused features to obtain a fusion recognition result;

    The image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
16. The readable storage medium according to claim 15, wherein the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and an image recognition result, include:

    The lung image to be identified is split into a red channel image, a green channel image and a blue channel image by the lung image recognition model, which is a network model constructed based on VGG19;

    The red channel image, the green channel image and the blue channel image are respectively extracted by convolution through the lung image recognition model to obtain a red feature vector corresponding to the red channel image, the green feature vector corresponding to the channel image and the blue feature vector corresponding to the blue channel image;

    Perform image recognition on the red feature vector, the green feature vector and the blue feature vector by using the lung image recognition model to obtain the image recognition result.
17. The readable storage medium according to claim 15, wherein the lung text feature extraction is performed on the to-be-recognized lung text description by the lung text recognition model to generate a lung text feature vector and a text recognition result ,include:

    The lung text description to be recognized is segmented by the lung text recognition model, and a text word vector corresponding to the lung text description to be recognized is constructed, and the lung text recognition model is a network constructed based on TextCNN Model;

    Perform channel expansion on the text word vector to generate a first text word vector, a second text word vector and a third text word vector;

    The first text word vector, the second text word vector and the third text word vector are respectively extracted by convolution through the lung text recognition model to obtain the first text word vector corresponding to the first text word vector. a text feature vector, a second text feature vector corresponding to the second text word vector, and a third text feature vector corresponding to the third text word vector;

    Perform text recognition on the first text feature vector, the second text feature vector and the third text feature vector by using the lung text recognition model to obtain the text recognition result.
18. The readable storage medium of claim 15, wherein the lung image feature vector and the lung text feature vector are fused by using an attention mechanism through the lung fusion recognition model, and the fusion is performed on the fused lung image feature vector and the lung text feature vector. Extract and identify the features of , and get the fusion recognition results, including:

    Using the attention mechanism technology, the lung image feature vector and the lung text feature vector are weighted and fused through the weight parameters corresponding to each convolutional layer in the lung fusion recognition model, and the results are obtained with each volume. The fusion feature vector corresponding to the product layer;

    Extracting the image-text fusion feature on the fusion feature vector through the lung fusion recognition model;

    The fusion recognition result is obtained by performing recognition by the lung fusion recognition model according to the extracted image-text fusion features.
19. The readable storage medium of claim 17, wherein the lung image feature vector includes a red feature vector, a green feature vector, and a blue feature vector; the lung text feature vector includes a first text feature vector, a The second text feature vector and the third text feature vector;

    The lung image recognition model, the lung text recognition model and the lung fusion recognition model all have the same convolution level, and each of the three models is provided with a convolution layer corresponding to each convolution level. ;

    The described lung image feature vector and the described lung text feature vector are weighted and fused by the weight parameters corresponding to each convolutional layer in the described lung fusion recognition model to obtain a fusion corresponding to each convolutional layer. eigenvectors, including:

    The red feature vector corresponding to the same convolution level and the first text feature vector are fused according to the first weight parameter corresponding to the convolution level to obtain the first fusion feature vector;

    The green feature vector corresponding to the same convolution level and the second text feature vector are fused according to the second weight parameter corresponding to the convolution level, and the second fusion feature vector is obtained;

    The blue feature vector corresponding to the same convolution level and the third text feature vector are fused according to the third weight parameter corresponding to the convolution level to obtain the third fusion feature vector;

    The first fusion feature vector, the second fusion feature vector and the third fusion feature vector corresponding to the same convolution level are weighted and averaged to obtain the fusion feature vector.
20. The readable storage medium according to claim 17, wherein the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain a The lung feature identification results corresponding to the data to be identified, including:

    Obtain the weight parameter corresponding to the last layer of the convolutional layer in the lung fusion recognition model;

    According to the obtained weight parameters, the voting parameters are determined;

    According to the voting parameters, the voting is performed on the image recognition result, the text recognition result and the fusion recognition result to obtain the lung feature recognition result.

Images