WO2022057309A1 - Procédé et appareil de reconnaissance de caractéristique pulmonaire, dispositif informatique et support de stockage - Google Patents

Procédé et appareil de reconnaissance de caractéristique pulmonaire, dispositif informatique et support de stockage Download PDF

Info

Publication number
WO2022057309A1
WO2022057309A1 PCT/CN2021/096366 CN2021096366W WO2022057309A1 WO 2022057309 A1 WO2022057309 A1 WO 2022057309A1 CN 2021096366 W CN2021096366 W CN 2021096366W WO 2022057309 A1 WO2022057309 A1 WO 2022057309A1
Authority
WO
WIPO (PCT)
Prior art keywords
lung
text
image
feature vector
feature
Prior art date
Application number
PCT/CN2021/096366
Other languages
English (en)
Chinese (zh)
Inventor
朱昭苇
孙行智
胡岗
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2022057309A1 publication Critical patent/WO2022057309A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/259—Fusion by voting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30061—Lung
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Definitions

  • the present application relates to the field of image classification of artificial intelligence, and in particular, to a method, device, computer equipment and storage medium for identifying lung features.
  • the identification of lung features mainly relies on medical personnel to manually identify lung image information based on their own experience. Because the movement of lung tissue is uneven and complex, the identification process not only costs medical personnel At the same time, in the process of identification, only the lung image information is often identified, and the main complaint information (text description for the lung image information) of the lung image information is not combined for identification. It is easy to lose the information of lung tissue movement, resulting in low accuracy and low efficiency.
  • the present application provides a lung feature identification method, device, computer equipment and storage medium, which realizes a lung feature identification model including a lung image identification model, a lung text identification model, and a lung fusion identification model, and uses attention It realizes the automatic, rapid and accurate identification of lung features, improves the accuracy and reliability of identification, and improves the efficiency of identification.
  • This application is applicable to fields such as smart medical care, and can further promote the construction of smart cities.
  • a lung feature recognition method comprising:
  • Acquiring data to be identified wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified;
  • the lung feature identification model includes a lung image identification model, a lung text identification model and a lung fusion identification model;
  • Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result.
  • the text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;
  • the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
  • a lung feature identification device comprising:
  • a receiving module for acquiring data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified;
  • an input module for inputting the data to be identified into a lung feature identification model, the lung feature identification model comprising a lung image identification model, a lung text identification model and a lung fusion identification model;
  • the first recognition module is used for performing lung image feature extraction on the to-be-recognized lung image through the lung image recognition model, generating a lung image feature vector and an image recognition result, and simultaneously through the lung text recognition model performing lung text feature extraction on the description of the to-be-recognized lung text to generate a lung text feature vector and a text recognition result;
  • the second recognition module is used to fuse the lung image feature vector and the lung text feature vector using the attention mechanism through the lung fusion recognition model, and extract and recognize the fused features to obtain a fusion recognition result;
  • a voting module configured to vote on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized;
  • the lung feature identification result indicates the lung feature category of the data to be identified.
  • a computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, the processor implementing the following steps when executing the computer-readable instructions:
  • Acquire data to be identified wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;
  • Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result.
  • the text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;
  • the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
  • One or more readable storage media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps:
  • Acquire data to be identified wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;
  • Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result.
  • the text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;
  • the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
  • the lung feature identification method, device, computer equipment and storage medium obtain the data to be identified; the data to be identified includes the image of the lung to be identified and the text description of the lung to be identified; the data to be identified is input to a lung feature recognition model including a lung image recognition model, a lung text recognition model and a lung fusion recognition model; the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, Generate lung image feature vectors and image recognition results, and perform lung text feature extraction on the description of the to-be-recognized lung text through the lung text recognition model to generate lung text feature vectors and text recognition results; The lung fusion recognition model uses the attention mechanism to fuse the lung image feature vector and the lung text feature vector, and extracts and recognizes the fused features to obtain a fusion recognition result; through the lung feature recognition model The image recognition result, the text recognition result and the fusion recognition result are voted, and the lung feature recognition result corresponding to the data to be recognized is obtained.
  • the recognition to be recognized by the lung image recognition model is realized.
  • Lung image get the image recognition result, identify the text description of the lung to be recognized through the lung text recognition model, get the text recognition result, and then combine the lung image to be recognized and the text description to be recognized, use the attention mechanism, through the lung fusion
  • the recognition model extracts the image and text fusion features for recognition, and obtains the fusion recognition result.
  • the text recognition result and the fusion recognition result voting is carried out, and the lung feature recognition result is obtained. Recognize lung images and text descriptions of the lungs to be recognized, and automatically, quickly and accurately identify lung features through the multimodal model-based lung feature recognition model, improve the recognition accuracy and reliability, and improve the recognition effectiveness.
  • FIG. 1 is a schematic diagram of an application environment of a lung feature identification method in an embodiment of the present application
  • FIG. 2 is a flowchart of a lung feature identification method in an embodiment of the present application.
  • step S30 of the lung feature identification method in an embodiment of the present application is a flowchart of step S30 of the lung feature identification method in an embodiment of the present application
  • step S30 of the lung feature identification method in another embodiment of the present application is a flowchart of step S30 of the lung feature identification method in another embodiment of the present application.
  • step S40 of the lung feature identification method in an embodiment of the present application
  • step S50 of the lung feature identification method in an embodiment of the present application is a flowchart of step S50 of the lung feature identification method in an embodiment of the present application.
  • FIG. 7 is a schematic block diagram of a lung feature identification device in an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a computer device in an embodiment of the present application.
  • the lung feature identification method provided by the present application can be applied in the application environment as shown in FIG. 1 , wherein the client (computer device) communicates with the server through the network.
  • the client computer equipment
  • the server includes but is not limited to various personal computers, notebook computers, smart phones, tablet computers, cameras and portable wearable devices.
  • the server can be implemented as an independent server or a server cluster composed of multiple servers.
  • a method for identifying lung features is provided, and its technical solution mainly includes the following steps S10-S50:
  • S10 Acquire data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified.
  • the lung image to be identified is an image collected by a lung imaging device, and the lung imaging device can be selected according to requirements, for example, the lung imaging device is a CT device, an X-ray machine, or a three-dimensional projection device, etc. etc.
  • the lung text description is a description of the lung features in the to-be-recognized lung image, that is, the lung text is described as the main complaint information for the to-be-recognized lung image
  • the lung features The features reflected by the movement of lung tissue, such as lung features including pleural concave features, air bronchus features, lung vacuole features, lung spur features, lung ground glass-like features, etc., after collecting the to-be-identified lung image, And after inputting the text description of the lung to be recognized for the image of the lung to be recognized, the image of the lung to be recognized and the text description of the lung to be recognized are determined as the data to be recognized, and a recognition request is triggered.
  • the identification request is a request for
  • the lung feature recognition model includes a lung image recognition model, a lung text recognition model, and a lung fusion recognition model.
  • the lung feature recognition model is a multimodal model that has been trained, and the lung feature recognition model can recognize the lung features of the data to be identified, and the lung feature recognition model includes lungs. part image recognition model, lung text recognition model and lung fusion recognition model, the lung image recognition model is to obtain the image recognition result by extracting the lung image features in the lung image to be recognized, and performing image recognition, And generate a lung image feature vector for the lung fusion recognition model, the lung image feature is the feature of the image space embodied by the movement of the lung tissue, and the network structure of the lung image recognition model can be based on the needs of image recognition.
  • the network structure of the lung image recognition model is VGG16, VGG19, GoogleNet or ResNet, etc.
  • the network structure of the lung image recognition model is the network structure of VGG19; the lung text recognition model is By extracting the lung text features in the text description of the lungs to be identified, and performing text recognition, the text recognition results are obtained, and a lung text feature vector for the lung fusion recognition model is generated, where the lung text features are lungs
  • the characteristics of the text space reflected by the movement of the external tissue, the network structure of the lung text recognition model can be set according to the needs of language recognition, for example, the network structure of the lung text recognition model is TextCNN, LSTM or BERT, etc., as a preference,
  • the network structure of the described lung text recognition model selects the network structure of TextCNN; the described lung fusion recognition model is to use the attention mechanism to fuse the described lung image feature vector and the described lung text feature vector, and extract the fused
  • the image-text fusion feature in the lung image feature vector and the lung text feature vector, and the fusion recognition result is identified, and the image-text fusion feature is the lung image feature vector and the lung
  • the method before the step S20, that is, before the input of the data to be identified into the lung feature identification model, the method includes:
  • Obtain a lung sample set where the lung sample set includes a plurality of lung samples, the lung samples include a lung image and a lung text description associated with the lung image, and the lung samples are associated with the lung image.
  • a lung feature class label association
  • the lung sample set is a collection of the lung samples
  • the historical collection of the lung samples includes a lung image and a lung text description associated with the lung image.
  • the lung sample is associated with a lung feature class label
  • the lung feature class label is a label related to the lung feature class marked on the lung sample
  • the lung image is a historically collected through the lung
  • the image picture of the lung collected by the photographing device is a description of the lung feature in the lung image associated therewith
  • the lung feature category is a classification of the lung feature
  • the lung feature categories include a pleural indentation feature class corresponding to a pleural indentation feature, an air bronchus feature class corresponding to an air bronchus feature, a lung vacuolar feature class corresponding to a pulmonary vacuole feature, and a lung burr feature corresponding to a pleural indentation feature class.
  • Lung spur feature class and lung ground glass features corresponding to lung ground glass features.
  • the multimodal model includes a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model.
  • the multimodal model is to match the similarity between images and texts, that is, to measure the similarity between an image and a piece of text (the global similarity between the image and the text), and identify the implicit relationship between the image and the text. The characteristics of the relationship are determined, and the classification result of the fusion of an image and a text is determined.
  • the multimodal model includes the initial parameters, and the initial parameters include the lung sample image recognition model and the lung sample text recognition model.
  • the parameters of the model and the lung sample fusion recognition model can be transferred directly from the parameters in the multimodal recognition models in other fields to the initial parameters in the multimodal model by means of transfer learning, simplifying
  • the training process shortens the training time and improves the training efficiency.
  • the multimodal model includes a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model.
  • the lung image recognition model The lung sample image recognition model that has been trained, the lung text recognition model is the lung sample text recognition model that has been trained, and the lung fusion recognition model is the lung sample fusion that has been trained. Identify the model.
  • the lung image feature is the feature of the image space embodied by the movement of lung tissue
  • the lung sample image feature vector is a vector matrix with the lung image feature
  • the image sample recognition result is all
  • the lung sample image recognition model identifies the results of the lung features in the lung image by the similarity of the image space based on the extracted lung image features
  • the lung sample text feature vector is a feature vector with the lung image.
  • a vector matrix of image features, and the text sample recognition result is that the lung sample text recognition model performs text space similarity according to the extracted lung text features to identify the lung features in the lung text description. result.
  • the image feature vector of the lung sample and the text feature vector of the lung sample are fused through the attention mechanism, and the learning to extract the image-text fusion feature is to capture the implicit similarity between the image and the text. Feature extraction, and local similarity measurement and extraction.
  • sample identification result and the lung feature category label are input into the loss function of the multimodal model, and the loss value is calculated through the loss function.
  • the convergence condition may be a condition that the loss value is small and will not decrease after 6,000 calculations, that is, the loss value is small and will not decrease after 6,000 calculations.
  • the convergence condition can also be the condition that the loss value is less than the set threshold, that is, when the loss value is less than the set threshold.
  • the training is stopped, and the multimodal model after convergence is recorded as a lung feature recognition model. In this way, when the loss value does not reach the preset convergence condition, the multimodal model is continuously adjusted.
  • the initial parameters in the model are triggered, and the lung image feature extraction is performed on the lung image through the lung sample image recognition model to generate a lung sample image feature vector and an image sample recognition result.
  • Part of the sample text recognition model extracts the lung text features from the lung text description, and generates the lung sample text feature vector and the text sample recognition results, which can continuously move closer to the accurate results, so that the accuracy of the recognition is higher. Come higher. In this way, the lung feature recognition of the multimodal model can be optimized, and the accuracy and reliability of the lung feature recognition are improved.
  • the lung image recognition model performs channel splitting and convolution on the to-be-recognized lung image, thereby extracting the lung image features
  • the lung image features are images embodied by the movement of lung tissue.
  • the lung image recognition model includes a plurality of convolutional layers
  • the convolutional layers of the lung image recognition model can be marked as image convolutional layers, and through each of the lung image recognition model
  • the image convolution layer convolves the to-be-identified lung image according to different convolution kernels, and generates the lung image feature vector corresponding to each image convolution layer, and the lung image feature vector has the lung image
  • the dimension of each described lung image feature vector is different according to the difference of each image convolution layer
  • the image recognition result is the lung image feature extracted by the lung image recognition model according to the
  • the lung text recognition model performs word vector conversion on the description of the lung text to be identified, and then performs convolution to extract the lung text features.
  • the lung text feature is the feature of the text space embodied by the movement of lung tissue
  • the lung text recognition model includes a plurality of convolution layers, and the convolution layer of the lung text recognition model can be marked as text convolution Layer, through each text convolution layer in the lung image recognition model, the description of the lung text to be recognized is convolved according to different convolution kernels, and the lung text features corresponding to each text convolution layer are generated.
  • vector the lung text feature vector is a vector matrix with the lung text feature, the dimension of each described lung text feature vector is different according to the difference of each text convolution layer, and the text recognition result is the
  • the lung text recognition model identifies the result of the lung features by performing text space similarity based on the extracted lung text features.
  • the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, and a lung image feature vector is generated. and image recognition results, including:
  • the lung image recognition model is a network model constructed based on VGG19.
  • the lung image to be identified is an image of three channels: red channel, green channel, and blue channel, that is, the lung image to be identified includes the red channel image corresponding to the red channel, and the green channel image corresponding to the red channel.
  • the green channel image corresponding to the channel and the blue channel image corresponding to the blue channel, through channel splitting, the to-be-identified lung image is split into the red channel image, the green channel image and the
  • the blue channel image, the red channel image is an image that reflects the degree of redness of each pixel through pixel values in the range of 0 to 255
  • the green channel image is an image that reflects the redness of each pixel through pixel values in the range of 0 to 255.
  • An image with a green level, and the blue channel image is an image in which the blue level of each pixel is represented by pixel values ranging from 0 to 255.
  • the lung image recognition model is a network model constructed based on VGG19, and the convolution depth of the lung image can be set to 19, that is, a network model with 19-level convolution layers.
  • the red channel image is convolved by the lung image recognition model to obtain the red feature vector
  • the red feature vector is the vector embodied in the red space extracted from the lung image features
  • the lung image recognition model convolves the green channel image to obtain the green feature vector.
  • the recognition model convolves the blue channel image to obtain the blue feature vector, where the blue feature vector is the vector embodied in the blue space extracted from the lung image features, and the red feature vector,
  • the green feature vector and the blue feature vector are determined as the lung image feature vector.
  • S303 Perform image recognition on the lung image feature vector by using the lung image recognition model to obtain the image recognition result.
  • image recognition is performed on the lung image feature vector by the lung image recognition model, and the image recognition is to perform fully connected classification according to the extracted lung image feature vector to obtain each lung feature category.
  • the probability distribution of so as to output the recognized image recognition result.
  • the present application realizes that the lung image to be recognized is split into a red channel image, a green channel image and a blue channel image through the lung image recognition model; the lung image recognition model is a network model constructed based on VGG19 ; Carry out convolution extraction on the red channel image, the green channel image and the blue channel image respectively by the lung image recognition model to obtain the lung image feature vector; by the lung image recognition model to The lung image feature vector is used for image recognition to obtain the image recognition result.
  • the constructed network model performs convolution on each channel image to extract the lung image features, obtains the lung image feature vector, and outputs the image recognition result according to the lung image feature vector, which can extract the lung image in the lung image to be identified.
  • the lung feature categories are identified through the extracted lung image features, which provides a data basis for subsequent identification and improves the accuracy and reliability of identification.
  • the lung text feature extraction is performed on the description of the lung text to be identified by the lung text recognition model, and the lung text feature is generated.
  • Vector and text recognition results including:
  • the word segmentation is to use a word dictionary to split the description of the lung text to be identified into individual words, and the word dictionary contains word vectors corresponding to all medical terms and words related to the lungs. , and then convert the split words into their corresponding word vectors, which can be converted by the conversion method of word2vec or Glove, and then splicing the converted word vectors to form the text word vector.
  • the pulmonary text recognition model is a network model constructed based on TextCNN, that is, the pulmonary text recognition model has the network structure of TextCNN, and the convolution depth of the pulmonary text recognition model is set to 19, that is, it has 19
  • the network model of the hierarchical convolution layer, the convolution depth of the lung text recognition model is the same as the convolution depth of the lung image recognition model, so as to facilitate the recognition of the subsequent lung fusion recognition model.
  • the channel expansion is a process of expanding the text word vector of a single channel to a vector matrix of a preset dimension and copying the vector matrix until the number of preset channels, that is, expanding the text word vector to the same size as the text word vector.
  • the vector matrix of the same dimension of the lung image feature vector, the expansion method can be set according to the needs, and the vector matrix is copied into a vector matrix with the same number of channels as the lung image feature vector, so as to obtain the same vector matrix as the vector matrix. Describe the first text word vector, the second text word vector and the third text word vector.
  • the first text word vector is convolved by the lung text recognition model to obtain the first text feature vector
  • the second text word vector is processed by the lung text recognition model.
  • the convolution kernel for convolving the vector, the convolution kernel for convolving the second text word vector, and the convolution kernel for convolving the third text word vector may be different, that is, from different text spaces.
  • the dimension of lung text feature vector is extracted, and the first text word vector, the second text word vector and the third text word vector are determined as the lung text feature vector.
  • text recognition is performed on the lung text feature vector by the lung text recognition model, and the text recognition is to perform a fully connected classification according to the extracted lung text feature vector to obtain each lung feature category.
  • the probability distribution of so as to output the recognized text recognition result.
  • the present application implements word segmentation for the description of the lung text to be identified through the lung text recognition model, and constructs a text word vector corresponding to the description of the lung text to be identified; the lung text recognition model is based on A network model constructed by TextCNN; channel expansion of the text word vector to generate a first text word vector, a second text word vector and a third text word vector;
  • the word vector, the second text word vector, and the third text word vector are extracted by convolution to obtain a lung text feature vector, and text recognition is performed on the lung text feature vector by the lung text recognition model, Obtaining the text recognition result, in this way, it is realized that the first text word vector, the second text word vector and the third text word vector are generated by segmenting the description of the lung text to be identified and
  • the lung feature category is identified through the extracted lung text features, which provides a data basis for subsequent identification and improves the accuracy and reliability of identification.
  • the attention mechanism is a mechanism learned by an additional feedforward neural network in neural network learning and recognition through the attention weight, and the lung image feature vector and the lung image feature vector and The implicit relationship between the lung text feature vectors, that is, the lung image feature vector and the lung text feature vector are carried out according to the weight parameters corresponding to each convolution layer learned through the attention mechanism.
  • Weighted fusion so as to obtain the fusion feature vector corresponding to each convolution layer, convolve all the fusion feature vectors, and extract the image-text fusion features, that is, extract the fused features, the image
  • the text fusion feature is an implicit feature associated between the lung image feature vector and the lung text feature vector, that is, the global similarity between the lung image feature vector and the lung text feature vector.
  • the feature is identified according to the extracted image-text fusion features, that is, the probability distribution of each lung feature category is classified by full connection, so as to output the fusion recognition result.
  • the lung image feature vector and the lung text feature vector are fused by using the attention mechanism through the lung fusion recognition model. , and extract image-text fusion features for recognition, and obtain fusion recognition results, including:
  • the attention mechanism technology is to enhance the useful information in the feature vector, that is, the weight parameters corresponding to each convolutional layer according to the useful vector of the lung image feature vector and the lung text feature vector.
  • a weighted average is performed and fused to generate a fused feature vector corresponding to each convolutional layer.
  • the convolution depth in the lung fusion recognition model is the same as the convolution depth of the lung image recognition model or the lung text recognition model, and the convolution depth in the lung fusion recognition model is preferably 19 levels.
  • the lung image feature vector includes a red feature vector, a green feature vector and a blue feature vector;
  • the lung text feature vector includes a first text feature vector, a second text feature vector feature vector and third text feature vector;
  • the lung image recognition model, the lung text recognition model and the lung fusion recognition model all have the same convolution level, and each of the three models is provided with a convolution layer corresponding to each convolution level. ;
  • the described lung image feature vector and the described lung text feature vector are weighted and fused by the weight parameters corresponding to each convolutional layer in the described lung fusion recognition model to obtain a fusion corresponding to each convolutional layer.
  • eigenvectors including:
  • the red feature vector and the first text feature vector corresponding to the same convolution level are weighted according to the first weight parameter of the convolution level, that is, the red feature vector and the first text feature
  • Each vector value in the vector is weighted and averaged according to the first weight parameter to obtain the first fusion feature vector, and the red feature vector, the first text feature vector and the first fusion feature vector have the same dimensions .
  • the green feature vector and the second text feature vector corresponding to the same convolution level are weighted according to the second weight parameter of the convolution level, that is, the green feature vector and the second text feature
  • the green feature vector and the second text feature vector are weighted according to the second weight parameter of the convolution level, that is, the green feature vector and the second text feature
  • Each vector value in the vector is weighted and averaged according to the second weight parameter to obtain the second fusion feature vector, and the green feature vector, the second text feature vector and the second fusion feature vector have the same dimensions .
  • the blue feature vector and the third text feature vector corresponding to the same convolution level are weighted according to the third weight parameter of the convolution level, that is, the blue feature vector and the third
  • the third weight parameter of the convolution level that is, the blue feature vector and the third
  • Each vector value in the text feature vector is weighted and averaged according to the third weight parameter to obtain the third fusion feature vector, the blue feature vector, the third text feature vector and the third fusion feature vector dimensions are the same.
  • the execution order of the steps S4011, S4012 and S4013 is not limited, and may be executed in series or in parallel, and the first weight parameter, the second weight parameter and the third weight parameter may be the same, or can all be different.
  • the weighted average is to average the first fusion feature vector, the second fusion feature vector and the third fusion feature vector after weighting, and the The first fusion feature vector, the second fusion feature vector and the third fusion feature vector are weighted and averaged to obtain the fusion feature vector corresponding to each convolution layer.
  • the extraction process of the image-text fusion feature may be to convolve the fusion feature vector of the convolutional layer of the first layer, and then perform the convolution with the fusion feature vector of the convolutional layer of the next layer of the convolutional layer.
  • the transfer feature vector is obtained by superposition, and then the transfer feature vector is convolved, and the transfer feature vector is continuously superimposed with the fusion feature vector of the convolution layer of the next layer to obtain the transfer feature vector, and the superimposed transfer feature vector is convolved until a one-dimensional The feature vector extraction process of .
  • the identification is performed according to the extracted image-text fusion feature by the lung fusion recognition model, and the identification is to obtain the probability distribution of each lung feature category according to the extracted image-text fusion feature, thereby outputting The identified fusion identification result.
  • the present application realizes the weighted fusion of the lung image feature vector and the lung text feature vector by using the attention mechanism technology and the weight parameters corresponding to each convolution layer in the lung fusion recognition model. , obtain the fusion feature vector corresponding to each convolution layer; perform the image-text fusion feature extraction on the fusion feature vector by the lung fusion recognition model; The image and text fusion features are used for recognition, and the fusion recognition result is obtained.
  • the attention mechanism can be used to enhance the useful information in the image and the text, and the global similarity between the image and the text can be better captured.
  • the lung image feature vector and the lung text feature vector are weighted and fused, and the image and text fusion features are extracted for identification, which can improve the accuracy and reliability of lung feature identification.
  • the voting is to perform a weighted average of the probability values corresponding to the same lung feature category in the image recognition result, the text recognition result and the fusion recognition result, and finally determine that the probability value is the highest. and the lung feature category with the highest probability value is used as the lung feature identification result, and the lung feature identification result includes the identified lung feature category and the probability value corresponding to the category, so
  • the lung feature recognition result indicates the lung feature category of the data to be identified, and the lung feature is the feature embodied by the movement of lung tissue.
  • the lung feature category is a classification of the lung feature, for example, the lung feature category includes the pleural depression feature class corresponding to the pleural depression feature, The air bronchus feature class corresponding to the air bronchus feature, the lung vacuole feature class corresponding to the lung vacuole feature, the lung spur feature class corresponding to the lung spur feature, and the lung ground glass feature corresponding to the lung ground glass feature.
  • the present application realizes by acquiring the data to be recognized in the recognition request; the data to be recognized includes the lung image to be recognized and the text description of the lung to be recognized; the data to be recognized is input into a recognition model containing a lung image , the lung feature recognition model of the lung text recognition model and the lung fusion recognition model; the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, and the lung image feature vector and the image are generated.
  • Recognition results while performing lung text feature extraction on the description of the to-be-recognized lung text by the lung text recognition model, to generate a lung text feature vector and a text recognition result; using the attention through the lung fusion recognition model
  • the mechanism fuses the lung image feature vector and the lung text feature vector, and extracts image-text fusion features for recognition, and obtains a fusion recognition result; through the lung feature recognition model, the image recognition result, the text
  • the identification results and the fusion identification results are voted for, and the lung feature identification results corresponding to the data to be identified are obtained. In this way, the identification of the lung images to be identified through the lung image identification model is realized, and the image identification results are obtained.
  • the lung text recognition model recognizes the text description of the lungs to be recognized, and obtains the text recognition result, and then combines the lung image to be recognized and the text description to be recognized, and uses the attention mechanism to extract the image and text fusion features through the lung fusion recognition model for recognition. Obtain the fusion recognition result, and finally vote according to the image recognition result, the text recognition result and the fusion recognition result, and obtain the lung feature recognition result, which realizes the combination of the lung image to be recognized and the lung text to be recognized. Described, through the lung feature recognition model based on the multimodal model, the lung features are automatically, quickly and accurately identified, the recognition accuracy and reliability are improved, and the recognition efficiency is improved.
  • step S50 the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model. , to obtain the lung feature identification results corresponding to the data to be identified, including:
  • the weight parameters corresponding to the last layer of the convolutional layer in the lung fusion recognition model include the image weights and all of the image weights provided to the lung image feature vector corresponding to the last layer of the convolutional layer. Describe the text weights of the lung text feature vector.
  • the obtained image weight and the text weight are kept unchanged, the image weight is used as the voting parameter of the image recognition result, and the text weight is used as the vote for the document recognition result.
  • Voting parameter a value of one is used as the voting parameter of the fusion identification result.
  • the final probability distribution of each lung feature category is obtained through weighted average, and the lung feature category with the highest probability value is determined as the lung feature identification result.
  • the weight parameters corresponding to the last layer of the convolutional layer in the lung fusion recognition model are obtained; voting parameters are determined according to the obtained weight parameters; according to the voting parameters, Carry out the voting on the image recognition result, the text recognition result and the fusion recognition result to obtain the lung feature recognition result.
  • the above fusion recognition results are objectively voted, and the lung feature categories are finally identified, which improves the accuracy and reliability of lung feature recognition.
  • a device for identifying lung features is provided, and the device for identifying lung features corresponds to the method for identifying lung features in the above embodiments.
  • the lung feature identification device includes a receiving module 11 , an input module 12 , a first identification module 13 , a second identification module 14 and a voting module 15 .
  • the detailed description of each functional module is as follows:
  • the receiving module 11 is configured to receive the identification request and obtain the data to be identified in the identification request; the data to be identified includes the lung image to be identified and the text description of the lung to be identified; the description of the lung text to be identified is A description of the lung features in the to-be-identified lung image;
  • the input module 12 is used to input the data to be recognized into the lung feature recognition model;
  • the lung feature recognition model includes a lung image recognition model, a lung text recognition model and a lung fusion recognition model;
  • the first recognition module 13 is used for performing lung image feature extraction on the to-be-recognized lung image by the lung image recognition model, generating a lung image feature vector and an image recognition result, and simultaneously identifying the lung text through the lung image.
  • the model performs lung text feature extraction on the description of the to-be-recognized lung text, and generates a lung text feature vector and a text recognition result;
  • the second recognition module 14 is configured to use the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract the image text fusion feature for recognition, and obtain a fusion recognition result ;
  • the voting module 15 is used for voting on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model, and obtains the lung feature recognition result corresponding to the data to be recognized ;
  • the lung feature identification result indicates the lung feature category of the data to be identified.
  • Each module in the above-mentioned lung feature identification device may be implemented in whole or in part by software, hardware and combinations thereof.
  • the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
  • a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 8 .
  • the computer device includes a processor, memory, a network interface and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities.
  • the memory of the computer device includes a readable storage medium, an internal memory.
  • the readable storage medium stores an operating system, computer readable instructions and a database.
  • the internal memory provides an environment for the execution of the operating system and computer-readable instructions in the readable storage medium.
  • the network interface of the computer device is used to communicate with an external terminal through a network connection.
  • the computer-readable instructions when executed by a processor, implement a lung feature identification method.
  • the readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.
  • a computer device including a memory, a processor, and computer-readable instructions stored on the memory and executable on the processor, and the processor implements the lungs in the above embodiments when the computer-readable instructions are executed. feature recognition method.
  • one or more readable storage media storing computer-readable instructions are provided, and the readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media medium; computer-readable instructions are stored on the readable storage medium, and when the computer-readable instructions are executed by one or more processors, cause the one or more processors to implement the lung feature identification method in the foregoing embodiment.
  • Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory may include random access memory (RAM) or external cache memory.
  • RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biomedical Technology (AREA)
  • Public Health (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Epidemiology (AREA)
  • Pathology (AREA)
  • Databases & Information Systems (AREA)
  • Primary Health Care (AREA)
  • Multimedia (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

Procédé et appareil de reconnaissance de caractéristique pulmonaire, dispositif informatique et support de stockage, qui appartiennent au domaine technique de l'intelligence artificielle. Le procédé consiste à : acquérir des données à reconnaître qui comprennent une image pulmonaire à reconnaître et une description de texte pulmonaire à reconnaître ; extraire des caractéristiques d'image pulmonaire au moyen d'un modèle de reconnaissance d'image pulmonaire et générer un vecteur de caractéristique d'image pulmonaire et un résultat de reconnaissance d'image, tout en extrayant des caractéristiques de texte pulmonaire au moyen d'un modèle de reconnaissance de texte pulmonaire et en générant un vecteur de caractéristique de texte pulmonaire et un résultat de reconnaissance de texte ; fusionner le vecteur de caractéristique d'image pulmonaire et le vecteur de caractéristique de texte pulmonaire au moyen d'un modèle de reconnaissance de fusion pulmonaire et en utilisant un mécanisme d'attention, extraire des caractéristiques de fusion de texte d'image pour la reconnaissance, et obtenir un résultat de reconnaissance de fusion ; et obtenir un résultat de reconnaissance de caractéristique pulmonaire au moyen d'un vote. Le procédé décrit permet une reconnaissance précise des caractéristiques pulmonaires et améliore la précision et la fiabilité de la reconnaissance. Le procédé décrit est applicable aux domaines du traitement médical intelligent, etc., et peut favoriser davantage la construction de villes intelligentes.
PCT/CN2021/096366 2020-09-21 2021-05-27 Procédé et appareil de reconnaissance de caractéristique pulmonaire, dispositif informatique et support de stockage WO2022057309A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010991495.4 2020-09-21
CN202010991495.4A CN111832581B (zh) 2020-09-21 2020-09-21 肺部特征识别方法、装置、计算机设备及存储介质

Publications (1)

Publication Number Publication Date
WO2022057309A1 true WO2022057309A1 (fr) 2022-03-24

Family

ID=72918738

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/096366 WO2022057309A1 (fr) 2020-09-21 2021-05-27 Procédé et appareil de reconnaissance de caractéristique pulmonaire, dispositif informatique et support de stockage

Country Status (2)

Country Link
CN (1) CN111832581B (fr)
WO (1) WO2022057309A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114863182A (zh) * 2022-05-23 2022-08-05 北京百度网讯科技有限公司 图像分类方法、图像分类模型的训练方法及装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111832581B (zh) * 2020-09-21 2021-01-29 平安科技（深圳）有限公司 肺部特征识别方法、装置、计算机设备及存储介质
CN112070069A (zh) * 2020-11-10 2020-12-11 支付宝(杭州)信息技术有限公司 遥感图像的识别方法和装置
CN113469049B (zh) * 2021-06-30 2024-05-10 平安科技（深圳）有限公司 一种疾病信息识别方法、系统、装置及存储介质
CN113409306A (zh) * 2021-07-15 2021-09-17 推想医疗科技股份有限公司 一种检测装置、训练方法、训练装置、设备和介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107392224A (zh) * 2017-06-12 2017-11-24 天津科技大学 一种基于三通道卷积神经网络的作物病害识别算法
CN108334605A (zh) * 2018-02-01 2018-07-27 腾讯科技（深圳）有限公司 文本分类方法、装置、计算机设备及存储介质
CN108399409A (zh) * 2018-01-19 2018-08-14 北京达佳互联信息技术有限公司 图像分类方法、装置及终端
CN109543714A (zh) * 2018-10-16 2019-03-29 北京达佳互联信息技术有限公司 数据特征的获取方法、装置、电子设备及存储介质
CN111460889A (zh) * 2020-02-27 2020-07-28 平安科技（深圳）有限公司 一种基于语音及图像特征的异常行为识别方法、装置及设备
CN111563551A (zh) * 2020-04-30 2020-08-21 支付宝(杭州)信息技术有限公司 一种多模态信息融合方法、装置及电子设备
CN111832581A (zh) * 2020-09-21 2020-10-27 平安科技（深圳）有限公司 肺部特征识别方法、装置、计算机设备及存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107665736B (zh) * 2017-09-30 2021-05-25 百度在线网络技术（北京）有限公司 用于生成信息的方法和装置
CN108334843B (zh) * 2018-02-02 2022-03-25 成都国铁电气设备有限公司 一种基于改进AlexNet的燃弧识别方法
CN110188343B (zh) * 2019-04-22 2023-01-31 浙江工业大学 基于融合注意力网络的多模态情感识别方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107392224A (zh) * 2017-06-12 2017-11-24 天津科技大学 一种基于三通道卷积神经网络的作物病害识别算法
CN108399409A (zh) * 2018-01-19 2018-08-14 北京达佳互联信息技术有限公司 图像分类方法、装置及终端
CN108334605A (zh) * 2018-02-01 2018-07-27 腾讯科技（深圳）有限公司 文本分类方法、装置、计算机设备及存储介质
CN109543714A (zh) * 2018-10-16 2019-03-29 北京达佳互联信息技术有限公司 数据特征的获取方法、装置、电子设备及存储介质
CN111460889A (zh) * 2020-02-27 2020-07-28 平安科技（深圳）有限公司 一种基于语音及图像特征的异常行为识别方法、装置及设备
CN111563551A (zh) * 2020-04-30 2020-08-21 支付宝(杭州)信息技术有限公司 一种多模态信息融合方法、装置及电子设备
CN111832581A (zh) * 2020-09-21 2020-10-27 平安科技（深圳）有限公司 肺部特征识别方法、装置、计算机设备及存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114863182A (zh) * 2022-05-23 2022-08-05 北京百度网讯科技有限公司 图像分类方法、图像分类模型的训练方法及装置

Also Published As

Publication number Publication date
CN111832581B (zh) 2021-01-29
CN111832581A (zh) 2020-10-27

Similar Documents

Publication Publication Date Title
WO2022057309A1 (fr) Procédé et appareil de reconnaissance de caractéristique pulmonaire, dispositif informatique et support de stockage
WO2019228317A1 (fr) Procédé et dispositif de reconnaissance faciale et support lisible par ordinateur
US11348249B2 (en) Training method for image semantic segmentation model and server
US11961233B2 (en) Method and apparatus for training image segmentation model, computer device, and storage medium
US11842487B2 (en) Detection model training method and apparatus, computer device and storage medium
WO2021135499A1 (fr) Procédés d'entraînement de modèle de détection de dommages et de détection de dommages de véhicule, dispositif, appareil et support
CN111062871B (zh) 一种图像处理方法、装置、计算机设备及可读存储介质
WO2021017261A1 (fr) Procédé et appareil d'entraînement de modèles de reconnaissance, procédé et appareil de reconnaissance d'images, et dispositif et support
WO2020228446A1 (fr) Procédé et appareil d'entraînement de modèles, et terminal et support de stockage
EP3327582A1 (fr) Procédé et appareil pour compléter un graphe de connaissances
WO2019007041A1 (fr) Procédé de récupération bidirectionnelle image-texte basé sur un espace de jonction multi-vues d'incorporation
EP4163831A1 (fr) Procédé et dispositif de distillation de réseau neuronal
CN110930417A (zh) 图像分割模型的训练方法和装置、图像分割方法和装置
WO2020024395A1 (fr) Procédé et appareil de détection de conduite dans un état de fatigue, dispositif informatique et support de stockage
CN111368672A (zh) 一种用于遗传病面部识别模型的构建方法及装置
WO2020238353A1 (fr) Procédé et appareil de traitement de données, support de stockage et dispositif électronique
US20230237771A1 (en) Self-supervised learning method and apparatus for image features, device, and storage medium
WO2021032062A1 (fr) Procédé de génération de modèle de traitement d'image, procédé de traitement d'image, appareil et dispositif électronique
CN112395979A (zh) 基于图像的健康状态识别方法、装置、设备及存储介质
CN110705489B (zh) 目标识别网络的训练方法、装置、计算机设备和存储介质
WO2021031704A1 (fr) Procédé et appareil de suivi d'objet, dispositif informatique et support de stockage
CN111583184A (zh) 图像分析方法、网络、计算机设备和存储介质
CN115050064A (zh) 人脸活体检测方法、装置、设备及介质
CN114330499A (zh) 分类模型的训练方法、装置、设备、存储介质及程序产品
CN113343981A (zh) 一种视觉特征增强的字符识别方法、装置和设备

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21868146

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21868146

Country of ref document: EP

Kind code of ref document: A1