CN116776872A - Medical data structured archiving system - Google Patents

Abstract

The application relates to the technical field of intelligent archiving, and particularly discloses a medical data structured archiving system which adopts an artificial intelligent detection technology based on deep learning to respectively extract high-dimensional implicit characteristic information of medical image images and text context semantic characteristics of text description from medical image data and mine multi-mode relevance characteristic distribution information of the high-dimensional implicit characteristic information and the text context semantic characteristics in a high-dimensional space so as to match subject labels. Thus, depth feature mining is fully and accurately performed on information in the medical image data, so that archiving of the medical image data is performed based on the subject tag to which the medical image data belongs more accurately, and archiving efficiency and accuracy of the medical image data are improved.

Description

Medical data structured archiving system

Technical Field

The application relates to the technical field of intelligent archiving, and more particularly relates to a medical data structured archiving system.

Background

The medical image data is huge in quantity, and the medical image data needs to be digitized and timely archived, so that the situation that a doctor cannot timely acquire the medical image data for diagnosis is avoided, and the timeliness of clinical diagnosis is further affected. The prior medical image data archiving work is performed manually, so that the efficiency is low, and the erroneous archiving can be caused by artificial negligence.

Thus, an optimized medical data structured archiving system is desired.

Disclosure of Invention

The present application has been made to solve the above-mentioned technical problems. The embodiment of the application provides a medical data structured filing system which adopts an artificial intelligent detection technology based on deep learning to respectively extract high-dimensional implicit characteristic information of medical image and text context semantic characteristics of text description from medical image data and mine multi-mode association characteristic distribution information of the two in a high-dimensional space so as to match affiliated theme labels. Thus, depth feature mining is fully and accurately performed on information in the medical image data, so that archiving of the medical image data is performed based on the subject tag to which the medical image data belongs more accurately, and archiving efficiency and accuracy of the medical image data are improved.

Accordingly, according to one aspect of the present application, there is provided a medical data structured archiving system, comprising: the data acquisition module is used for acquiring medical image data to be archived at the cloud; the data extraction module is used for extracting medical image images and text descriptions from the medical image data; the medical image feature extraction module is used for enabling the medical image to pass through a convolutional neural network model serving as a filter to obtain a medical image feature vector; the medical image text understanding module is used for enabling the text description to pass through a context encoder comprising a word embedding layer to obtain semantic understanding feature vectors; the multi-mode feature association module is used for carrying out association coding on the medical image feature vector and the semantic understanding feature vector so as to obtain a multi-mode association semantic feature matrix; the matching result generation module is used for enabling the multi-mode associated semantic feature matrix to pass through a classifier to obtain a classification result, wherein the classification result is used for representing a subject label to which medical image data to be archived belong; and the result response module is used for archiving the medical image data to be archived in the folder corresponding to the theme label based on the classification result.

In the above medical data structured archiving system, the medical image feature extraction module is further configured to: each layer using the convolutional neural network model is performed in forward transfer of the layer: carrying out convolution processing on input data to obtain a convolution characteristic diagram; carrying out mean pooling based on a local feature matrix on the convolution feature map to obtain a pooled feature map; performing nonlinear activation on the pooled feature map to obtain an activated feature map; the output of the last layer of the convolutional neural network model is the medical image feature vector, and the input of the first layer of the convolutional neural network model is the medical image.

In the medical data structured archiving system, the convolutional neural network model used as the filter is a depth residual network model.

In the above medical data structured archiving system, the medical image text understanding module includes: the word segmentation unit is used for carrying out word segmentation processing on the text description to obtain a plurality of words; a word embedding unit, configured to pass the plurality of words through a word embedding layer to convert each word in the plurality of words into a word embedding vector to obtain a sequence of word embedding vectors, where the embedding layer performs embedded encoding on each word using a learnable embedding matrix; a context encoding unit for inputting the sequence of word embedding vectors into the context encoder to obtain the plurality of context semantic feature vectors; and a concatenation unit, configured to concatenate the plurality of context semantic feature vectors to obtain the semantic understanding feature vector.

In the above medical data structured archiving system, the context encoding unit is further configured to: arranging the sequence of word embedding vectors into an input vector; respectively converting the input vector into a query vector and a key vector through a learning embedding matrix; calculating the product between the query vector and the transpose vector of the key vector to obtain a self-attention correlation matrix; carrying out standardization processing on the self-attention association matrix to obtain a standardized self-attention association matrix; inputting the standardized self-attention association matrix into a Softmax activation function to activate so as to obtain a self-attention feature matrix; and multiplying the self-attention feature matrix with each word embedding vector in the sequence of word embedding vectors as a value vector to obtain the context semantic feature vectors.

In the above medical data structured archiving system, the multi-modal feature association module is further configured to: and carrying out Hilbert space constraint of a vector mode basis on the medical image feature vector and the semantic understanding feature vector so as to carry out association coding on the medical image feature vector and the semantic understanding feature vector to obtain the multi-mode association semantic feature matrix.

In the above medical data structured archiving system, the multi-modal feature association module is further configured to: performing Hilbert space constraint of a vector mode basis on the medical image feature vector and the semantic understanding feature vector by using the following formula to perform association coding on the medical image feature vector and the semantic understanding feature vector so as to obtain the multi-mode association semantic feature matrix; wherein, the formula is:wherein->And->Separate tableShowing the medical image feature vector and the semantic understanding feature vector, < >>Representing the two norms of the vector, ">Expressed as convolution operator +.>For matrix->One-dimensional convolution is performed, wherein->And->Is a weight super parameter, and +.>And->Are column vectors, +.>Representing the multimodal associated semantic feature matrix, < >>Representing vector multiplication.

In the above medical data structured archiving system, the matching result generating module includes: the unfolding unit is used for unfolding the multi-mode associated semantic feature matrix into a classification feature vector according to a row vector or a column vector; the label probability unit is used for inputting the classification feature vector into a Softmax classification function of the classifier to obtain a probability value of the classification feature vector belonging to each classification label; and the classification result generation unit is used for determining the classification label corresponding to the maximum probability value as the classification result.

According to another aspect of the present application, there is also provided a medical data structured archiving method, comprising: acquiring medical image data to be archived at a cloud; extracting medical image images and text descriptions from the medical image data; the medical image is passed through a convolutional neural network model serving as a filter to obtain a medical image feature vector; passing the text description through a context encoder comprising a word embedding layer to obtain a semantic understanding feature vector; performing association coding on the medical image feature vector and the semantic understanding feature vector to obtain a multi-mode association semantic feature matrix; the multi-mode associated semantic feature matrix is passed through a classifier to obtain a classification result, wherein the classification result is used for representing a subject label to which medical image data to be archived belong; and archiving the medical image data to be archived in the folder corresponding to the theme label based on the classification result.

In the above method for structured archiving of medical data, the step of passing the medical image through a convolutional neural network model as a filter to obtain a medical image feature vector includes: each layer using the convolutional neural network model is performed in forward transfer of the layer: carrying out convolution processing on input data to obtain a convolution characteristic diagram; carrying out mean pooling based on a local feature matrix on the convolution feature map to obtain a pooled feature map; performing nonlinear activation on the pooled feature map to obtain an activated feature map; the output of the last layer of the convolutional neural network model is the medical image feature vector, and the input of the first layer of the convolutional neural network model is the medical image.

In the above medical data structured archiving method, the convolutional neural network model as a filter is a depth residual network model.

In the above method for structured archiving of medical data, said passing said text description through a context encoder comprising a word embedding layer to obtain a semantic understanding feature vector comprises: word segmentation processing is carried out on the text description to obtain a plurality of words; the words pass through a word embedding layer to convert each word in the words into a word embedding vector to obtain a sequence of word embedding vectors, wherein the embedding layer uses a learnable embedding matrix to carry out embedded coding on each word; inputting the sequence of word embedding vectors into the context encoder to obtain the plurality of context semantic feature vectors; and concatenating the plurality of context semantic feature vectors to obtain the semantic understanding feature vector.

In the above method for structured archiving of medical data, said inputting the sequence of word embedding vectors into the context encoder to obtain the plurality of context semantic feature vectors comprises: arranging the sequence of word embedding vectors into an input vector; respectively converting the input vector into a query vector and a key vector through a learning embedding matrix; calculating the product between the query vector and the transpose vector of the key vector to obtain a self-attention correlation matrix; carrying out standardization processing on the self-attention association matrix to obtain a standardized self-attention association matrix; inputting the standardized self-attention association matrix into a Softmax activation function to activate so as to obtain a self-attention feature matrix; and multiplying the self-attention feature matrix with each word embedding vector in the sequence of word embedding vectors as a value vector to obtain the context semantic feature vectors.

In the above method for structured archiving of medical data, the performing association encoding on the medical image feature vector and the semantic understanding feature vector to obtain a multi-modal associated semantic feature matrix includes: and carrying out Hilbert space constraint of a vector mode basis on the medical image feature vector and the semantic understanding feature vector so as to carry out association coding on the medical image feature vector and the semantic understanding feature vector to obtain the multi-mode association semantic feature matrix.

In the above method for structured archiving of medical data, the performing association encoding on the medical image feature vector and the semantic understanding feature vector to obtain a multi-modal associated semantic feature matrix includes: vector-based Hilbert spatial constraint on the medical image feature vector and the semantically understood feature vector is performed with the following formulaPerforming association coding on the medical image feature vector and the semantic understanding feature vector to obtain the multi-mode association semantic feature matrix; wherein, the formula is:wherein->And->Representing the medical image feature vector and the semantic understanding feature vector, respectively, < >>Representing the two norms of the vector, " >Expressed as convolution operator +.>For matrix->One-dimensional convolution is performed, wherein->And->Is a weight super parameter, and +.>And->Are column vectors, +.>Representing the multimodal associated semantic feature matrix, < >>Representing vector multiplication. The medical data structured filing methodThe step of passing the multi-mode associated semantic feature matrix through a classifier to obtain a classification result, wherein the classification result is used for representing a subject tag to which medical image data to be archived belongs, and the method comprises the following steps: expanding the multi-mode associated semantic feature matrix into a classification feature vector according to a row vector or a column vector; inputting the classification feature vector into a Softmax classification function of the classifier to obtain a probability value of the classification feature vector belonging to each classification label; and determining the classification label corresponding to the maximum probability value as the classification result.

According to still another aspect of the present application, there is provided an electronic apparatus including: a processor; and a memory having stored therein computer program instructions that, when executed by the processor, cause the processor to perform the medical data structured archiving method as described above.

According to yet another aspect of the present application, there is provided a computer readable medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the medical data structured archiving method as described above.

Compared with the prior art, the medical data structured filing system provided by the application adopts an artificial intelligent detection technology based on deep learning, so that matching of the subject labels is carried out by respectively extracting high-dimensional implicit characteristic information of medical image images and text context semantic characteristics of text description from medical image data and mining multi-mode associative characteristic distribution information of the two in a high-dimensional space. Thus, depth feature mining is fully and accurately performed on information in the medical image data, so that archiving of the medical image data is performed based on the subject tag to which the medical image data belongs more accurately, and archiving efficiency and accuracy of the medical image data are improved.

Drawings

The above and other objects, features and advantages of the present application will become more apparent by describing embodiments of the present application in more detail with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate the application and together with the embodiments of the application, and not constitute a limitation to the application. In the drawings, like reference numerals generally refer to like parts or steps.

FIG. 1 is a block diagram of a medical data structured archiving system in accordance with an embodiment of the present application.

Fig. 2 is a schematic architecture diagram of a medical data structured archiving system according to an embodiment of the present application.

Fig. 3 is a block diagram of a medical image text understanding module in a medical data structured archiving system according to an embodiment of the present application.

Fig. 4 is a flow chart of a method of structured archiving of medical data in accordance with an embodiment of the present application.

Fig. 5 is a block diagram of an electronic device according to an embodiment of the application.

Detailed Description

Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

Summary of the application: as described above, the conventional medical image data archiving operation is performed manually, which is inefficient and may cause erroneous archiving due to human carelessness. Thus, an optimized medical data structured archiving system is desired.

Accordingly, in order to timely and accurately archive the medical image data, feature information of the medical image data needs to be fully and accurately feature mined, and then the matching of the subject labels is performed according to the hidden features of the medical image data, so that the archiving is completed. However, considering that the medical image data comprises the medical image and the text description, which are of different data types, the difficulty is how to mine the multi-modal relevance feature distribution information of the high-dimensional implicit feature information of the image in the medical image data and the text context semantic feature information of the text description, so that the depth feature mining is fully and accurately performed on the information in the medical image data, and the medical image data is more accurately filed based on the topic label to which the medical image data belongs, so that the filing efficiency and the accuracy of the medical image data are improved.

In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. In addition, deep learning and neural networks have also shown levels approaching and even exceeding humans in the fields of image classification, object detection, semantic segmentation, text translation, and the like.

The development of deep learning and neural networks provides new solutions and schemes for mining multi-modal associative feature distribution information of high-dimensional implicit feature information of image images and text context semantic feature information of text descriptions in the medical image data. Those of ordinary skill in the art will appreciate that deep learning based deep neural network models can adjust parameters of the deep neural network model by appropriate training strategies, such as by a gradient descent back-propagation algorithm, to enable modeling of complex nonlinear correlations between things, which is obviously suitable for modeling and mining multi-modal associative feature distribution information of high-dimensional implicit feature information of image images and text-described text-context semantic feature information in the medical image material.

Specifically, in the technical scheme of the application, firstly, medical image data to be archived is obtained through a cloud. Next, since the medical image data and the text data are different from each other in consideration of the existence of the medical image data and the text data in the medical image data, it is necessary to extract data features of the two parts in order to sufficiently perform depth feature mining on the medical image data. Specifically, first, a medical image and a text description are extracted from the medical image material.

Then, considering that since the medical image in the medical image data is of an image data type, in the technical scheme of the present application, feature mining of the medical image is performed using a convolutional neural network model as a filter having excellent performance in terms of implicit feature extraction of images to extract high-dimensional implicit feature distribution information about the medical image of the patient in the medical image, thereby obtaining a medical image feature vector.

Further, for the text description of the medical image material, since the text description is composed of a plurality of words, and each word in the text description has a semantic association relationship of context with different degrees, that is, each word may compose a phrase or sentence, and each word or sentence does not exist independently, and also has semantic feature information of context. Therefore, in the technical scheme of the application, in order to sufficiently and accurately perform semantic understanding on the text description so as to improve the archiving accuracy of the medical image data, the text description is further subjected to word segmentation processing so as to avoid word sequence confusion, and then is subjected to semantic encoding in a context encoder comprising a word embedding layer so as to extract global context semantic understanding characteristics about the medical image data in the text description, thereby obtaining semantic understanding characteristic vectors. That is, based on the transformer concept, the converter is used to capture the long-distance context-dependent characteristic, and global context-based semantic coding is performed on each word in the text description to obtain the semantic understanding feature vector with the context semantic feature information of the text description as a whole.

Then, considering that the medical image and text in the medical image data are described as different types of data, in order to improve the accuracy of archiving the medical image data by using the relevance characteristic information of the medical image and text, in the technical scheme of the application, the medical image characteristic vector and the semantic understanding characteristic vector are further associated and encoded in a high-dimensional space, so as to obtain a multi-mode relevance semantic characteristic matrix with the medical image characteristic and the text description semantic understanding characteristic. Accordingly, in a specific example of the present application, the multi-modal associated semantic feature matrix may be obtained by calculating a vector multiplication of the transpose vector of the medical image feature vector and the semantic understanding feature vector.

And then, further taking the multi-mode associated semantic feature matrix as a classification feature matrix to perform classification processing in a classifier so as to obtain a subject tag classification result for representing the medical image data to be archived. In other words, in the technical solution of the present application, the tag of the classifier is a topic tag to which the medical image data to be archived belongs, so that the multi-mode associated semantic feature matrix with the fusion feature of the image feature and the text semantic feature of the medical image data is classified, so as to match the topic tag to which the medical image data to be archived belongs to obtain the classification result, and then the medical image data to be archived is archived in the folder corresponding to the topic tag based on the classification result. Therefore, the topic labels to which the medical image data to be archived belong can be accurately matched, and the medical image data is archived.

In particular, in the technical scheme of the application, when the medical image feature vector and the semantic understanding feature vector are subjected to association coding to obtain the multi-mode association semantic feature matrix, the medical image feature vector is obtained by carrying out convolution coding on the medical image, and the semantic understanding feature vector is obtained by carrying out context semantic understanding on text description in the medical image data to be archived, namely, the medical image feature vector and the semantic understanding feature vector are respectively obtained by coding data of different modes in different coding modes, so that the medical image feature vector and the semantic understanding feature vector have semantic mismatch, even data problems such as pathological alignment and the like in a high-dimensional feature space, and the convergence of feature distribution of the multi-mode association semantic feature matrix in the high-dimensional feature space is poor after association coding is carried out, thereby influencing the fitting effect of a classifier.

Based on this, the medical treatment is performedImage feature vectorAnd the semantic understanding feature vector +.>Hilbert space constraint of vector mode basis is performed to obtain a multi-modal associated semantic feature matrix, e.g. denoted +. >Expressed as: representing one-dimensional convolution operations, i.e. with the convolution operator +.>For matrix->One-dimensional convolution is performed, wherein->And->Is a weight super parameter, andand->Are column vectors.

Here, the semantic feature matrix is associated with multiple modes by convolving the matrix with convolution operators in Hilbert space defining vector sum modulo and vector inner productConstraint is carried out, and a multi-mode associated semantic feature matrix can be +.>Is defined in a finite closed domain in a Hilbert space based on the modulus of the vector and promotes a multimodal associative semantic feature matrix +.>Orthogonality between the base dimensions of the high-dimensional manifold of the feature distribution, thereby achieving sparse correlation between feature values while maintaining convergence of the feature distribution as a whole. In this way, the fitting effect of the multi-mode associated semantic feature matrix through the classifier and the accuracy of the classification result are improved. Thus, the medical image data can be accurately archived, so that the archiving efficiency and accuracy of the medical image data can be improved.

Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described in detail with reference to the accompanying drawings.

Exemplary System: FIG. 1 is a block diagram of a medical data structured archiving system in accordance with an embodiment of the present application. As shown in fig. 1, a medical data structured archiving system 100 according to an embodiment of the present application includes: the data acquisition module 110 is used for acquiring medical image data to be archived at the cloud; a data extraction module 120 for extracting a medical image and a text description from the medical image material; a medical image feature extraction module 130, configured to pass the medical image through a convolutional neural network model serving as a filter to obtain a medical image feature vector; a medical image text understanding module 140 for passing the text description through a context encoder comprising a word embedding layer to obtain a semantic understanding feature vector; the multi-mode feature association module 150 is configured to perform association encoding on the medical image feature vector and the semantic understanding feature vector to obtain a multi-mode associated semantic feature matrix; the matching result generating module 160 is configured to pass the multi-mode associated semantic feature matrix through a classifier to obtain a classification result, where the classification result is used to represent a subject tag to which medical image data to be archived belongs; and a result response module 170, configured to archive the medical image data to be archived in a folder corresponding to the theme tag based on the classification result.

Fig. 2 is a schematic architecture diagram of a medical data structured archiving system according to an embodiment of the present application. As shown in fig. 2, first, medical image data to be archived is acquired at a cloud end; then, extracting a medical image and a text description from the medical image data; then, the medical image is passed through a convolutional neural network model as a filter to obtain a medical image feature vector, and simultaneously, the text description is passed through a context encoder comprising a word embedding layer to obtain a semantic understanding feature vector; then, carrying out association coding on the medical image feature vector and the semantic understanding feature vector to obtain a multi-mode association semantic feature matrix; the multi-mode associated semantic feature matrix is passed through a classifier to obtain a classification result, wherein the classification result is used for representing a subject label to which medical image data to be archived belong; and finally, based on the classification result, archiving the medical image data to be archived in the folder corresponding to the theme label.

As described above, the conventional medical image data archiving operation is performed manually, which is inefficient and may cause erroneous archiving due to human carelessness. Thus, an optimized medical data structured archiving system is desired.

Accordingly, in order to timely and accurately archive the medical image data, feature information of the medical image data needs to be fully and accurately feature mined, and then the topic labels are matched according to hidden features of the medical image data, so that archiving is completed. However, considering that the medical image data comprises the medical image and the text description, which are of different data types, the difficulty is how to mine the multi-modal relevance feature distribution information of the high-dimensional implicit feature information of the image in the medical image data and the text context semantic feature information of the text description, so that the depth feature mining is fully and accurately performed on the information in the medical image data, and the medical image data is more accurately filed based on the topic label to which the medical image data belongs, so that the filing efficiency and the accuracy of the medical image data are improved.

In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. In addition, deep learning and neural networks have also shown levels approaching and even exceeding humans in the fields of image classification, object detection, semantic segmentation, text translation, and the like.

The development of deep learning and neural networks provides new solutions and schemes for mining multi-modal associative feature distribution information of high-dimensional implicit feature information of image images and text context semantic feature information of text descriptions in the medical image data. Those of ordinary skill in the art will appreciate that deep learning based deep neural network models can adjust parameters of the deep neural network model by appropriate training strategies, such as by a gradient descent back-propagation algorithm, to enable modeling of complex nonlinear correlations between things, which is obviously suitable for modeling and mining multi-modal associative feature distribution information of high-dimensional implicit feature information of image images and text-described text-context semantic feature information in the medical image material.

In the above medical data structured archiving system 100, the data acquisition module 110 and the data extraction module 120 are configured to acquire medical image data to be archived at the cloud end, and extract medical image images and text descriptions from the medical image data. Since the medical image data and the text data are different types in consideration of existence of the medical image data in the medical image data, it is necessary to extract data features of the two parts in order to sufficiently perform depth feature mining on the medical image data. Specifically, first, a medical image and a text description are extracted from the medical image material.

In the above medical data structured archiving system 100, the medical image feature extraction module 130 is configured to pass the medical image through a convolutional neural network model as a filter to obtain a medical image feature vector. In view of the fact that the medical image in the medical image data is of an image data type, in the technical scheme of the application, after the medical image is extracted from the medical image data, a convolution neural network model which is a filter and has excellent performance in terms of implicit feature extraction of images is used for carrying out feature mining on the medical image so as to extract high-dimensional implicit feature distribution information about the medical image of the patient in the medical image, thereby obtaining medical image feature vectors. In a specific example of the present application, the convolutional neural network model as a filter is a depth residual network model.

It should be understood that the development of the neural network is brought into a stage of several tens of layers by the network model such as AlexNet, VGG, googLeNet, and the deeper the layer number of the network is, the more likely the generalization capability is obtained. But as the model deepens, the network becomes more and more difficult to train, mainly due to gradient diffusion and gradient explosion phenomena. In a neural network with a deeper layer, when gradient information is transmitted from the last layer of the network to the first layer of the network layer by layer, the gradient is close to 0 or the gradient value is very large in the transmission process. Since the shallow neural network is not prone to gradient phenomena, an attempt may be made to add a mechanism to the deep neural network to fall back to the shallow neural network. When the deep neural network can easily fall back to the shallow neural network, the deep neural network can obtain model performance equivalent to that of the shallow neural network. Thus, a deep residual network (Residual Neural Network, resNet) model is proposed that allows neural networks to have rollback capability by adding a directly connected Skip Connection between the input and output.

Specifically, in the embodiment of the present application, the medical image feature extraction module 130 uses the layers of the convolutional neural network model as the filter to perform the following steps in forward transfer of the layers: carrying out convolution processing on input data to obtain a convolution characteristic diagram; carrying out mean pooling based on a local feature matrix on the convolution feature map to obtain a pooled feature map; performing nonlinear activation on the pooled feature map to obtain an activated feature map; the output of the last layer of the convolutional neural network model is the medical image feature vector, and the input of the first layer of the convolutional neural network model is the medical image.

In the above-described medical data structured archiving system 100, the medical image text understanding module 140 is configured to pass the text description through a context encoder including a word embedding layer to obtain a semantic understanding feature vector. For the text description of the medical image data, since the text description is composed of a plurality of words, and each word in the text description has a semantic association relationship of context with different degrees, that is, each word can compose a phrase or sentence, and each word or sentence does not exist independently, and also has semantic feature information of context. Therefore, in the technical scheme of the application, in order to sufficiently and accurately perform semantic understanding on the text description so as to improve the archiving accuracy of the medical image data, the text description is further subjected to word segmentation processing so as to avoid word sequence confusion, and then is subjected to semantic encoding in a context encoder comprising a word embedding layer so as to extract global context semantic understanding characteristics about the medical image data in the text description, thereby obtaining semantic understanding characteristic vectors. That is, based on the transformer concept, the converter is used to capture the long-distance context-dependent characteristic, and global context-based semantic coding is performed on each word in the text description to obtain the semantic understanding feature vector with the context semantic feature information of the text description as a whole.

Fig. 3 is a block diagram of a medical image text understanding module in a medical data structured archiving system according to an embodiment of the present application. As shown in fig. 3, the medical image text understanding module 140 includes: a word segmentation unit 141, configured to perform word segmentation processing on the text description to obtain a plurality of words; a word embedding unit 142, configured to pass the plurality of words through a word embedding layer to convert each word in the plurality of words into a word embedding vector to obtain a sequence of word embedding vectors, where the embedding layer performs embedded encoding on each word using a learnable embedding matrix; a context encoding unit 143 for inputting the sequence of word embedding vectors into the context encoder to obtain the plurality of context semantic feature vectors; and a concatenation unit 144, configured to concatenate the plurality of context semantic feature vectors to obtain the semantic understanding feature vector.

More specifically, in the embodiment of the present application, the encoding process of the context encoding unit 143 is: firstly, arranging a sequence of word embedding vectors into input vectors; then, the input vector is respectively converted into a query vector and a key vector through a learning embedding matrix; then, calculating the product between the query vector and the transpose vector of the key vector to obtain a self-attention correlation matrix; then, carrying out standardization processing on the self-attention association matrix to obtain a standardized self-attention association matrix; subsequently, the standardized self-attention association matrix is input into a Softmax activation function to be activated so as to obtain a self-attention feature matrix; and finally, multiplying the self-attention feature matrix by each word embedding vector in the sequence of word embedding vectors as a value vector to obtain the context semantic feature vectors.

In the above medical data structured archiving system 100, the multi-modal feature association module 150 is configured to perform association encoding on the medical image feature vector and the semantic understanding feature vector to obtain a multi-modal associated semantic feature matrix. In order to improve the accuracy of archiving the medical image data by using the association characteristic information of the medical image data and the text description as different types of data, the medical image characteristic vector and the semantic understanding characteristic vector are further associated and encoded in a high-dimensional space in the technical scheme of the application, so that a multi-mode association semantic characteristic matrix with the medical image characteristic and the text description semantic understanding characteristic is obtained. Accordingly, in a specific example of the present application, the multi-modal associated semantic feature matrix may be obtained by calculating a vector multiplication of the transpose vector of the medical image feature vector and the semantic understanding feature vector.

In another specific example of the present application, the medical image feature vector and the semantic understanding feature vector may be subjected to hilbert space constraint of vector modulus base to perform association encoding on the medical image feature vector and the semantic understanding feature vector to obtain the multi-mode association semantic feature matrix. In the technical scheme of the application, when the medical image feature vector and the semantic understanding feature vector are subjected to association coding to obtain the multi-mode association semantic feature matrix, the medical image feature vector is obtained by carrying out convolution coding on the medical image, and the semantic understanding feature vector is obtained by carrying out context semantic understanding on text description in medical image data to be archived, namely, the medical image feature vector and the semantic understanding feature vector are respectively obtained by coding data of different modes in different coding modes, so that the medical image feature vector and the semantic understanding feature vector have semantic mismatch, even disease alignment and other data problems in a high-dimensional feature space, and the feature distribution of the multi-mode association semantic feature matrix has poor convergence in the high-dimensional feature space after association coding, thereby influencing the fitting effect of a classifier.

Based on the above, the medical image feature vectorAnd the semantic understanding feature vector +.>Hilbert space constraint of vector mode basis is performed to obtain a multi-modal associated semantic feature matrix, e.g. denoted +.>Expressed as:wherein->And->Representing the medical image feature vector and the semantic understanding feature vector, respectively, < >>Representing the two norms of the vector, ">Expressed in terms of convolution operatorsFor matrix->One-dimensional convolution is performed, wherein->And->Is a weight super parameter, and +.>And->Are column vectors, +.>Representing the multimodal associated semantic feature matrix, < >>Representing vector multiplication.

Here, the semantic feature matrix is associated with multiple modes by convolving the matrix with convolution operators in Hilbert space defining vector sum modulo and vector inner productConstraint is carried out, and a multi-mode associated semantic feature matrix can be +.>Is defined in a finite closed domain in a Hilbert space based on the modulus of the vector and promotes a multimodal associative semantic feature matrix +.>Orthogonality between the base dimensions of the high-dimensional manifold of the feature distribution, thereby achieving sparse correlation between feature values while maintaining convergence of the feature distribution as a whole. In this way, the fitting effect of the multi-mode associated semantic feature matrix through the classifier and the accuracy of the classification result are improved. Thus, the medical image data can be accurately archived, so that the archiving efficiency and accuracy of the medical image data can be improved.

In the above medical data structured archiving system 100, the matching result generating module 160 is configured to pass the multi-mode associated semantic feature matrix through a classifier to obtain a classification result, where the classification result is used to represent a subject tag to which the medical image data to be archived belongs. That is, in the technical scheme of the present application, the label of the classifier is a subject label to which the medical image data to be archived belongs.

Specifically, in the embodiment of the present application, the encoding process of the matching result generating module 160 includes: firstly, expanding the multi-mode associated semantic feature matrix into a classification feature vector according to a row vector or a column vector through an expanding unit; then, inputting the classification feature vector into a Softmax classification function of the classifier through a label probability unit to obtain a probability value of the classification feature vector belonging to each classification label; and finally, determining the classification label corresponding to the maximum probability value as the classification result through a classification result generation unit.

In the above medical data structured archiving system 100, the result response module 170 is configured to archive the medical image data to be archived in a folder corresponding to the theme tag based on the classification result. The method comprises the steps of classifying the multi-mode associated semantic feature matrix with fusion features of image features and text semantic features of the medical image data, so as to match theme labels of the medical image data to be archived to obtain classification results, and archiving the medical image data to be archived in folders corresponding to the theme labels based on the classification results. Therefore, the topic labels to which the medical image data to be archived belong can be accurately matched, and the medical image data is archived.

In summary, the medical data structured archiving system 100 according to the embodiment of the present application is illustrated, which adopts an artificial intelligence detection technology based on deep learning to extract the high-dimensional implicit feature information of the medical image and the text context semantic features of the text description from the medical image data, and mine the multi-modal associated feature distribution information of the two in the high-dimensional space, so as to perform matching of the subject tag. Thus, depth feature mining is fully and accurately performed on information in the medical image data, so that archiving of the medical image data is performed based on the subject tag to which the medical image data belongs more accurately, and archiving efficiency and accuracy of the medical image data are improved.

As described above, the medical data structured archiving system 100 according to the embodiment of the present application can be implemented in various terminal devices, such as a server or the like for structured archiving of medical data. In one example, the medical data structured archiving system 100 according to an embodiment of the present application may be integrated into the terminal device as a software module and/or hardware module. For example, the medical data structured archiving system 100 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the medical data structured archiving system 100 could equally be one of the plurality of hardware modules of the terminal device.

Alternatively, in another example, the medical data structured archiving system 100 and the terminal device may be separate devices, and the medical data structured archiving system 100 may be connected to the terminal device via a wired and/or wireless network and communicate the interaction information in accordance with the agreed data format.

An exemplary method is: fig. 4 is a flow chart of a method of structured archiving of medical data in accordance with an embodiment of the present application. As shown in fig. 4, a method for structured archiving of medical data according to an embodiment of the present application includes: s110, acquiring medical image data to be archived at a cloud end; s120, extracting medical image images and text descriptions from the medical image data; s130, the medical image is processed through a convolutional neural network model serving as a filter to obtain a medical image feature vector; s140, enabling the text description to pass through a context encoder comprising a word embedding layer to obtain semantic understanding feature vectors; s150, performing association coding on the medical image feature vector and the semantic understanding feature vector to obtain a multi-mode association semantic feature matrix; s160, passing the multi-mode associated semantic feature matrix through a classifier to obtain a classification result, wherein the classification result is used for representing a subject label to which medical image data to be archived belong; and S170, archiving the medical image data to be archived in the folder corresponding to the theme tag based on the classification result.

Here, it will be appreciated by those skilled in the art that the respective steps and operations in the above-described medical data structured archiving method have been described in detail in the above description of the medical data structured archiving system 100 with reference to fig. 1 to 3, and thus, repetitive descriptions thereof will be omitted.

Exemplary electronic device: next, an electronic device according to an embodiment of the present application is described with reference to fig. 5. Fig. 5 is a block diagram of an electronic device according to an embodiment of the application. As shown in fig. 5, the electronic device 10 includes one or more processors 11 and a memory 12.

The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer readable storage medium that can be executed by the processor 11 to perform the functions in the medical data structured archiving method of the various embodiments of the present application described above and/or other desired functions. Various content, such as medical image material to be archived, may also be stored in the computer-readable storage medium.

In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other forms of connection mechanisms (not shown).

The input means 13 may comprise, for example, a keyboard, a mouse, etc.

The output device 14 may output various information including the classification result and the like to the outside. The output means 14 may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, etc.

Of course, only some of the components of the electronic device 10 that are relevant to the present application are shown in fig. 5 for simplicity, components such as buses, input/output interfaces, etc. are omitted. In addition, the electronic device 10 may include any other suitable components depending on the particular application.

Exemplary computer program product and computer readable storage Medium: in addition to the methods and apparatus described above, embodiments of the application may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform steps in the functions of the medical data structured archiving method according to the various embodiments of the application described in the "exemplary methods" section of this specification.

The computer program product may write program code for performing operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the application may also be a computer-readable storage medium, having stored thereon computer program instructions, which when executed by a processor, cause the processor to perform steps in the functions of the medical data structured archiving method according to the various embodiments of the application described in the above "exemplary methods" section of the present description.

The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The basic principles of the present application have been described above in connection with specific embodiments, however, it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are merely examples and not intended to be limiting, and these advantages, benefits, effects, etc. are not to be considered as essential to the various embodiments of the present application. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, as the application is not necessarily limited to practice with the above described specific details.

The block diagrams of the devices, apparatuses, devices, systems referred to in the present application are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.

It is also noted that in the apparatus, devices and methods of the present application, the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent aspects of the present application.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the application to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.

Claims (8)

1. A medical data structured archiving system, comprising: the data acquisition module is used for acquiring medical image data to be archived at the cloud; the data extraction module is used for extracting medical image images and text descriptions from the medical image data; the medical image feature extraction module is used for enabling the medical image to pass through a convolutional neural network model serving as a filter to obtain a medical image feature vector; the medical image text understanding module is used for enabling the text description to pass through a context encoder comprising a word embedding layer to obtain semantic understanding feature vectors; the multi-mode feature association module is used for carrying out association coding on the medical image feature vector and the semantic understanding feature vector so as to obtain a multi-mode association semantic feature matrix; the matching result generation module is used for enabling the multi-mode associated semantic feature matrix to pass through a classifier to obtain a classification result, wherein the classification result is used for representing a subject label to which medical image data to be archived belong; and the result response module is used for archiving the medical image data to be archived in the folder corresponding to the theme label based on the classification result.

2. The medical data structured archiving system of claim 1, wherein the medical image feature extraction module is further to: each layer using the convolutional neural network model is performed in forward transfer of the layer: carrying out convolution processing on input data to obtain a convolution characteristic diagram; carrying out mean pooling based on a local feature matrix on the convolution feature map to obtain a pooled feature map; non-linear activation is carried out on the pooled feature map so as to obtain an activated feature map; the output of the last layer of the convolutional neural network model is the medical image feature vector, and the input of the first layer of the convolutional neural network model is the medical image.

3. The medical data structured archiving system of claim 2, wherein the convolutional neural network model as a filter is a depth residual network model.

4. The medical data structured archiving system of claim 3, wherein the medical image text understanding module comprises: the word segmentation unit is used for carrying out word segmentation processing on the text description to obtain a plurality of words; a word embedding unit, configured to pass the plurality of words through a word embedding layer to convert each word in the plurality of words into a word embedding vector to obtain a sequence of word embedding vectors, where the embedding layer performs embedded encoding on each word using a learnable embedding matrix; a context encoding unit for inputting the sequence of word embedding vectors into the context encoder to obtain the plurality of context semantic feature vectors; and a concatenation unit, configured to concatenate the plurality of context semantic feature vectors to obtain the semantic understanding feature vector.

5. The medical data structured archiving system of claim 4, wherein the context encoding unit is further to: arranging the sequence of word embedding vectors into an input vector; respectively converting the input vector into a query vector and a key vector through a learning embedding matrix; calculating the product between the query vector and the transpose vector of the key vector to obtain a self-attention correlation matrix; carrying out standardization processing on the self-attention association matrix to obtain a standardized self-attention association matrix; inputting the standardized self-attention association matrix into a Softmax activation function to activate so as to obtain a self-attention feature matrix; and multiplying the self-attention feature matrix with each word embedding vector in the sequence of word embedding vectors as a value vector to obtain the plurality of context semantic feature vectors.

6. The medical data structured archiving system of claim 5, wherein the multi-modal feature association module is further to: and carrying out Hilbert space constraint of a vector mode basis on the medical image feature vector and the semantic understanding feature vector so as to carry out association coding on the medical image feature vector and the semantic understanding feature vector to obtain the multi-mode association semantic feature matrix.

7. The medical data structured archiving system of claim 6, wherein the multi-modal feature association module is further to: performing Hilbert space constraint of a vector mode basis on the medical image feature vector and the semantic understanding feature vector by using the following formula to perform association coding on the medical image feature vector and the semantic understanding feature vector so as to obtain the multi-mode association semantic feature matrix; wherein, the formula is:wherein->And->Representing the medical image feature vector and the semantic understanding feature vector, respectively, < >>Representing the two norms of the vector, ">Expressed in terms of convolution operatorsFor matrix->One-dimensional convolution is performed, wherein->And->Is a weight super parameter, and +.>And->Are column vectors, +.>Representing the multimodal associated semantic feature matrix, < >>Representing vector multiplication.

8. The medical data structured archiving system of claim 7, wherein the match result generation module comprises: the unfolding unit is used for unfolding the multi-mode associated semantic feature matrix into a classification feature vector according to a row vector or a column vector; the label probability unit is used for inputting the classification feature vector into a Softmax classification function of the classifier to obtain a probability value of the classification feature vector belonging to each classification label; and the classification result generation unit is used for determining the classification label corresponding to the maximum probability value as the classification result.