CN111008957A - Medical information processing method and device - Google Patents
Medical information processing method and device Download PDFInfo
- Publication number
- CN111008957A CN111008957A CN201911135115.0A CN201911135115A CN111008957A CN 111008957 A CN111008957 A CN 111008957A CN 201911135115 A CN201911135115 A CN 201911135115A CN 111008957 A CN111008957 A CN 111008957A
- Authority
- CN
- China
- Prior art keywords
- information
- clinical
- feature vector
- vector
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/353—Clustering; Classification into predefined classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
The application embodiment provides a medical information processing method and device, electronic equipment and a computer readable storage medium, and solves the problem that the existing medical information processing mode only depends on medical analysis information acquired by medical images and is not comprehensive enough and not accurate enough. The medical information processing method includes: inputting a medical image into an image feature extraction model to obtain an image feature vector output by the image feature extraction model, wherein the image feature extraction model is a convolutional neural network model established through a training process; converting clinical text information into a clinical information feature vector through a coding process, wherein a vector value on each dimension of the clinical information feature vector is used for representing one type of clinical information in the clinical text information; splicing the image feature vector and the clinical information feature vector to obtain a spliced vector; and processing the spliced vector based on the first full-link layer to obtain a classification result.
Description
Technical Field
The application relates to the technical field of artificial intelligence, in particular to a medical information processing method and device, electronic equipment and a computer readable storage medium.
Background
With the continuous development of information technology, clinicians often need to acquire medical analysis information by means of medical images to assist in diagnosis of diseases. But considering only image features in medical images is not sufficient for complex medical application scenarios. Because in the actual doctor's inquiry scene, much information is actually the non-medical image information obtained by the description of the patient and the doctor's summary, the medical image is only one means for assisting the doctor to make a diagnosis. Therefore, when the medical history information of the patient cannot be combined, a lot of non-image information is lost when only the medical image is analyzed, so that the acquired medical analysis information is not comprehensive and accurate enough.
Disclosure of Invention
In view of this, embodiments of the present application provide a medical information processing method and apparatus, an electronic device, and a computer-readable storage medium, which solve the problem that the existing medical information processing method only depends on medical analysis information obtained from a medical image and is not comprehensive enough and not accurate enough.
According to an aspect of the present application, an embodiment of the present application provides a medical information processing method including: inputting a medical image into an image feature extraction model to obtain an image feature vector output by the image feature extraction model, wherein the image feature extraction model is a convolution neural network model established through a training process; converting clinical text information into a clinical information feature vector through a coding process, wherein a vector value on each dimension of the clinical information feature vector is used for representing one type of clinical information in the clinical text information; stitching the image feature vector and the clinical information feature vector to obtain a stitched vector; and processing the spliced vector based on the first full-link layer to obtain a classification result.
In an embodiment of the present application, before stitching the image feature vector and the clinical information feature vector to obtain a stitched vector, the method further includes: processing the clinical information feature vector based on a second fully connected layer to extend a number of dimensions of the clinical information feature vector.
In an embodiment of the present application, the number of dimensions of the image feature vector is 1024 dimensions, and the number of dimensions of the clinical information feature vector after expansion is 128 dimensions.
In an embodiment of the present application, the image feature extraction model is an image detection model, wherein the inputting the medical image into the image feature extraction model to obtain the image feature vector output by the image feature extraction model includes: inputting a medical image into a first convolutional neural network to obtain a first feature map, wherein the first convolutional neural network comprises at least one first convolutional layer and at least one first pooling layer; and inputting the region of interest in the first feature map into a region of interest pooling layer to obtain a region of interest feature vector output by the region of interest pooling layer as the image feature vector.
In an embodiment of the present application, the image feature extraction model is an image classification model, wherein the inputting the medical image into the image feature extraction model to obtain the image feature vector output by the image feature extraction model includes: inputting the medical image into a second convolutional neural network to obtain the image feature vector.
In an embodiment of the present application, the converting the clinical text information into the clinical information feature vector through an encoding process includes: and determining a corresponding coding mode according to the category of the clinical information in the clinical text information, and coding the clinical information in the clinical text information in the corresponding coding mode.
In an embodiment of the present application, one of the clinical information items includes a plurality of mutually exclusive discrete values, where encoding the clinical information item in the corresponding encoding manner includes: encoding the mutually exclusive discrete values as different vector values in a single dimension corresponding to the clinical information, respectively.
In an embodiment of the present application, one of the clinical information in the clinical text information is represented by a quantitative value, wherein the encoding the clinical information in the clinical text information in the corresponding encoding manner includes: unifying the quantitative value of one clinical information in the clinical text information; and a vector value on a dimension corresponding to the clinical information in the clinical information feature vector is (the quantized value-the lowest quantized value of the clinical information)/the largest quantized value of the clinical information.
In an embodiment of the present application, one of the clinical information items includes a plurality of different status information items, wherein the encoding the clinical information item in the corresponding encoding manner includes: and encoding each of the plurality of different types of status information into a plurality of dimensions of vector values corresponding to the clinical information, wherein the status information uniquely corresponds to one of the plurality of dimensions in a vector value combination manner.
According to another aspect of the present application, an embodiment of the present application provides a medical information processing apparatus including: the image feature vector acquisition module is configured to input the medical image into an image feature extraction model to acquire an image feature vector output by the image feature extraction model, wherein the image feature extraction model is a convolutional neural network model established through a training process; the clinical information feature vector acquisition module is configured to convert clinical text information into a clinical information feature vector through an encoding process, wherein a vector value in each dimension of the clinical information feature vector is used for representing one type of clinical information in the clinical text information; a stitching module configured to stitch the image feature vector and the clinical information feature vector to obtain a stitched vector; and the classification module is configured to process the splicing vector based on the first full-link layer to obtain a classification result.
In an embodiment of the present application, the apparatus further comprises: a dimension expansion module configured to process the clinical information feature vector based on a second fully connected layer to expand a number of dimensions of the clinical information feature vector before stitching the image feature vector and the clinical information feature vector to obtain a stitched vector.
In an embodiment of the present application, the number of dimensions of the image feature vector is 1024 dimensions, and the number of dimensions of the clinical information feature vector after expansion is 128 dimensions.
In an embodiment of the present application, the image feature extraction model is an image detection model, where the image detection model includes: a first convolutional neural network configured to output a first feature map based on the medical image, wherein the first convolutional neural network comprises at least one first convolutional layer and at least one first pooling layer; and a region-of-interest pooling layer configured to output a region-of-interest feature vector as the image feature vector based on a region of interest in the first feature map.
In an embodiment of the present application, the image feature extraction model is an image classification model, where the image classification model includes: a second convolutional neural network configured to output the image feature vector based on the medical image.
In an embodiment of the present application, the clinical information feature vector obtaining module is further configured to:
and respectively encoding a plurality of different discrete state information of one clinical information in the clinical text information into different vector values on a single dimension corresponding to the clinical information.
In an embodiment of the present application, the clinical information feature vector obtaining module is further configured to:
unifying quantitative values of one clinical information in the clinical text information; and
and a vector value on a dimension corresponding to the clinical information in the clinical information feature vector is (the quantized value-the lowest quantized value of the clinical information)/the maximum quantized value of the clinical information.
In an embodiment of the present application, the clinical information feature vector obtaining module is further configured to:
and encoding each discrete state information in a plurality of different discrete state information of one clinical information in the clinical text information into vector values on a plurality of dimensions corresponding to the clinical information, wherein the discrete state information is uniquely corresponding to one vector value combination mode of the plurality of dimensions.
According to another aspect of the present application, an embodiment of the present application provides an electronic device, including: a processor; a memory; and computer program instructions stored in the memory, which when executed by the processor, cause the processor to perform the medical information processing method as in any one of the above.
According to another aspect of the present application, an embodiment of the present application provides a computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the medical information processing method according to any one of the preceding claims.
According to another aspect of the present application, an embodiment of the present application provides a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the medical information processing method as described in any one of the above.
According to the medical information processing method, the medical information processing device, the electronic equipment and the computer-readable storage medium, the image feature vector in the medical image is extracted through the image feature extraction model, the clinical text information possibly recorded with non-medical image information such as a patient medical history record is converted into the clinical information feature vector, and the image feature vector and the clinical information feature vector are spliced in a vector splicing mode to obtain the spliced vector. Therefore, the splicing vector comprises the characteristic information of the medical image and the characteristic information of the clinical text information, and the classification result integrating the medical image and the clinical text information can be obtained as medical analysis information by classifying through the first full-connection layer based on the splicing vector. Compared with the method for obtaining the medical analysis information by simply depending on the medical image, the medical analysis information obtained by the embodiment of the application is more comprehensive and accurate.
Drawings
Fig. 1 is a schematic flowchart illustrating a medical information processing method according to an embodiment of the present application.
Fig. 2 is a flow chart illustrating a medical information processing method according to another embodiment of the present application.
Fig. 3 is a flow chart illustrating a medical information processing method according to another embodiment of the present application.
Fig. 4 is a flowchart illustrating a medical information processing method according to another embodiment of the present application.
Fig. 5 is a schematic structural diagram of a medical information processing apparatus according to an embodiment of the present application.
Fig. 6 is a schematic structural diagram of a medical information processing apparatus according to another embodiment of the present application.
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Summary of the application
As described above, although the conventional medical information processing method can obtain medical analysis information depending on medical image information, the obtained medical analysis information is not comprehensive and accurate because non-medical image information such as medical history of a patient cannot be referred to. In view of the above technical problems, the present application provides a medical information processing method, which is expected to integrate an analysis process of clinical text information while analyzing a medical image. Because the data volume of the clinical text information is usually small, the speed of medical image analysis cannot be influenced by increasing the clinical text information, and the accuracy of the obtained medical analysis information can be effectively improved. Specifically, not only are image feature vectors in medical images extracted through an image feature extraction model, but also clinical text information which possibly records non-medical image information such as medical history records of patients is converted into clinical information feature vectors, and classification results integrating the medical images and the clinical text information are obtained through a vector splicing mode to serve as medical analysis information. Compared with the method for obtaining the medical analysis information by simply depending on the medical image, the medical analysis information obtained by the embodiment of the application is more comprehensive and accurate.
It should be understood that the medical information processing method provided by the embodiment of the application is applicable to any medical information analysis scene with medical images participating. The content of the medical image itself can be adjusted according to the actual application scene requirements, for example, the transmission image in the fracture examination scene of orthopedics department or the transmission image in the breast gland examination scene of gynecology department. The specific medical information analysis scenario to which the medical information processing method is applied is not strictly limited.
Having described the general principles of the present application, various non-limiting embodiments of the present application will now be described with reference to the accompanying drawings.
Exemplary medical information processing method
Fig. 1 is a schematic flowchart illustrating a medical information processing method according to an embodiment of the present application. As shown in fig. 1, the medical information processing method includes the steps of:
step 101: inputting the medical image into an image feature extraction model to obtain an image feature vector output by the image feature extraction model, wherein the image feature extraction model is a convolution neural network model established through a training process.
The image feature vector can be regarded as a point in a high-dimensional space, and the value of the point in each dimension can be regarded as a feature for representing one aspect of the medical image, and the values in all dimensions are integrated to be used for representing the whole or part of the medical image.
The image feature extraction model is an artificial intelligence model for obtaining image feature vectors based on medical images, and the image feature extraction model can be established through a pre-training process by adopting a convolutional neural network architecture (for example, a Faster regional convolutional neural network (fastr R-CNN) model). For example, medical image samples may be input into the convolutional neural network to obtain output image feature vector samples, and the image feature vector samples are compared with standard image feature vectors to calculate the loss of the model, and continuously and iteratively optimized in a back propagation manner until the regression accuracy is satisfied. After a large number of medical image samples are trained, the convolutional neural network can have the capability of acquiring image feature vectors based on medical images. It should be understood that the specific category and architecture of the image feature extraction model may be adjusted according to the requirements of the actual application scenario, and the specific category and architecture of the image feature extraction model is not strictly limited in the present application.
Step 102: the clinical text information is converted into a clinical information feature vector through an encoding process, wherein a vector value in each dimension of the clinical information feature vector is used for representing one type of clinical information in the clinical text information.
The clinical text information is the text information recorded with the non-medical image information such as the personal information, the medical history, the medical record and the like of the patient. Clinical text information can be directly obtained by a third party or converted by information in other media forms. The content of the clinical information in the clinical text information can be represented in the form of codes through the encoding process, and the values of the codes can be regarded as the values of one point in a high-dimensional space on one dimension, so that the values of all the codes correspond to the clinical information feature vector in the high-dimensional space. It should be understood that the number of dimensions of the clinical information feature vector and the value of the clinical information feature vector in each dimension may be adjusted according to the content of the clinical text information and the specific encoding manner, and the specific number of dimensions of the clinical information feature vector and the encoding value manner of each dimension are not strictly limited in the present application.
It should also be understood that the content of the clinical text information can be adjusted according to the actual medical analysis scene, for example, in the fracture examination scene of orthopedics department, the content of the clinical text information can include the sex, age, chest pain examination result, rib fracture review result, lung nodule review result, etc. The specific content of the clinical text information is not strictly limited in the present application.
Step 103: and splicing the image feature vector and the clinical information feature vector to obtain a spliced vector.
The image feature vector is used for representing the medical image, the clinical information feature vector is used for representing the clinical text information, and the image feature vector and the clinical information feature vector are spliced to synthesize the features of the medical image and the features of the clinical text information so as to provide more comprehensive data support for subsequently acquiring medical analysis information.
Step 104: the spliced vectors are processed based on the first full link layer to obtain classification results.
The first fully-connected layer may be implemented by a convolution operation that maps the learned "distributed feature representation" to the sample label space to act as a "classifier" throughout the convolutional neural network. The spliced vectors are processed by the first full-connection layer to obtain a classification result, and the content of the classification result is medical analysis information.
Therefore, according to the medical information processing method provided by the embodiment of the application, the image feature vector in the medical image is extracted through the image feature extraction model, the clinical text information which possibly records non-medical image information such as a medical history and a medical history of a patient is converted into the clinical information feature vector, and the image feature vector and the clinical information feature vector are spliced in a vector splicing mode to obtain the spliced vector. Therefore, the splicing vector comprises the characteristic information of the medical image and the characteristic information of the clinical text information, and the classification result integrating the medical image and the clinical text information can be obtained as medical analysis information by classifying through the first full-connection layer based on the splicing vector. Compared with the method for obtaining the medical analysis information by simply depending on the medical image, the medical analysis information obtained by the embodiment of the application is more comprehensive and accurate.
In an embodiment of the present application, as shown in fig. 2, the number of dimensions of the obtained clinical information feature vector may also be low in consideration of the limited data content of the clinical text information. If the resolution of the medical image itself is high, the number of dimensions of the obtained image feature vectors may be high. However, if the dimension number of the clinical information feature vector is too much different from the dimension number of the image feature vector, the influence of the clinical information feature vector on the final classification result is too small. Therefore, in order to balance the dimensional quantitative relationship between the clinical information feature vector and the image feature vector so that the clinical information feature vector has a substantial influence on the final classification result, before the image feature vector and the clinical information feature vector are spliced to obtain a spliced vector, the medical information processing method may further include:
step 201: the clinical information feature vector is processed based on the second fully-connected layer to extend a number of dimensions of the clinical information feature vector. The second fully-connected layer may be implemented by convolution operations, and the number of dimensions of the clinical information feature vector may be expanded by inputting the clinical information feature vector into the second fully-connected layer. For example, the number of dimensions of the image feature vector is 1024 dimensions, and the expanded number of dimensions of the clinical information feature vector with 64 dimensions may be 128 dimensions.
In an embodiment of the present application, the image feature extraction model is an image detection model, which means that the image feature vectors output by the image feature extraction model may not be used to characterize the entire medical image, but are regions of interest in the medical image, for example, in a breast tumor examination scene, a breast tumor region needs to be detected in the breast image and characterized by using the image feature vectors. Specifically, as shown in fig. 3, the process of inputting the medical image into the image feature extraction model to obtain the image feature vector output by the image feature extraction model may specifically include:
step 1011: the medical image is input into a first convolutional neural network to obtain a first feature map, wherein the first convolutional neural network comprises at least one first convolutional layer and at least one first pooling layer.
Step 1012: and inputting the region of interest in the first feature map into the region of interest pooling layer to obtain a region of interest feature vector output by the region of interest pooling layer as an image feature vector.
And then splicing the characteristic vector of the region of interest and the clinical information characteristic vector to obtain a spliced vector, and classifying through a first full-connection layer based on the spliced vector to obtain final medical analysis information.
In another embodiment of the present application, the image feature extraction model is an image classification model, which omits the extraction process of the region of interest compared to the aforementioned image detection model. At this time, the process of inputting the medical image into the image feature extraction model to obtain the image feature vector output by the image feature extraction model may include: the medical image is input into a second convolutional neural network to obtain an image feature vector.
In an embodiment of the present application, in order to further improve accuracy and efficiency of an encoding process of a clinical information feature vector, a corresponding encoding method may be determined according to a category of clinical information in clinical text information, and the clinical information in the clinical text information is encoded in the corresponding encoding method. Specifically, as shown in fig. 4, according to the clinical text information in different categories of clinical information, the following three different encoding methods can be adopted:
step 1021: one clinical information in the clinical text information includes a plurality of mutually exclusive discrete values, which are encoded as different vector values in a single dimension corresponding to the clinical information, respectively.
For example, the clinical text information includes clinical information of the sex of the patient, and since the sex of the patient is either male or female, the two values are discrete values and mutually exclusive, at this time, the vector value of female may be encoded as 0, and the vector value of male may be encoded as 1.
Step 1022: one clinical information in the clinical text information is represented by a quantitative value, and the quantitative value of one clinical information in the clinical text information is unified; and the vector value on the dimension corresponding to the clinical information in the clinical information characteristic vector is (quantitative value-the lowest quantitative value of the clinical information)/the maximum quantitative value of the clinical information.
For example, the clinical text information includes clinical information such as patient age, duration of symptom occurrence, and the like, and the clinical information is characterized by quantitative value of time. For convenience of encoding, the quantization values of the clinical information are unified into a unit, for example, when the clinical information is a symptom, the specific quantization values can be unified into a unit of day. Taking the symptom appearing time length of 5.5 days as an example, the lowest quantitative value of the clinical information may be the lowest value of the symptom appearing time length of 1 day in the clinical case record, the maximum quantitative value of the clinical information may be the maximum value of the symptom appearing time length of 8 days in the clinical case record, and then the vector value on the dimension corresponding to the quantitative value of the symptom appearing time length is (5.5-1)/8 is 0.5625.
In another embodiment of the present application, the encoding process may be implemented by a convolutional neural network, which is established by a pre-training process. At this time, the lowest quantization value and the highest quantization value of the clinical information may also be the minimum value and the maximum value in the training sample of the convolutional neural network corresponding to the clinical information.
Step 1023: one type of clinical information in the clinical text information comprises a plurality of types of different state information, and each type of state information in the plurality of types of different state information is encoded into vector values in a plurality of dimensions corresponding to the clinical information, wherein the state information and one type of vector value combination mode of the plurality of dimensions only correspond to each other.
For example, the clinical text information includes clinical information of the relevant medical history and the occurrence of symptoms, etc., and at this time, the encoding process may be performed according to the disease category to be detected and classified. For example, for a patient to be subjected to a fracture examination, reference is made to the reason, i.e., the medical history, of the previous transmission image examination of that type of patient. The medical history obtained for the fracture examination generally includes three medical history forms, chest pain caused by car accidents, rib fracture review and pulmonary nodule review. These three forms of medical history can be categorized into the following three types of status information: the three kinds of state information can be coded into three vector value combination modes of (1, 0, 0), (0, 1, 0) and (0, 0, 1) respectively. That is, each form in the history corresponding to the fracture examination corresponds to a unique vector value combination mode, and each vector value combination mode corresponds to three vector values in three dimensions.
It should be understood that, although three corresponding encoding manners determined according to the category of the clinical information in the clinical text information are given above, in other embodiments of the present application, the encoding manner for obtaining the clinical information feature vector based on the clinical text information may also be implemented by a plurality of manners, such as word embedding, RNN (Recurrent Neural Network), and the present application does not limit the specific mechanism of the encoding manner.
In order to verify the effect of the medical information processing method provided by the embodiment of the application, 500 patients are selected for training and 200 patients are selected for testing by using the collected fracture detection data. The types of training samples comprise fresh fracture, fracture in healing period and old fracture, and the sample amount is respectively as follows: 16352, 9604, and 1103. the medical images used for training are CT (Computed Tomography) images, and the results of the test set comparisons are shown in the following table:
table 1 shows the test results of the medical information processing method provided in the embodiment of the present application
Table 2 shows the test results of medical analysis information obtained only by CT image
| Sensitivity (recall) | Specificity (accuracy) | F1 value | |
| Fresh fracture | 0.8868 | 0.6912 | 0.7769 |
| Fracture at healing stage | 0.8268 | 0.8457 | 0.8362 |
| Old fracture | 0.1711 | 0.9286 | 0.2889 |
As can be seen from the two tables, after the analysis of clinical text information is added and the fusion of the feature vectors is carried out, the prediction results of the three fractures are improved to different degrees, and the old fracture is most obvious. The result shows that the analysis of the added clinical text information can greatly make up the deficiency of the data volume and improve the accuracy of the obtained medical analysis information.
Exemplary medical information processing apparatus
Fig. 5 is a schematic structural diagram of a medical information processing apparatus according to an embodiment of the present application. As shown in fig. 5, the medical information processing apparatus 50 includes:
an image feature vector obtaining module 501, configured to input the medical image into an image feature extraction model to obtain an image feature vector output by the image feature extraction model, where the image feature extraction model is a convolutional neural network model established through a training process;
a clinical information feature vector obtaining module 502 configured to convert clinical text information into a clinical information feature vector through an encoding process, wherein a vector value in each dimension of the clinical information feature vector is used for representing one type of clinical information in the clinical text information;
a stitching module 503 configured to stitch the image feature vector and the clinical information feature vector to obtain a stitched vector; and
a classification module 504 configured to process the stitched vector based on the first fully connected layer to obtain a classification result.
In an embodiment of the present application, as shown in fig. 6, the medical information processing apparatus 50 further includes:
a dimension expansion module 505 configured to process the clinical information feature vector based on the second fully connected layer to expand the number of dimensions of the clinical information feature vector before stitching the image feature vector and the clinical information feature vector to obtain a stitched vector.
In an embodiment of the present application, the number of dimensions of the image feature vector is 1024 dimensions, and the number of dimensions of the clinical information feature vector after expansion is 128 dimensions.
In an embodiment of the present application, the image feature extraction model is an image detection model, where the image detection model includes:
a first convolutional neural network configured to output a first feature map based on the medical image, wherein the first convolutional neural network comprises at least one first convolutional layer and at least one first pooling layer; and
and the region-of-interest pooling layer is configured to output the region-of-interest feature vector as an image feature vector based on the region of interest in the first feature map.
In an embodiment of the present application, the image feature extraction model is an image classification model, where the image classification model includes: a second convolutional neural network configured to output an image feature vector based on the medical image.
In an embodiment of the present application, the clinical information feature vector obtaining module 502 is further configured to:
and respectively encoding a plurality of different discrete state information of one clinical information in the clinical text information into different vector values on a single dimension corresponding to the clinical information.
In an embodiment of the present application, the clinical information feature vector obtaining module 502 is further configured to:
unifying quantitative values of one clinical information in the clinical text information; and
the vector value of the clinical information feature vector in the dimension corresponding to the clinical information is (quantitative value-the lowest quantitative value of the clinical information)/the maximum quantitative value of the clinical information.
In an embodiment of the present application, the clinical information feature vector obtaining module 502 is further configured to:
encoding each discrete state information in a plurality of different discrete state information of one clinical information in the clinical text information into vector values on a plurality of dimensions corresponding to the clinical information, wherein the discrete state information uniquely corresponds to one vector value combination mode of the plurality of dimensions.
The detailed functions and operations of the respective modules in the medical information processing apparatus 50 described above have been described in detail in the medical information processing method described above with reference to fig. 1 to 4, and therefore, a repetitive description thereof will be omitted here.
It should be noted that the medical information processing apparatus 50 according to the embodiment of the present application may be integrated into the electronic device 60 as a software module and/or a hardware module, in other words, the electronic device 60 may include the medical information processing apparatus 50. For example, the medical information processing apparatus 50 may be a software module in the operating system of the electronic device 60, or may be an application developed therefor; of course, the medical information processing apparatus 50 may also be one of many hardware modules of the electronic device 60.
In another embodiment of the present application, the medical information processing apparatus 50 and the electronic device 60 may also be separate devices (e.g., servers), and the medical information processing apparatus 50 may be connected to the electronic device 60 through a wired and/or wireless network and transmit the interactive information according to an agreed data format.
Exemplary electronic device
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 7, the electronic device 60 includes: one or more processors 601 and memory 602; and computer program instructions stored in the memory 602, which, when executed by the processor 601, cause the processor 601 to perform the medical information processing method according to any of the embodiments described above.
The processor 601 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
The memory 602 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, Random Access Memory (RAM), cache memory (or the like). The non-volatile memory may include, for example, Read Only Memory (ROM), a hard disk, flash memory, and the like. One or more computer program instructions may be stored on a computer-readable storage medium and executed by the processor 601 to implement the steps of the medical information processing method of the various embodiments of the present application described above and/or other desired functions. Information such as light intensity, compensation light intensity, position of filters, etc. may also be stored in the computer readable storage medium.
In one example, the electronic device 60 may further include: an input device 603 and an output device 604, which are interconnected by a bus system and/or other form of connection mechanism (not shown in fig. 7).
For example, when the electronic device is a robot in an industrial production line, the input device 603 may be a camera for capturing the position of the part to be processed. When the electronic device is a stand-alone device, the input means 603 may be a communication network connector for receiving the acquired input signal from an external removable device. Further, the input device 603 may include, for example, a keyboard, a mouse, a microphone, and the like.
The output device 604 may output various information to the outside, and may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for the sake of simplicity, only some of the components of the electronic apparatus 60 relevant to the present application are shown in fig. 7, and components such as a bus, an input device/output interface, and the like are omitted. In addition, electronic device 60 may include any other suitable components depending on the particular application.
Exemplary computer program product and computer-readable storage Medium
In addition to the above-described methods and apparatuses, embodiments of the present application may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps of the medical information processing method according to any of the above-described embodiments.
The computer program product may include program code for carrying out operations for embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps in the medical information processing method according to various embodiments of the present application described in the "exemplary medical information processing method" section above in this specification.
A computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory ((RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are exemplary only and not limiting, and should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to limit the application to the details which may be employed.
The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. As used herein, the words "or" and "refer to, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations should be considered equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modifications, equivalents and the like that are within the spirit and principle of the present application should be included in the scope of the present application.
Claims (12)
1. A medical information processing method characterized by comprising:
inputting a medical image into an image feature extraction model to obtain an image feature vector output by the image feature extraction model, wherein the image feature extraction model is a convolutional neural network model established through a training process;
converting clinical text information into a clinical information feature vector through a coding process, wherein a vector value on each dimension of the clinical information feature vector is used for representing one type of clinical information in the clinical text information;
splicing the image feature vector and the clinical information feature vector to obtain a spliced vector; and
and processing the splicing vector based on the first full-connection layer to obtain a classification result.
2. The method of claim 1, wherein prior to stitching the image feature vector and the clinical information feature vector to obtain a stitched vector, the method further comprises:
processing the clinical information feature vector based on a second fully connected layer to extend a number of dimensions of the clinical information feature vector.
3. The method of claim 2, wherein the image feature vector has 1024 dimensions, and the expanded clinical information feature vector has 128 dimensions.
4. The method according to claim 1, wherein the image feature extraction model is an image detection model, and wherein the inputting the medical image into the image feature extraction model to obtain the image feature vector output by the image feature extraction model comprises:
inputting a medical image into a first convolutional neural network to obtain a first feature map, wherein the first convolutional neural network comprises at least one first convolutional layer and at least one first pooling layer; and
inputting the region of interest in the first feature map into a region of interest pooling layer to obtain a region of interest feature vector output by the region of interest pooling layer as the image feature vector.
5. The method according to claim 1, wherein the image feature extraction model is an image classification model, and wherein the inputting the medical image into the image feature extraction model to obtain the image feature vector output by the image feature extraction model comprises:
inputting the medical image into a second convolutional neural network to obtain the image feature vector.
6. The method of claim 1, wherein transforming the clinical textual information into a clinical information feature vector through an encoding process comprises:
and determining a corresponding coding mode according to the category of the clinical information in the clinical text information, and coding the clinical information in the clinical text information in the corresponding coding mode.
7. The method according to claim 6, wherein one of the clinical text messages includes a plurality of mutually exclusive discrete values, and wherein the encoding the clinical message in the corresponding encoding manner includes:
encoding the mutually exclusive discrete values as different vector values in a single dimension corresponding to the clinical information, respectively.
8. The method according to claim 6, wherein one of the clinical text messages is characterized by a quantitative value, and wherein the encoding the clinical information in the clinical text message in the corresponding encoding manner comprises:
unifying quantitative values of one clinical information in the clinical text information; and
and a vector value on a dimension corresponding to the clinical information in the clinical information feature vector is (the quantized value-the lowest quantized value of the clinical information)/the maximum quantized value of the clinical information.
9. The method of claim 6, wherein one of the clinical text messages comprises a plurality of different status messages, and wherein the encoding the clinical message in the corresponding encoding manner comprises:
and encoding each of the plurality of different types of status information into a plurality of dimensions of vector values corresponding to the clinical information, wherein the status information uniquely corresponds to one of the plurality of dimensions in a vector value combination manner.
10. A medical information processing apparatus characterized by comprising:
the image feature vector acquisition module is configured to input the medical image into an image feature extraction model to acquire an image feature vector output by the image feature extraction model, wherein the image feature extraction model is a convolutional neural network model established through a training process;
the clinical information feature vector acquisition module is configured to convert clinical text information into a clinical information feature vector through a coding process, wherein a vector value in each dimension of the clinical information feature vector is used for representing one type of clinical information in the clinical text information;
a stitching module configured to stitch the image feature vector and the clinical information feature vector to obtain a stitched vector; and
and the classification module is configured to process the splicing vector based on the first full-link layer to obtain a classification result.
11. A computer-readable storage medium characterized in that the storage medium stores a computer program for executing the medical information processing method according to any one of claims 1 to 9.
12. An electronic device, characterized in that the electronic device comprises:
a processor;
a memory for storing the processor-executable instructions;
the processor for executing the medical information processing method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911135115.0A CN111008957A (en) | 2019-11-19 | 2019-11-19 | Medical information processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911135115.0A CN111008957A (en) | 2019-11-19 | 2019-11-19 | Medical information processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111008957A true CN111008957A (en) | 2020-04-14 |
Family
ID=70112917
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911135115.0A Pending CN111008957A (en) | 2019-11-19 | 2019-11-19 | Medical information processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111008957A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111898338A (en) * | 2020-07-15 | 2020-11-06 | 北京字节跳动网络技术有限公司 | Text generation method and device and electronic equipment |
| CN111951252A (en) * | 2020-08-17 | 2020-11-17 | 中国科学院苏州生物医学工程技术研究所 | Multi-sequence image processing method, electronic device and storage medium |
| CN112133441A (en) * | 2020-08-21 | 2020-12-25 | 广东省人民医院 | Establishment method and terminal of MH post-operation fissure hole state prediction model |
| CN112466436A (en) * | 2020-11-25 | 2021-03-09 | 北京小白世纪网络科技有限公司 | Intelligent traditional Chinese medicine evolution model training method and device based on recurrent neural network |
| CN112489043A (en) * | 2020-12-21 | 2021-03-12 | 无锡祥生医疗科技股份有限公司 | Heart disease detection device, model training method, and storage medium |
| CN113499033A (en) * | 2021-05-20 | 2021-10-15 | 北京鹰瞳科技发展股份有限公司 | Medical data method and system |
| WO2021236702A1 (en) | 2020-05-20 | 2021-11-25 | F. Hoffmann-La Roche Ag | Intelligent workflow analysis for treatments using exposable cloud-based registries |
| CN113850753A (en) * | 2021-08-17 | 2021-12-28 | 苏州鸿熙融合智能医疗科技有限公司 | Medical image information calculation method and device, edge calculation equipment and storage medium |
| CN111898338B (en) * | 2020-07-15 | 2024-04-30 | 北京字节跳动网络技术有限公司 | Text generation method and device and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109919928A (en) * | 2019-03-06 | 2019-06-21 | 腾讯科技（深圳）有限公司 | Detection method, device and the storage medium of medical image |
| CN109978022A (en) * | 2019-03-08 | 2019-07-05 | 腾讯科技（深圳）有限公司 | A kind of medical treatment text message processing method and device, storage medium |
| CN110033019A (en) * | 2019-03-06 | 2019-07-19 | 腾讯科技（深圳）有限公司 | Method for detecting abnormality, device and the storage medium of human body |
| US10361802B1 (en) * | 1999-02-01 | 2019-07-23 | Blanding Hovenweep, Llc | Adaptive pattern recognition based control system and method |
| CN110246109A (en) * | 2019-05-15 | 2019-09-17 | 清华大学 | Merge analysis system, method, apparatus and the medium of CT images and customized information |
-
2019
- 2019-11-19 CN CN201911135115.0A patent/CN111008957A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10361802B1 (en) * | 1999-02-01 | 2019-07-23 | Blanding Hovenweep, Llc | Adaptive pattern recognition based control system and method |
| CN109919928A (en) * | 2019-03-06 | 2019-06-21 | 腾讯科技（深圳）有限公司 | Detection method, device and the storage medium of medical image |
| CN110033019A (en) * | 2019-03-06 | 2019-07-19 | 腾讯科技（深圳）有限公司 | Method for detecting abnormality, device and the storage medium of human body |
| CN109978022A (en) * | 2019-03-08 | 2019-07-05 | 腾讯科技（深圳）有限公司 | A kind of medical treatment text message processing method and device, storage medium |
| CN110246109A (en) * | 2019-05-15 | 2019-09-17 | 清华大学 | Merge analysis system, method, apparatus and the medium of CT images and customized information |
Non-Patent Citations (2)
| Title |
|---|
| 于玉海: "跨模态多标签生物医学图像分类建模识别", 《中国图象图形学报》 * |
| 吴迪: "《智能环境下基于音视频多模态融合的身份识别》", 31 March 2018 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021236702A1 (en) | 2020-05-20 | 2021-11-25 | F. Hoffmann-La Roche Ag | Intelligent workflow analysis for treatments using exposable cloud-based registries |
| CN111898338A (en) * | 2020-07-15 | 2020-11-06 | 北京字节跳动网络技术有限公司 | Text generation method and device and electronic equipment |
| CN111898338B (en) * | 2020-07-15 | 2024-04-30 | 北京字节跳动网络技术有限公司 | Text generation method and device and electronic equipment |
| CN111951252B (en) * | 2020-08-17 | 2024-01-23 | 中国科学院苏州生物医学工程技术研究所 | Multi-time sequence image processing method, electronic equipment and storage medium |
| CN111951252A (en) * | 2020-08-17 | 2020-11-17 | 中国科学院苏州生物医学工程技术研究所 | Multi-sequence image processing method, electronic device and storage medium |
| CN112133441A (en) * | 2020-08-21 | 2020-12-25 | 广东省人民医院 | Establishment method and terminal of MH post-operation fissure hole state prediction model |
| CN112133441B (en) * | 2020-08-21 | 2024-05-03 | 广东省人民医院 | Method and terminal for establishing MH postoperative crack state prediction model |
| CN112466436A (en) * | 2020-11-25 | 2021-03-09 | 北京小白世纪网络科技有限公司 | Intelligent traditional Chinese medicine evolution model training method and device based on recurrent neural network |
| CN112466436B (en) * | 2020-11-25 | 2024-02-23 | 北京小白世纪网络科技有限公司 | Intelligent traditional Chinese medicine prescription model training method and device based on cyclic neural network |
| CN112489043A (en) * | 2020-12-21 | 2021-03-12 | 无锡祥生医疗科技股份有限公司 | Heart disease detection device, model training method, and storage medium |
| CN113499033A (en) * | 2021-05-20 | 2021-10-15 | 北京鹰瞳科技发展股份有限公司 | Medical data method and system |
| WO2023020366A1 (en) * | 2021-08-17 | 2023-02-23 | 苏州鸿熙融合智能医疗科技有限公司 | Medical image information computing method and apparatus, edge computing device, and storage medium |
| CN113850753B (en) * | 2021-08-17 | 2023-09-01 | 苏州鸿熙融合智能医疗科技有限公司 | Medical image information computing method, device, edge computing equipment and storage medium |
| CN113850753A (en) * | 2021-08-17 | 2021-12-28 | 苏州鸿熙融合智能医疗科技有限公司 | Medical image information calculation method and device, edge calculation equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111008957A (en) | Medical information processing method and device | |
| US11423538B2 (en) | Computer-implemented machine learning for detection and statistical analysis of errors by healthcare providers | |
| CN109741806B (en) | Auxiliary generation method and device for medical image diagnosis report | |
| AU2020260078B2 (en) | Computer-implemented machine learning for detection and statistical analysis of errors by healthcare providers | |
| CN111325745B (en) | Fracture region analysis method and device, electronic equipment and readable storage medium | |
| CN111316281A (en) | Semantic classification of numerical data in natural language context based on machine learning | |
| JP6885517B1 (en) | Diagnostic support device and model generation device | |
| CN110826908A (en) | Evaluation method and device for artificial intelligent prediction, storage medium and electronic equipment | |
| CN111564223B (en) | Infectious disease survival probability prediction method, and prediction model training method and device | |
| US11094059B2 (en) | Vulnerable plaque identification method, application server thereof, and computer readable medium | |
| KR102097743B1 (en) | Apparatus and Method for analyzing disease based on artificial intelligence | |
| CN111915584A (en) | Focus follow-up assessment method and system based on CT (computed tomography) image | |
| Strzelecki et al. | Machine learning for biomedical application | |
| CN112529149A (en) | Data processing method and related device | |
| CN110570425A (en) | Lung nodule analysis method and device based on deep reinforcement learning algorithm | |
| CN111226287A (en) | Method for analyzing a medical imaging dataset, system for analyzing a medical imaging dataset, computer program product and computer readable medium | |
| Gaggion et al. | Chexmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images | |
| CN111402213A (en) | Bone age evaluation method and device, electronic equipment and computer-readable storage medium | |
| Gupta et al. | Application of Convolutional Neural Networks for COVID-19 Detection in X-ray Images Using InceptionV3 and U-Net | |
| KR102403076B1 (en) | Sleep disorder inspecting apparatus and method thereof | |
| CN111915553A (en) | Part identification method and device based on time sequence modeling | |
| Fagbuagun et al. | Model development for pneumonia detection from chest radiograph using transfer learning | |
| CN110889836A (en) | Image data analysis method and device, terminal equipment and storage medium | |
| CN116612885B (en) | Prediction device for acute exacerbation of chronic obstructive pulmonary disease based on multiple modes | |
| CN117116476B (en) | Downstream task prediction method and device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room B401, floor 4, building 1, No. 12, Shangdi Information Road, Haidian District, Beijing 100085 Applicant after: Tuxiang Medical Technology Co., Ltd Address before: Room B401, floor 4, building 1, No. 12, Shangdi Information Road, Haidian District, Beijing 100085 Applicant before: Beijing Tuoxiang Technology Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200414 |