CN112396597A

CN112396597A - Method and device for rapidly screening unknown cause pneumonia images

Info

Publication number: CN112396597A
Application number: CN202011370420.0A
Authority: CN
Inventors: 朱华栋; 吴及; 李妍; 高键东; 孙岳川; 孙成章; 刘业成
Original assignee: Peking Union Medical College Hospital Chinese Academy of Medical Sciences
Current assignee: Tsinghua University; Peking Union Medical College Hospital Chinese Academy of Medical Sciences
Priority date: 2020-11-30
Filing date: 2020-11-30
Publication date: 2021-02-23

Abstract

The invention discloses a method and a device for rapidly screening unknown cause pneumonia images, wherein the method comprises the following steps: establishing an image analysis model by using a small quantity of acquired lung CT image sequences, CT data and related data thereof in advance; the relevant data includes any one or more of: medical history, assay index; acquiring patient lung CT image data and related data of the patient lung; performing fusion processing on the CT image data and related data thereof to obtain multidimensional characteristics; and inputting the multi-dimensional features into the image analysis model, and obtaining the lung disease category and the confidence thereof according to the output of the image analysis model. By using the invention, the early warning can be rapidly and accurately carried out on patients with viral pneumonia of unknown reason, and the diagnosis and treatment efficiency of pneumonia diseases is improved.

Description

Method and device for rapidly screening unknown cause pneumonia images

Technical Field

The invention relates to the field of image processing, in particular to a method and a device for rapidly screening unknown-cause pneumonia images.

Background

COVID-19 caused by SARS-COV-2 is currently abused worldwide, the number of the morbid people is rapidly increased, and in addition, early symptoms of the course of the disease are light and have mild cases, the early symptoms are difficult to diagnose and isolate in time, the difficulty in controlling the spread of the disease is high, the serious threat is caused to the health of people, and huge loss is caused to social economy. Multiple studies show that the chest imaging has high prediction value for diagnosing COVID-19 patients, and meanwhile, the early recognition of the pneumonia caused by unknown reasons and the timely early warning of the pneumonia caused by lung infection patients have great significance for preventing and treating new important infectious diseases.

Although the infectious disease prevention and control system is improved and the infectious disease diagnosis and treatment level is improved in recent years, the early warning system is still to be improved in the face of the outbreak of COVID-19. Emergency treatment and fever outpatient service are the first department of clinical care for epidemic situations, and rapid and accurate identification of patients with high-risk infectious diseases is urgently needed. The establishment of a rapid screening system for pneumonia of unknown reason is imminent, which is beneficial to the control of COVID-19 epidemic situation on one hand and the early prevention and treatment of new serious infectious diseases on the other hand.

Currently, the academic field has shown the results of using X-ray film or CT image data to determine the type and severity of pneumonia, but the data used in the current research is relatively long in disease course, and is not suitable for the actual situations of emergency treatment and fever clinic.

Disclosure of Invention

The invention provides a method and a device for rapidly screening unknown-cause pneumonia images, so that early warning can be rapidly and accurately carried out on patients with unknown-cause viral pneumonia, and the diagnosis and treatment efficiency of pneumonia diseases is improved.

Therefore, the invention provides the following technical scheme:

an image rapid screening method for pneumonia of unknown cause, which comprises the following steps:

establishing an image analysis model by using a small quantity of acquired lung CT image sequences, CT data and related data thereof in advance; the relevant data includes any one or more of: medical history, assay index;

acquiring CT image data of the lung of a patient and related data thereof;

performing fusion processing on the CT image data and the related data thereof so as to convert the CT image data and the related data into a uniform feature space and obtain multi-dimensional features;

and inputting the multi-dimensional features into the image analysis model, and obtaining the lung disease category and the confidence thereof according to the output of the image analysis model.

Optionally, the establishing an image analysis model by using the acquired small number of lung CT image sequences, CT data and related data thereof includes:

collecting a small number of lung CT image sequences, CT data and related data thereof, and labeling each CT image to obtain a label corresponding to each CT image;

performing enhancement processing on the CT data;

training by using the enhanced CT data and the label information of the CT image to obtain a baseline model;

performing fusion processing on the CT data and the related data to obtain multi-dimensional characteristics;

and adjusting the network structure of the baseline model according to the multidimensional characteristics to obtain an image analysis model.

Optionally, the enhancing the CT data includes:

the window level and the window width of the DICOM-format CT data are adjusted to achieve different degrees of contrast enhancement.

Optionally, the label corresponding to each CT image includes any one or more of the following: whether there are signs of infection, space occupation, pulmonary edema.

Optionally, the training to obtain the baseline model by using the enhanced CT data and the label information of the CT image includes:

selecting a plurality of image classification deep learning network structures as candidate model network structures;

and training the candidate model network structure based on the enhanced CT data and the label information of the CT image, and performing ensemble learning on the output probability of the candidate model in an equal-weight summation mode to obtain a baseline model.

Optionally, the network structure adjustment of the baseline model according to the fusion features to obtain an image analysis model includes:

and adjusting the network structure by using a neural network structure searching technology according to the fusion characteristics to obtain an image analysis model.

Optionally, the creating an image analysis model by using the acquired small number of lung CT image sequences and the related data thereof further includes:

performing quality evaluation on the label to obtain a quality score of the label;

determining a weight of the label in training the baseline model according to the quality score of the label; or

And when the quality score of the label is lower than a set threshold value, correcting the label according to the lung disease category and the credibility obtained every time in the process of training the baseline model.

Optionally, the fusing the related data includes:

and performing fusion processing on the related data by adopting a pre-fusion mode, a post-fusion mode or a slow fusion mode.

An apparatus for rapidly screening unknown cause pneumonia images, the apparatus comprising:

the analysis model establishing module is used for establishing an image analysis model by utilizing a small quantity of acquired lung CT image sequences, CT data and related data thereof in advance; the relevant data includes any one or more of: medical history, assay index;

the input information acquisition module is used for acquiring the lung CT image data of the patient and the related data thereof;

the data conversion module is used for carrying out fusion processing on the CT data and the related data thereof to obtain multidimensional characteristics;

and the screening module is used for inputting the multidimensional characteristics into the image analysis model and obtaining the lung disease category and the confidence coefficient thereof according to the output of the image analysis model.

Optionally, the analysis model building module includes:

the training data acquisition module is used for acquiring a small number of lung CT image sequences and related data thereof, and labeling each CT image to obtain a label corresponding to each CT image;

the data enhancement module is used for enhancing the CT data;

the baseline model training module is used for training by utilizing the enhanced CT data and the label information of the CT image to obtain a baseline model;

the data fusion processing module is used for carrying out fusion processing on the CT data and the related data to obtain multidimensional characteristics;

and the network structure adjusting module is used for adjusting the network structure of the baseline model according to the multidimensional characteristics to obtain an image analysis model.

Optionally, the data enhancement module is specifically configured to adjust a window level and a window width of the DICOM-formatted CT data to achieve different degrees of contrast enhancement.

Optionally, the baseline model training module comprises:

a network selection unit for selecting a plurality of image classification deep learning network structures as candidate model network structures;

and the integrated training unit is used for training the candidate model network structure based on the enhanced CT data and the label information of the CT image, and performing integrated learning on the output probability of the candidate model in an equal-weight summation mode to obtain a baseline model.

Optionally, the network structure adjusting module is specifically configured to perform network structure adjustment by using a neural network structure search technology according to the fusion feature, so as to obtain an image analysis model.

Optionally, the analysis model building module further includes: the label quality evaluation module, the label selection module and/or the label correction module;

the label quality evaluation module is used for carrying out quality evaluation on the label to obtain a quality score of the label;

the label selection module is used for determining the weight of the label in the process of training the baseline model according to the quality score of the label;

and the label correction module is used for correcting the label according to the lung disease category and the credibility obtained every time in the baseline model training process when the quality score of the label is lower than a set threshold value.

Optionally, the data conversion module is specifically configured to perform fusion processing on the related data in an early stage fusion, a later stage fusion, or a slow fusion mode.

According to the quick screening method and device for the unknown-cause pneumonia images, provided by the embodiment of the invention, a small number of lung CT image sequences, CT data and related data thereof are collected to establish an image analysis model; when screening images of unknown pneumonia, fusing lung CT image data of a patient to be screened and related data thereof to obtain multidimensional characteristics; and inputting the multidimensional characteristics into an image analysis model, and obtaining the lung disease category and the confidence thereof according to the output of the image analysis model. Because the establishment of the image analysis model not only considers CT image data, but also integrates other pathological information, such as assay indexes, patient symptoms and other information, the condition of an illness can be rapidly and accurately comprehensively judged even for a patient with a shorter course of treatment and less lung image consolidation in emergency treatment and fever outpatient service, and the accuracy rate and the model performance of judgment are effectively improved.

Furthermore, early warning suggestions can be given according to the lung disease category and the confidence level thereof output by the image analysis model, and the method is more suitable for the actual application requirements of emergency treatment and fever outpatient service.

Drawings

FIG. 1 is a flow chart of an embodiment of the present invention for creating an image analysis model;

FIG. 2 is a flowchart of a method for rapidly screening images of unknown cause pneumonia according to an embodiment of the present invention;

FIG. 3 is a block diagram of a fast screening apparatus for images of pneumonia of unknown cause according to an embodiment of the present invention;

fig. 4 is a block diagram of an analysis model building module according to an embodiment of the present invention.

Detailed Description

Aiming at the situations that most of emergency treatment and fever outpatient treatment are patients with short disease course, lung images are less in reality and are not suitable for the existing scheme of judging the type and severity of pneumonia by utilizing CT image data from a single source, the embodiment of the invention provides a rapid screening method and a rapid screening device for pneumonia images with unknown reasons.

First, the process of creating the image analysis model will be described in detail.

As shown in fig. 1, it is a flowchart of establishing an image analysis model in the embodiment of the present invention, and the flowchart includes the following steps:

step 101, a small number of lung CT image sequences, CT data and related data thereof are collected, and each CT image is labeled to obtain a label corresponding to each CT image.

Unlike conventional medical image labeling, in the embodiment of the present invention, there is no need to individually perform pixel-level labeling or bounding box labeling on the lesion, and only the entire CT image is labeled for the presence of signs of infection, space occupation, and pulmonary edema, that is, the label corresponding to each CT image includes, but is not limited to, any one or more of the following: whether there are signs of infection, space occupation, pulmonary edema.

This method of labeling not only greatly reduces the amount of labeling for the clinician, but also facilitates the incorporation of new data. Once a new major infectious disease occurs, the labeling method can quickly collect data for the model, reduce the time for training and deploying the model, and further meet the urgent need of epidemic situation prevention and control.

And 102, performing enhancement processing on the CT data.

As the medical image data has the great characteristic of small data volume, for a deep learning model, a large amount of data is more apt to be used for training the model so as to achieve the purposes of offsetting noise and retaining characteristics, and finally obtain better model performance. In order to overcome the difficulty of small data quantity, the scheme of the invention uses a data enhancement method, can effectively enhance the usability of data and improve the classification performance of the model.

Since acute pneumonia in early stages is less severe, it is necessary to retain all the effective information of the whole CT slice. In the image data enhancement method, in addition to the traditional rotation, turning, translation and the like, aiming at special data of lung CT, contrast enhancement in different degrees can be achieved by adjusting the window level and the window width of the original DICOM format data, the enhancement effect on different lung regions is simulated, a small amount of CT data is effectively expanded, and the improvement of the classification performance of the model is facilitated.

And 103, training by using the enhanced CT data and the label information of the CT image to obtain a baseline model.

Specifically, a plurality of image classification deep learning network structures (e.g., VGG, ResNet, inclusion, AlexNet, etc.) may be selected as candidate model network structures, then the candidate model network structures are trained based on the CT data after enhancement processing and the label information of the CT image, and the output probabilities of the candidate models are integrally learned in an equal-weight summation manner to obtain the baseline model.

And 104, performing fusion processing on the CT data and the related data to obtain multi-dimensional characteristics.

During modeling, various forms and types of data such as medical image data, medical records, assay indexes and the like are adopted. In general, different forms and types of data are not directly related to each other, and cannot be compared or operated on. In order to comprehensively utilize these data, it needs to be converted into a uniform feature space so that they can be directly compared or operated.

Specifically, a plurality of methods such as late fusion, early fusion, and slow fusion may be selected for performance comparison, and the feature fusion method with the best interpretability and performance may be selected within an acceptable computation amount range by comprehensively considering model performance, interpretability, and computation amount consumption.

And 105, adjusting the network structure of the baseline model according to the multi-dimensional characteristics to obtain an image analysis model.

Specifically, on the basis of the baseline model, factors such as improvement of model performance and interpretability by feature fusion processing are considered to design the structure of the image analysis model. In order to improve the deployment speed of the model, a neural network structure search technology can be used, and the network structure can be adjusted automatically by using a machine so that the obtained model can better fit with data.

Further, given the complexity of the patient's actual lung condition, the annotation information may exist that does not fully conform to the patient's condition development. In order to make the image analysis model have better robustness, a pure tag (or called as a high-quality tag) can be used as the input of the image analysis model, and a noise tag or an error tag can also be used as the normal input of the image analysis model, therefore, the model can be designed to have the capability of evaluating the quality of the tag, and the capability of improving the quality of a part of the tag by using the whole data information.

Specifically, the influence of the selected learning, label reestimation, diffusion and other technologies on the model performance can be overcome, and the label quality can be improved. Such as:

adding a label quality evaluation model in the image analysis model for carrying out quality evaluation on the label to obtain a quality score of the label; determining a weight of the label in training the baseline model according to the quality score of the label; or when the quality score of the label is lower than a set threshold value, correcting the label according to the lung disease category and the credibility obtained every time in the process of training the baseline model.

Through the evaluation of the label quality, the defective or error label information can be better utilized, and the label quality can be improved through the adjustment of the model.

Referring to fig. 2, a flowchart of a method for rapidly screening an image of an unknown cause pneumonia according to an embodiment of the present invention includes the following steps:

step 201, an image analysis model is established in advance by using a small number of acquired lung CT image sequences, CT data and related data thereof.

The related data can be in various forms and types such as medical records, assay indexes and the like.

Step 202, acquiring lung CT image data to be screened and related data thereof.

Step 203, performing fusion processing on the CT image data and the related data to obtain a multidimensional feature.

The purpose of the fusion processing is to convert the CT image data, medical history, assay indexes and other data in various forms and types into a uniform feature space. Specifically, feature fusion methods such as late fusion, early fusion, and slow fusion may be selected, and the embodiment of the present invention is not limited thereto.

Step 204, inputting the multidimensional characteristics into the image analysis model, and obtaining the lung condition category and the confidence thereof according to the output of the image analysis model.

Specifically, the output of the image analysis model may include a plurality of categories of medical conditions and a confidence level for each category. The disease category corresponds to a label in training the model, and includes: whether infection, space occupation, pulmonary edema signs and the like exist or not can be determined, and the method can be classified into more detailed categories without limitation. The confidence level for each category may be represented by a probability value for that category.

By using the method provided by the embodiment of the invention, the overall judgment on the disease condition can be rapidly and accurately carried out, and effective prompt can be given to a clinician especially for patients with shorter disease course and less lung image consolidation in emergency treatment and fever outpatient service. Furthermore, the output result of the image analysis model can be divided into a high region, a middle region and a low region according to the probability to prompt the credibility of the image analysis result for doctors, for example, for the analysis result with low confidence, more intervention evaluation can be performed by doctors, so that the accuracy of patient diagnosis is further improved in a man-machine cooperation mode.

Correspondingly, an embodiment of the present invention further provides a rapid screening device for images of pneumonia of unknown cause, as shown in fig. 3, which is a structural block diagram of the device.

In this embodiment, the apparatus includes the following modules:

an analysis model establishing module 301, configured to establish an image analysis model by using a small number of acquired lung CT image sequences, CT data, and related data thereof in advance; the relevant data includes any one or more of: medical history, assay index;

an input information obtaining module 302, configured to obtain patient lung CT image data and related data thereof;

a data conversion module 303, configured to perform fusion processing on the CT data and related data thereof to obtain a multidimensional feature; for example, the related data is fused in an early stage fusion mode, a later stage fusion mode or a slow fusion mode;

the screening module 304 is configured to input the multidimensional features into the image analysis model, and obtain a lung condition category and a confidence thereof according to an output of the image analysis model.

As shown in fig. 4, the structural block diagram of the analysis model building module in the embodiment of the present invention includes the following modules:

a training data acquisition module 411, configured to acquire a small number of lung CT image sequences and related data thereof, and label each CT image to obtain a label corresponding to each CT image;

a data enhancement module 412, configured to perform enhancement processing on the CT data;

a baseline model training module 413, configured to train to obtain a baseline model by using the enhanced CT data and the label information of the CT image;

a data fusion processing module 414, configured to perform fusion processing on the CT data and the related data to obtain a multidimensional feature; for example, processing modes such as late fusion, early fusion, slow fusion, and the like may be selected, and the embodiment of the present invention is not limited thereto.

And a network structure adjusting module 415, configured to perform network structure adjustment on the baseline model according to the multidimensional features to obtain an image analysis model.

In the embodiment of the present invention, the related data may be various forms and types of data such as medical records, assay indexes, and the like. Labels for each CT image include, but are not limited to, any one or more of the following: whether there are signs of infection, space occupation, pulmonary edema.

As the medical image data has the great characteristic of small data volume, for a deep learning model, a large amount of data is more apt to be used for training the model so as to achieve the purposes of offsetting noise and retaining characteristics, and finally obtain better model performance. In order to overcome the difficulty of small data quantity, the scheme of the invention uses a data enhancement method, can effectively enhance the usability of data and improve the classification performance of the model. Specifically, the data enhancement module 412 may adjust the window level and the window width of the DICOM-formatted CT data to achieve different degrees of contrast enhancement.

The baseline model training module 413 may specifically select a plurality of image classification deep learning network structures (e.g., VGG, ResNet, inclusion, AlexNet, etc.) as candidate model network structures, respectively train each candidate model network structure based on the enhanced CT data and the label information of the CT image, and synthesize the plurality of image classification deep learning network structures by using an equal-weight summation method to obtain a baseline model. Accordingly, one specific structure of the baseline model training module 413 may include the following units:

In the embodiment of the present invention, the network structure adjusting module 415 may design the structure of the image analysis model based on the baseline model, considering factors such as the improvement of model performance and interpretability by the feature fusion processing. Specifically, a neural network structure search technology can be used for network structure adjustment according to the fusion characteristics to obtain an image analysis model, so that the obtained model can better fit various types and forms of data.

Accordingly, in another embodiment of the analysis model building module, the following modules may be further included: the label quality evaluation module, the label selection module and/or the label correction module. Wherein:

The quick screening device for the unknown-cause pneumonia images, provided by the embodiment of the invention, is used for collecting a small number of lung CT image sequences, CT data and related data thereof to establish an image analysis model; when screening images of unknown pneumonia, fusing lung CT image data of a patient to be screened and related data thereof to obtain multidimensional characteristics; and inputting the multidimensional characteristics into an image analysis model, and obtaining the lung disease category and the confidence thereof according to the output of the image analysis model. Because the establishment of the image analysis model not only considers CT image data, but also integrates other pathological information, such as assay indexes, patient symptoms and other information, the condition of an illness can be rapidly and accurately comprehensively judged even for a patient with a shorter course of treatment and less lung image consolidation in emergency treatment and fever outpatient service, and the accuracy rate and the model performance of judgment are effectively improved.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. Furthermore, the above-described system embodiments are merely illustrative, wherein modules and units illustrated as separate components may or may not be physically separate, i.e., may be located on one network element, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Those skilled in the art will appreciate that all or part of the steps in the above method embodiments may be implemented by a program to instruct relevant hardware to perform the steps, and the program may be stored in a computer-readable storage medium, referred to herein as a storage medium, such as: ROM/RAM, magnetic disk, optical disk, etc.

Correspondingly, the embodiment of the invention also provides a device for the rapid screening method of the unknown-cause pneumonia images, and the device is an electronic device, such as a mobile terminal, a computer, a tablet device, a personal digital assistant and the like. The electronic device may include one or more processors, memory; wherein the memory is used for storing computer executable instructions and the processor is used for executing the computer executable instructions to realize the method of the previous embodiments.

The present invention has been described in detail with reference to the embodiments, and the description of the embodiments is provided to facilitate the understanding of the method and apparatus of the present invention, and is intended to be a part of the embodiments of the present invention rather than the whole embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without any creative effort shall fall within the protection scope of the present invention, and the content of the present description shall not be construed as limiting the present invention. Therefore, any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image rapid screening method for pneumonia of unknown cause is characterized by comprising the following steps:

acquiring CT image data of the lung of a patient and related data thereof;

2. The method of claim 1, wherein the using the acquired small lung CT image sequences, CT data and related data to build an image analysis model comprises:

performing enhancement processing on the CT data;

3. The method of claim 2, wherein the enhancing the CT data comprises:

4. The method of claim 2, wherein the label corresponding to each CT image comprises any one or more of: whether there are signs of infection, space occupation, pulmonary edema.

5. The method of claim 2, wherein the training with the enhanced CT data and the label information of the CT image to obtain the baseline model comprises:

6. The method of claim 2, wherein the network structure adjusting the baseline model according to the fusion features to obtain an image analysis model comprises:

7. The method of any one of claims 2 to 6, wherein the image analysis modeling using the acquired small number of lung CT image sequences and their associated data further comprises:

8. The method according to claim 1, wherein the fusing the related data comprises:

9. An apparatus for rapidly screening unknown cause pneumonia images, the apparatus comprising:

10. The apparatus of claim 9, wherein the analytical model building module comprises:

the data enhancement module is used for enhancing the CT data;

11. The apparatus of claim 10,

the data enhancement module is specifically used for adjusting the window level and the window width of the DICOM-format CT data to achieve contrast enhancement of different degrees.

12. The apparatus of claim 10, wherein the label corresponding to each CT image comprises any one or more of: whether there are signs of infection, space occupation, pulmonary edema.

13. The apparatus of claim 10, wherein the baseline model training module comprises:

14. The apparatus of claim 10,

the network structure adjusting module is specifically configured to perform network structure adjustment by using a neural network structure search technology according to the fusion characteristics to obtain an image analysis model.

15. The apparatus of any of claims 10 to 14, wherein the analytical model building module further comprises: the label quality evaluation module, the label selection module and/or the label correction module;

16. The apparatus of claim 9,

and the data conversion module is specifically used for carrying out fusion processing on the related data in an early fusion mode, a later fusion mode or a slow fusion mode.