CN110619359B

CN110619359B - Method and device for determining pulmonary tuberculosis grading according to human body medical image

Info

Publication number: CN110619359B
Application number: CN201910843662.8A
Authority: CN
Inventors: 房劬; 刘维平
Original assignee: Shanghai Xingmai Information Technology Co ltd
Current assignee: Shanghai Xingmai Information Technology Co ltd
Priority date: 2019-09-06
Filing date: 2019-09-06
Publication date: 2020-11-27
Anticipated expiration: 2039-09-06
Also published as: CN110619359A

Abstract

The invention aims to provide a method for determining the grade of tuberculosis according to a human medical image. The computing equipment obtains feature vectors under different scales through multi-scale pooling operation according to an image feature map of the medical image of the breast of the human body, wherein the multi-scale pooling operation comprises pooling operation performed under different scales; and connecting the feature vectors under different scales, and determining the corresponding tuberculosis grade according to the connected feature vectors. Compared with the prior art, the method can more effectively represent the characteristics in the sub-regions with different sizes in the original input image by introducing the multi-scale pooling operation, thereby ensuring that the minimal pulmonary tuberculosis signs are not lost on the characteristic diagram due to the pooling operation.

Description

Method and device for determining pulmonary tuberculosis grading according to human body medical image

Technical Field

The invention relates to the technical field of medical image processing, in particular to a technology for determining pulmonary tuberculosis grading according to human medical images.

Background

Chest X-ray radiation is widely used clinically as the most common method of tuberculosis examination. In the diagnosis of tuberculosis, it is often necessary to grade/stage suspected cases. The usual grading results are negative, inactive and active (positive). Wherein, negative is no tuberculosis infection; inactive tuberculosis is a non-infectious case that is usually in a cured state; active tuberculosis is an infectious case, and special treatment and even isolation are needed for the case. Due to significant differences in clinical pathways, staging is one of the most important issues in the process of tuberculosis diagnosis.

By means of X-ray radiography, tuberculosis can be effectively staged. Among them, negative cases do not have typical signs of tuberculosis; signs of inactive tuberculosis often include only fibrous and calcific foci; active tuberculosis has other symptoms such as exudation besides the fibrous and calcific foci.

In the prior art, the stage of the pulmonary tuberculosis through X-ray radiography still needs to depend on manual judgment. The method has higher requirements on personal experience and capability of doctors; meanwhile, manual film reading also has the problems of high cost, long time consumption, easy interference of human factors such as doctor states and the like.

With the rapid development of artificial intelligence, particularly in the field of deep learning, a great deal of researchers have tried to solve the problem of classification, staging or typing of medical images through such techniques. Conventional multi-classification networks (e.g., inclusion, ResNet, etc.) do not achieve ideal results given the particular problem of tuberculosis classification.

The reasons for the above problems are: 1) the size of the signs of inactive tuberculosis is smaller than the large-area signs of active tuberculosis, and conventional classification networks do not take scale differences into full consideration and also lead to poor accuracy. 2) The objective function of the classification network does not take into account the severity of the misclassification. While in the specific problem of tuberculosis grading, it is obviously more serious to misclassify active tuberculosis as negative than inactive tuberculosis, so that the conventional classification network is not suitable for penalizing errors.

Disclosure of Invention

The invention aims to provide a method, a device and a computing device for determining the pulmonary tuberculosis grade according to human medical images, a computer readable storage medium and a computer program product.

According to an aspect of the present invention, a method for determining a tuberculosis grade from medical images in a computing device is provided, wherein the method comprises the following steps:

acquiring an image characteristic diagram of a medical image of the chest of a human body;

obtaining feature vectors under different scales through multi-scale pooling operation according to the image feature map, wherein the multi-scale pooling operation comprises pooling operation performed under different scales;

connecting the feature vectors under different scales to obtain connected feature vectors;

performing at least one of the following operations on the connected feature vectors to determine a corresponding tuberculosis classification:

-classifying the feature vectors by a classifier, the respective classes comprising negative, inactive and positive tuberculosis;

-regressing the feature vectors by a regressor, the respective regression results indicating negative, inactive or positive tuberculosis, wherein the distance between negative and positive tuberculosis is further than the distance between negative and inactive tuberculosis.

According to an aspect of the present invention, there is also provided an apparatus for determining a tuberculosis classification from medical images in a computing device, wherein the apparatus comprises:

the characteristic acquisition device is used for acquiring an image characteristic diagram of the medical image of the breast of the human body;

the multi-scale pooling device is used for obtaining the feature vectors under different scales through multi-scale pooling operation according to the image feature map, wherein the multi-scale pooling operation comprises pooling operation performed under different scales;

the characteristic connecting device is used for connecting the characteristic vectors under different scales so as to obtain the connected characteristic vectors;

a grade determination device for performing at least one of the following operations on the connected feature vectors to determine a corresponding tuberculosis grade:

According to an aspect of the present invention, there is also provided a computing device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the computer program when executed by the processor implements a method for determining a tuberculosis classification from medical images in a computing device according to an aspect of the present invention.

According to an aspect of the invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a method for determining a tuberculosis grade from medical images in a computing device according to an aspect of the invention.

According to an aspect of the invention, there is also provided a computer program product which, when executed by a computing device, implements a method for determining a tuberculosis grade from medical images in a computing device according to an aspect of the invention.

Compared with the prior art, the method can more effectively represent the characteristics in the sub-regions with different sizes in the original input image by introducing the multi-scale pooling operation, thereby ensuring that the minimal pulmonary tuberculosis signs are not lost on the characteristic diagram due to the pooling operation. Furthermore, by introducing a "regression function" and setting the distance between negative and positive for tuberculosis to be longer than the distance between negative and inactive tuberculosis in consideration of the problem of the severity of the misclassification, it is possible to reduce the possibility that active tuberculosis is erroneously judged as negative to ensure higher clinical feasibility.

Drawings

Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 illustrates a flow diagram of a method for determining a tuberculosis grade from a medical image of a human body, according to an embodiment of the present invention;

FIG. 2 shows a schematic diagram of an apparatus for determining a tuberculosis grade from a medical image of a human body according to another embodiment of the present invention.

The same or similar reference numbers in the drawings identify the same or similar elements.

Detailed Description

Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments of the present invention are described as an apparatus represented by a block diagram and a process or method represented by a flow diagram. Although a flowchart depicts a sequence of process steps in the present invention, many of the operations can be performed in parallel, concurrently, or simultaneously. In addition, the order of the operations may be re-arranged. The process of the present invention may be terminated when its operations are performed, but may include additional steps not shown in the flowchart. The processes of the present invention may correspond to methods, functions, procedures, subroutines, and the like.

The methods illustrated by the flow diagrams and apparatus illustrated by the block diagrams discussed below may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as storage medium. The processor(s) may perform the necessary tasks.

Similarly, it will be further appreciated that any flow charts, flow diagrams, state transition diagrams, and the like represent various processes which may be substantially described as program code stored in computer readable media and so executed by a computing device or processor, whether or not such computing device or processor is explicitly shown.

As used herein, the term "storage medium" may refer to one or more devices for storing data, including Read Only Memory (ROM), Random Access Memory (RAM), magnetic RAM, kernel memory, magnetic disk storage media, optical storage media, flash memory devices, and/or other machine-readable media for storing information. The term "computer-readable medium" can include, but is not limited to portable or fixed storage devices, optical storage devices, and various other mediums capable of storing and/or containing instructions and/or data.

A code segment may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program descriptions. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, information passing, token passing, network transmission, etc.

The term "computing device" in this context refers to an electronic device that can perform predetermined processes such as numerical calculations and/or logical calculations by executing predetermined programs or instructions, and may include at least a processor and a memory, wherein the predetermined processes are performed by the processor executing program instructions prestored in the memory, or by hardware such as ASIC, FPGA, DSP, or by a combination of the above two.

The "computing device" described above is typically embodied in the form of a general purpose computing device, whose components may include, but are not limited to: one or more processors or processing units, system memory. The system memory may include computer readable media in the form of volatile memory, such as Random Access Memory (RAM) and/or cache memory. "computing device" may further include other removable/non-removable, volatile/nonvolatile computer-readable storage media. The memory may include at least one computer program product having a set (e.g., at least one) of program modules that are configured to perform the functions and/or methods of embodiments of the present invention. The processor executes various functional applications and data processing by executing programs stored in the memory.

For example, a computer program for executing the functions and processes of the present invention is stored in the memory, and when the processor executes the corresponding computer program, the present invention is implemented for determining the tuberculosis classification based on the human medical image.

Typically, the computing devices include, for example, user devices and network devices. Wherein the user equipment includes but is not limited to a Personal Computer (PC), a notebook computer, a mobile terminal, etc., and the mobile terminal includes but is not limited to a smart phone, a tablet computer, etc.; the network device includes, but is not limited to, a single network server, a server group consisting of a plurality of network servers, or a Cloud Computing (Cloud Computing) based Cloud consisting of a large number of computers or network servers, wherein Cloud Computing is one of distributed Computing, a super virtual computer consisting of a collection of loosely coupled computers. Wherein the computing device is capable of operating alone to implement the invention, or of accessing a network and performing the invention by interoperating with other computing devices in the network. The network in which the computing device is located includes, but is not limited to, the internet, a wide area network, a metropolitan area network, a local area network, a VPN network, and the like.

It should be noted that the user devices, network devices, networks, etc. are merely examples, and other existing or future computing devices or networks may be suitable for the present invention, and are included in the scope of the present invention and are incorporated by reference herein.

Specific structural and functional details disclosed herein are merely representative and are provided for purposes of describing example embodiments of the present invention. The present invention may, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be termed a second element, and, similarly, a second element may be termed a first element, without departing from the scope of example embodiments. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be noted that, in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may, in fact, be executed substantially concurrently, or the figures may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

The present invention is described in further detail below with reference to the attached drawing figures.

FIG. 1 shows a flow diagram of a method according to an embodiment of the invention, which particularly shows a process for determining a tuberculosis grade from medical images of a human body.

Typically, the invention is implemented by a computing device. When a general purpose computing device is configured with program modules embodying the present invention, it will become a specific purpose computing device for determining the grade of tuberculosis from a medical image of a human body, rather than any general purpose computer or processor. However, those skilled in the art will appreciate that the foregoing description is intended only to illustrate that the present invention may be applied to any general purpose computing device, which, when applied to a general purpose computing device, becomes a specific computing device for determining a tuberculosis grade based on medical images of a human body for practicing the present invention.

As shown in fig. 1, in step S1, the computing device acquires an image feature map of the medical image of the chest of the human body; in step S2, the computing device obtains feature vectors at different scales through multi-scale pooling operations according to the image feature map, wherein the multi-scale pooling operations include pooling operations performed at different scales; in step S3, the computing device concatenates the feature vectors at the different scales, thereby obtaining concatenated feature vectors; in step S4, the computing device performs at least one of the following operations on the concatenated feature vectors to determine a corresponding tuberculosis rating: classifying the feature vectors through a classifier, wherein the corresponding categories comprise negative tuberculosis, inactive tuberculosis and positive tuberculosis; and regressing the characteristic vectors by a regressor, wherein the corresponding regression results indicate negative tuberculosis, inactive tuberculosis or positive tuberculosis, and the distance between the negative tuberculosis and the positive tuberculosis is longer than that between the negative tuberculosis and the inactive tuberculosis.

Specifically, in step S1, the computing device acquires an image feature map of the medical image of the chest of the human body.

Here, an image feature map (feature map) of the medical image of the human breast may be extracted through a classification network based on deep learning, for example, a high-dimensional feature map is common.

Deep learning is a series of algorithms in the field of machine learning, which attempt to perform multi-layer abstraction on data by using multiple nonlinear transformations, and not only learns the nonlinear mapping between input and output, but also learns the hidden structure of the input data vector, so as to perform intelligent identification or prediction on new samples.

The general classification network comprises a feature extraction part and a classification part, wherein the feature extraction part mainly comprises convolution layers, an activation layer and a pooling layer which are alternately arranged. The convolutional layer can extract image features within a certain range through convolution operation; the activation layer can carry out nonlinear operation on the convolution output, so that the neural network can represent a complex nonlinear relation; the pooling layer may reduce the size of the high-dimensional feature map by averaging or maximizing. After a plurality of cycles of the convolution layer, the activation layer and the pooling layer, a high-dimensional image feature map is finally generated.

Deep learning based classification Networks that may be used with the present invention are, for example, CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks), AlexNet Convolutional Neural Networks, GoogleNet (Google addition Net), VGG (visual Geometry group), and the like. The present invention is not particularly limited in this regard. Any existing or future classification network is included within the scope of the present invention if its feature extraction function can implement the high-dimensional image feature map claimed by the present invention, and is incorporated herein by reference.

According to an embodiment of the present invention, the computing device may be integrated with a classification network to extract the corresponding high-dimensional feature map of the input human chest medical image, or may invoke a classification network and extract the corresponding high-dimensional feature map of the input human chest medical image through the invoked classification network. In addition, the extraction of the corresponding high-dimensional feature map from the human chest medical image can be performed by any third-party device, and the extracted high-dimensional feature map is transmitted to the computing device by the third-party device through any available communication mode.

In step S2, the computing device obtains feature vectors at different scales through multi-scale pooling operations on the high-dimensional image feature map, where the multi-scale pooling operations include pooling operations performed at different scales.

Here, the multi-scale pooling operation is intended to obtain feature vectors at different scales. According to one embodiment of the invention, a computing device employs a multi-scale pooling operation to obtain feature vectors at different scales. The multi-scale representation of the characteristic diagram can more effectively represent the characteristics in the subareas with different sizes in the original input image. To correctly select the different size parameters, statistics are taken of the size of the tuberculosis signs to ensure that the smallest signs are not lost on the signature by pooling. Multi-scale Average Pooling can be performed at a number of different scales to obtain Average levels at different scales, as compared to Global Average Pooling (Global Average Pooling), which can only obtain Average levels of feature maps.

Through statistics on training samples, it was found that the pulmonary tuberculosis signs were mostly 0.1 to 0.5 times the image size in a single dimension (length/width). For example, with a 1024 x 1 image as an input, the length and width dimensions of the pulmonary tuberculosis signs are generally distributed between 102 and 512 pixels. After feature extraction, the 1024 × 1 image outputs a 32 × 2048-dimensional high-dimensional feature map, and the scale of the corresponding tuberculosis feature is 3 × 2048 to 16 × 2048. From this, it is found that the average value of the minimum signs of pulmonary tuberculosis can be effectively obtained by the 2 × 2 averaging operation. Therefore, starting with the smallest 2 × 2 average operation, ending with the global average (32 × 32), and increasing quadratically as a single step, the final window widths of the multi-scale pooling operations are 2 × 2, 4 × 4, 8 × 8, 16 × 16, 32 × 32 in that order.

Accordingly, the starting point scale of the multi-scale pooling operation is determined according to the scale of the pulmonary tuberculosis signs, such as the scale of the smallest signs, to ensure that any signs of pulmonary tuberculosis are not lost due to the pooling operation, and the ending point scale is the global scale of the high-dimensional feature map. The step size between adjacent scales is, for example, quadratic in growth. Because each scale is gradually increased from small to large according to a single step length, the feature pyramid pooling can obtain the features under different scales so as to ensure the sensitivity of the detection algorithm of the invention to the signs of each scale.

Further, to reduce the amount of computation, the high-dimensional feature map obtained in step S1 may be subjected to a multi-scale average pooling operation after dimensionality reduction. For example, the computing device may perform dimensionality reduction on the high-dimensional feature map by 1-by-1 convolution.

Taking a 1024 × 1 input image as an example, feature extraction is performed on the input image through an Xception network, so as to obtain a high-dimensional feature map with a dimension of 32 × 2048. Subsequently, the computing device performs dimensionality reduction on the 32 × 2048-dimensional feature map by 1 × 1 convolution to obtain a dimensionality-reduced feature map, that is, a feature map with one dimension being 32 × 3. And then, the computing device obtains feature vectors under different scales by carrying out multi-scale average pooling operation on the 32 × 3-dimensional feature map. For each feature map in the depth dimension, the computing device extracts the average of the features over 2 × 2, 4 × 4, 8 × 8, 16 × 16, 32 × 32 (i.e., global) scales, respectively, to obtain the output of the average pooling at each scale, respectively: namely, 16 × 3 eigenvectors, 8 × 3 eigenvectors, 4 × 3 eigenvectors, 2 × 3 eigenvectors, and 1 × 3 eigenvectors.

In step S3, the computing device concatenates the feature vectors at different scales, thereby obtaining concatenated feature vectors.

Still by way of example, before the connection, the computing device may, for example, perform a morphing operation on each feature vector output by the multi-scale average pooling operation to obtain feature vectors with dimensions 768(16 × 3), 192(8 × 3), 48(4 × 3), 12(2 × 3), and 3(1 × 3), respectively, and then perform a connection operation on the feature vectors to obtain a feature vector with a dimension of 1023 (768+192+48+12+ 3).

In step S4, the computing device performs at least one of the following operations on the concatenated feature vectors to determine a corresponding tuberculosis rating:

1) classifying the feature vectors through a classifier, wherein the corresponding categories comprise negative tuberculosis, inactive tuberculosis and positive tuberculosis;

2) and performing regression on the feature vectors through a regressor, wherein the corresponding regression result indicates negative tuberculosis, inactive tuberculosis or positive tuberculosis, and the distance between the negative tuberculosis and the positive tuberculosis is longer than that between the negative tuberculosis and the inactive tuberculosis.

Here, the computing device may train the classifier in advance, and the classifier to which the present invention is applicable includes, but is not limited to, a support vector machine, a decision tree, a deep learning based classification network, and the like, which is not limited by the present invention.

The training samples are respectively marked with classification labels, including negative tuberculosis, inactive tuberculosis and positive tuberculosis. The classification label may be implemented in the form of a one-hot code, for example, a negative tuberculosis label of [1,0,0], an inactive tuberculosis label of [0,1,0], and an active tuberculosis label of [0,0,1 ]. Accordingly, the sample feature vectors labeled with the classification labels are input into the classifier to train the classifier, so that the trained classifier is obtained.

The computing device may also pre-train the regressor, and the regression function applicable to the present invention includes, but is not limited to, logistic regression, linear regression, etc., which is not limited by the present invention. By adding a regression function, the possibility that the active tuberculosis is misjudged to be negative can be reduced, so that higher clinical feasibility is ensured.

The training samples are labeled with regression labels, respectively, the regression labels indicating negative, inactive or positive tuberculosis, respectively, with values between [0,1 ]. Considering the severity of the misclassification, the distance between negative and positive tuberculosis should be longer than the distance between negative and inactive tuberculosis. For example, regression labeling is implemented in discrete numbers, with a negative tuberculosis label of 0, an inactive tuberculosis label of 0.5, and an active tuberculosis label of 1. Alternatively, the regression label indicates negative, inactive or positive tuberculosis in three intervals divided between [0, 1], for example, the label interval of negative tuberculosis is [0, 0.3 ], the label interval of inactive tuberculosis is [ 0.4,0.6 ] and the label interval of active tuberculosis is [ 0.7,1 ].

When a classifier and a regressor are employed to determine the tuberculosis grade, the training objective function is a weighted sum of the classification sub-objective functions of the classifier and the regression sub-objective functions of the regressor when the classifier and the regressor are trained. Here, for example, the output of the classifier and the output of the regressor may be adjusted to the same order of magnitude, and the respective weights of both may be set according to the experimental result. The output adjustment may be, for example, by a threshold adjustment, and the weights of the classifier and the regressor may be, for example, 1.

In obtaining the feature vectors concatenated in step S3, the computing device inputs these feature vectors into a trained classifier to obtain output classification labels. The computing device may also input these feature vectors into a trained regressor to obtain an output regression result, e.g., 0, 0.5, or 1, to indicate negative, inactive, or positive tuberculosis, respectively.

Further, the computing device may determine a corresponding tuberculosis grade by reducing dimensions of the connected feature vectors in step S3. For example, the computing device may perform dimensionality reduction on the feature vectors through operations of the fully-connected layer, such as two fully-connected operations. Specifically, in step S3, a 1023-dimensional feature vector is obtained, and the computing device obtains a 10-dimensional feature vector after dimensionality reduction through two full-join operations. Therefore, when training the classifier and the regressor, the training samples may also use 10-dimensional feature vectors.

Here, the computing device may determine the tuberculosis rating for the connected feature vectors using only one of the classifier or the regressor, or may combine both. When the classifier and the regressor are combined to make determination, the determination strategy can be that any one is determined to be positive, namely determined to be positive; or, the respective outputs of the classifier and the regressor are used as inputs, and then a classifier (such as a random forest) is trained to perform final classification.

Fig. 2 shows a schematic diagram of an apparatus according to an embodiment of the present invention, which particularly shows an apparatus for determining a tuberculosis classification from medical images of a human breast.

Typically, the apparatus of the present invention can be implemented as a functional module in any general-purpose computing device. When a general purpose computing device is configured with the apparatus of the present invention, it will become a specific purpose computing device for determining the grade of tuberculosis from medical images of the human breast, rather than any general purpose computer or processor. However, it will be appreciated by those skilled in the art that the foregoing description is intended merely to illustrate that the apparatus of the present invention may be applied to any general purpose computing device, which, when applied to a general purpose computing device, becomes a specific computing device for performing the method of the present invention for determining a tuberculosis grade based on medical images of a human breast, and may therefore also be referred to as a "computing apparatus". Also, the "computing means" may be implemented in a computer program, hardware, or a combination thereof.

As shown in fig. 2, the computing device 20 is disposed in a computing apparatus 200. The computing means 20 further comprises feature acquisition means 21, multi-scale pooling means 22, feature connection means 23 and rank determination means 24.

Wherein, the characteristic acquisition device 21 acquires an image characteristic diagram of the medical image of the breast of the human body; the multi-scale pooling device 22 obtains feature vectors at different scales through multi-scale pooling operations according to the image feature map, wherein the multi-scale pooling operations comprise pooling operations performed at different scales; the feature connecting device 23 connects the feature vectors under the different scales to obtain connected feature vectors; the grade determining means 24 performs at least one of the following operations on the connected feature vectors to determine a corresponding tuberculosis grade: classifying the feature vectors through a classifier, wherein the corresponding categories comprise negative tuberculosis, inactive tuberculosis and positive tuberculosis; and regressing the characteristic vectors by a regressor, wherein the corresponding regression results indicate negative tuberculosis, inactive tuberculosis or positive tuberculosis, and the distance between the negative tuberculosis and the positive tuberculosis is longer than that between the negative tuberculosis and the inactive tuberculosis.

Specifically, the feature acquisition device 21 acquires an image feature map of a medical image of the breast of the human body.

Here, the image feature map of the medical image of the human chest can be extracted through a classification network based on deep learning, for example, a high-dimensional feature map is common.

According to an embodiment of the present invention, the feature obtaining device 21 may be integrated with a classification network to extract the corresponding high-dimensional feature map from the input human chest medical image, or may invoke a classification network and extract the corresponding high-dimensional feature map from the input human chest medical image through the invoked classification network. In addition, the extraction of the corresponding high-dimensional feature map from the human chest medical image may be performed by any third-party device, and the extracted high-dimensional feature map is transmitted to the feature obtaining apparatus 21 by the third-party device through any available communication method.

Subsequently, the scale averaging device 22 obtains feature vectors at different scales through a multi-scale pooling operation on the high-dimensional image feature map, wherein the multi-scale pooling operation comprises pooling operations performed at different scales.

Further, to reduce the amount of computation, the high-dimensional feature map obtained by the feature obtaining device 21 may be subjected to a multi-scale average pooling operation after dimensionality reduction by, for example, a first dimensionality reduction device (not shown in fig. 2). For example, the first dimension reduction means may reduce the dimension of the high-dimensional feature map by 1 × 1 convolution. The first dimension reduction means may be provided as a stand-alone device or may be integrated in the feature acquisition means 21 or the multi-scale pooling means 22. When the first dimension reduction means is integrated in the feature obtaining means 21, the feature obtaining means 21 reduces the dimension of the high-dimensional feature map and transmits the reduced dimension to the multi-scale pooling means 22.

Taking an input image of 1024 × 1 as an example, feature extraction is performed on the input image through the Xception network, and the feature obtaining device 21 obtains a high-dimensional feature map with a dimension of 32 × 2048. Subsequently, the first dimension reduction device reduces the dimensions of the 32 × 2048-dimensional feature map by 1 × 1 convolution, and obtains a feature map after dimension reduction, that is, a feature map with one dimension being 32 × 3. Further, the multi-scale pooling device 22 performs a multi-scale average pooling operation on the 32 × 3-dimensional feature map to obtain feature vectors at different scales. For each feature map in the depth dimension, the multi-scale pooling device 22 extracts the average of the features in the 2 × 2, 4 × 4, 8 × 8, 16 × 16, 32 × 32 (i.e., global) scale range, respectively, to obtain the output of the average pooling at each scale, respectively: namely, 16 × 3 eigenvectors, 8 × 3 eigenvectors, 4 × 3 eigenvectors, 2 × 3 eigenvectors, and 1 × 3 eigenvectors.

Subsequently, the feature connecting means 23 connects the feature vectors at different scales, thereby obtaining connected feature vectors.

Still by way of example, before the connection, the feature connection device 23 may, for example, perform a transformation operation on each feature vector output by the multi-scale average pooling operation to obtain feature vectors with dimensions 768(16 × 3), 192(8 × 3), 48(4 × 3), 12(2 × 3), and 3(1 × 3), respectively, and then perform a connection operation on the feature vectors to obtain a feature vector with dimensions of 1023 (768+192+48+12+ 3).

Next, the grade determining means 24 performs at least one of the following operations on the connected feature vectors to determine a corresponding tuberculosis grade:

In this case, the classification determination device 24 may be integrated with a classifier and/or a regressor, or may call an external classifier and/or regressor.

Classifiers are trained in advance, and classifiers to which the present invention is applicable include, but are not limited to, support vector machines, decision trees, deep learning based classification networks, and the like, which are not limited by the present invention.

Regressor classifier and/or regressor, regression functions applicable to the present invention include, but are not limited to, logistic regression, linear regression, etc., which the present invention is not limited to. By adding a regression function, the possibility that the active tuberculosis is misjudged to be negative can be reduced, so that higher clinical feasibility is ensured.

After obtaining the feature vectors connected by the feature connecting means 23, the classification determining means 24 inputs these feature vectors into a trained classifier to obtain an output classification label. The rank determination means 24 may also input these feature vectors into a trained regressor to obtain an output regression result, e.g. 0, 0.5 or 1, to indicate negative, inactive or positive tuberculosis, respectively.

Further, the connected feature vectors in the feature connection device 23 may be dimension-reduced by a second dimension reduction device (not shown in fig. 2) to determine the corresponding tuberculosis grade. For example, the second dimension reduction means may perform dimension reduction on the feature vector through an operation of a full connection layer, such as two full connection operations. Specifically, the feature connecting means 23 obtains a feature vector with 1023 dimensions, and the second dimension reducing means obtains a 10-dimensional feature vector after dimension reduction through two full connecting operations. Therefore, when training the classifier and the regressor, the training samples may also use 10-dimensional feature vectors.

The second dimension reduction means may be provided as a stand-alone device or may be integrated in the feature connection means 23 or the hierarchy determination means 24. When the second dimension reduction means is integrated in the feature connection means 23, the feature connection means 23 transfers the deformed and connected feature vector after dimension reduction to the hierarchical determination means 24.

Here, the grade determining means 24 may determine the tuberculosis grade for the connected feature vectors using only one of the classifier or the regressor, or may determine the tuberculosis grade for the connected feature vectors in combination of both. When the classifier and the regressor are combined to make determination, the determination strategy can be that any one is determined to be positive, namely determined to be positive; or, the respective outputs of the classifier and the regressor are used as inputs, and then a classifier (such as a random forest) is trained to perform final classification.

According to the various embodiments described above, the following clauses are proposed:

clause 1. a method for use in a computing device for determining a tuberculosis classification from a medical image, wherein the method comprises the steps of:

Clause 2. the method of clause 1, wherein the method further comprises:

and performing the multi-scale pooling operation after the dimension reduction of the image feature map.

Clause 3. the method of clause 1 or 2, wherein the method further comprises:

and reducing the dimension of the connected feature vectors and determining the corresponding tuberculosis grade.

Clause 4. the method of any one of clauses 1-3, wherein,

sample feature vectors labeled with classification labels including negative, inactive, and positive tuberculosis are input to the classifier to obtain the trained classifier.

Clause 5. the method of any one of clauses 1-4, wherein,

inputting the sample feature vectors marked with regression labels into the regressor to obtain the trained regressor, wherein the regression labels respectively indicate negative tuberculosis, inactive tuberculosis or positive tuberculosis with interval values between [0, 1], and the distance between the negative tuberculosis and the positive tuberculosis is farther than the distance between the negative tuberculosis and the inactive tuberculosis.

Clause 6. the method of any of clauses 1-5, wherein a starting point scale of the multi-scale pooling operation is determined from a scale of a tuberculosis sign, and an ending point scale is a global scale of the image feature map.

Clause 7. the method of any of clauses 1-6, wherein the step size between adjacent scales grows quadratically.

Clause 8. the method of any of clauses 1-7, wherein the multi-scale pooling operation comprises a multi-scale average pooling operation.

Clause 9. the method of any of clauses 1-8, wherein the image feature map is extracted by a classification network based on deep learning.

Clause 10. the method of any of clauses 1-9, wherein the concatenating operation concatenates the feature vectors at the different scales into one 1-dimensional feature vector.

Clause 11. an apparatus for use in a computing device for determining a tuberculosis classification from a medical image, wherein the apparatus comprises:

Clause 12. the apparatus of clause 11, wherein the apparatus further comprises:

and the first dimension reduction device is used for performing the multi-scale pooling operation after the dimension reduction of the image feature map.

Clause 13. the apparatus of clause 11 or 12, wherein the apparatus further comprises:

and the second dimension reduction device is used for determining the corresponding tuberculosis grade after the dimension reduction of the connected feature vectors.

Clause 14. the apparatus of any one of clauses 11-13, wherein,

the classifier obtains training by inputting sample feature vectors labeled with classification labels, which include negative, inactive and positive tuberculosis.

Clause 15. the apparatus of any one of clauses 11-14, wherein,

the regressor obtains training by inputting a sample feature vector marked with a regression label, wherein the regression label respectively indicates negative tuberculosis, inactive tuberculosis or positive tuberculosis according to interval values between [0, 1], and the distance between the negative tuberculosis and the positive tuberculosis is farther than the distance between the negative tuberculosis and the inactive tuberculosis.

Clause 16. the apparatus of any of clauses 11-15, wherein a starting point scale of the multi-scale pooling operation is determined from a scale of a tuberculosis sign, and an ending point scale is a global scale of the image feature map.

Clause 17. the apparatus of any one of clauses 11-16, wherein the step size between adjacent scales grows quadratically.

Clause 18. the apparatus of any one of clauses 11-17, wherein the multi-scale pooling operation comprises a multi-scale average pooling operation.

Clause 19. the apparatus of any one of clauses 11-18, wherein the image feature map is extracted by a classification network based on deep learning.

Clause 20. the apparatus of any one of clauses 11-19, wherein the concatenating operation concatenates the feature vectors at the different scales into one 1-dimensional feature vector.

Clause 21. a computing device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any of clauses 1-10.

Clause 22. a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method of any of clauses 1 to 10.

Clause 23. a computer program product implementing the method of any one of clauses 1 to 10 when executed by a computing device.

It should be noted that the present invention may be implemented in software and/or in a combination of software and hardware, for example, as an Application Specific Integrated Circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention may be executed by a processor to implement the steps or functions described above. Also, the software programs (including associated data structures) of the present invention can be stored in a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. Additionally, some of the steps or functions of the present invention may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions_。

In addition, at least a portion of the present invention may be implemented as a computer program product, such as computer program instructions, which, when executed by a computing device, may invoke or provide methods and/or aspects in accordance with the present invention through operation of the computing device. Program instructions which invoke/provide the methods of the present invention may be stored on fixed or removable recording media and/or transmitted via a data stream over a broadcast or other signal-bearing medium, and/or stored in a working memory of a computing device operating in accordance with the program instructions.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.

Claims

1. An apparatus for use in a computing device for determining a tuberculosis classification from a medical image, the apparatus comprising:

the characteristic acquisition device is used for acquiring an image characteristic diagram of the medical image of the breast of the human body; the image feature map is extracted from the chest medical image by a classification network based on deep learning;

the multi-scale pooling device is used for obtaining the feature vectors under different scales through multi-scale pooling operation according to the image feature map, wherein the multi-scale pooling operation comprises pooling operation performed under different scales; wherein, the starting point scale of the multi-scale pooling operation is determined according to the scale of the pulmonary tuberculosis sign, the end point scale is the global scale of the image feature map, and the step length between adjacent scales is increased by a quadratic power;

a grade determining device, configured to perform dimension reduction processing on the connected feature vectors, and perform at least one of the following operations to determine a corresponding tuberculosis grade:

-classifying the feature vectors by a classifier, the classification resulting in classes comprising negative, inactive and positive tuberculosis;

2. The apparatus of claim 1, wherein: the starting point for the multi-scale pooling operation is the average of the minimal signs of tuberculosis.

3. The apparatus of claim 1 or 2, wherein: the multi-scale pooling operation comprises a multi-scale average pooling operation.

4. A computing device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein: the processor, when executing the computer program, implements a method of determining a tuberculosis grade from a medical image;

wherein the method of determining a tuberculosis grade from a medical image comprises the steps of:

acquiring an image characteristic diagram of a medical image of the chest of a human body; the image feature map is extracted from the chest medical image by a classification network based on deep learning;

obtaining feature vectors under different scales through multi-scale pooling operation according to the image feature map, wherein the multi-scale pooling operation comprises pooling operation performed under different scales; wherein, the starting point scale of the multi-scale pooling operation is determined according to the scale of the pulmonary tuberculosis sign, the end point scale is the global scale of the image feature map, and the step length between adjacent scales is increased by a quadratic power;

performing dimensionality reduction on the connected feature vectors, and performing at least one of the following operations to determine corresponding tuberculosis grades:

5. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when executed by a processor, implements a method of determining a tuberculosis grade from a medical image;