WO2024123057A1 - Method and analysis device for visualizing bone tumor in humerus using chest x-ray image - Google Patents

Abstract

A method for visualizing a bone tumor in the humerus using a chest X-ray image comprises the steps in which: an analysis device receives a chest X-ray image of a subject; the analysis device inputs the chest X-ray image to a trained segmentation model to distinguish a humerus region and generates a humerus patch of a certain size; and the analysis device inputs the humerus patch to a pre-trained deep learning model and outputs a class activation map (CAM) for the humerus patch and whether or not a bone tumor is present.

Description

Bone tumor visualization method and analysis device of the humerus using chest X-ray images

The technology described below relates to a technique for classifying lesions of the humerus in thoracic medical images. In particular, the technology described below relates to a technique for diagnosing a bone tumor of the humerus.

A chest X-ray contains information about various diseases or lesions. However, chest x-rays also include the humerus area. Lesions in the humerus area can also be determined through chest X-ray. Meanwhile, various learning models for interpreting medical images have recently been studied.

Supervised learning models require large amounts of learning data to learn the model. However, medical institutions do not have a large number of chest X-rays for diagnosing lesions of the humerus. Accordingly, it is difficult with conventional techniques to create a model that accurately classifies humeral lesions in chest X-rays.

The technology described below seeks to provide a technique for accurately classifying humeral lesions in chest X-rays using a learning model.

The method of visualizing a bone tumor of the humerus using a chest It includes generating a patch and the analysis device inputting the humerus patch into a previously learned deep learning model and outputting a CAM (Class Activation Map) for the humerus patch and whether or not it is a bone tumor.

The analysis device that classifies bone tumors of the humerus is an interface device that receives the subject's chest X-ray image, a segmentation model that distinguishes the humerus region in the chest ) and a storage device for storing a deep learning model that generates the input chest It includes a calculation device that inputs the deep learning model and calculates CAM and bone tumor for the humerus patch.

The technology described below can classify bone tumors of the humerus by focusing only on the humerus region in chest X-rays. Additionally, the technique described below can accurately visualize the area of the bone tumor on a chest x-ray.

Figure 1 is an example of a system for classifying humeral bone tumors using chest X-rays.

Figure 2 is a schematic example of the learning process of a learning model for classifying humeral bone tumors.

Figure 3 is an example of a learning data construction process or data preprocessing process.

Figure 4 is an example of a specific learning process of a learning model for classifying humeral bone tumors.

Figure 5 shows the results of evaluating the performance of a learning model for classifying humeral bone tumors.

Figure 6 is an example of an analysis device for classifying humeral bone tumors.

The technology described below may be subject to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, B, etc. may be used to describe various components, but the components are not limited by the terms, and are only used for the purpose of distinguishing one component from other components. It is used only as For example, a first component may be named a second component without departing from the scope of the technology described below, and similarly, the second component may also be named a first component. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

In terms used in this specification, singular expressions should be understood to include plural expressions, unless clearly interpreted differently from the context, and terms such as “including” refer to the described features, numbers, steps, operations, and components. , it means the existence of parts or a combination thereof, but should be understood as not excluding the possibility of the presence or addition of one or more other features, numbers, step operation components, parts, or combinations thereof.

Before providing a detailed description of the drawings, it would be clarified that the division of components in this specification is merely a division according to the main function each component is responsible for. That is, two or more components, which will be described below, may be combined into one component, or one component may be divided into two or more components for more detailed functions. In addition to the main functions that each component is responsible for, each component to be described below may additionally perform some or all of the functions that other components are responsible for, and some of the main functions that each component is responsible for may be performed by other components. Of course, it can also be carried out exclusively by .

In addition, when performing a method or operation method, each process that makes up the method may occur in a different order from the specified order unless a specific order is clearly stated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the opposite order.

The technology described below is a technique for determining whether a subject has a disease using medical images. Medical images can be diverse, such as X-ray images, ultrasound images, CT (Computer Tomography) images, and MRI (Magnetic Resonance Imaging) images. For convenience of explanation, the explanation below will be based on X-ray images.

The technology described below is a technique to classify lesions of the humerus using chest X-ray images. For example, the technology described below can classify bone tumors of the humerus in chest X-ray images.

Bone tumors include primary tumors such as osteosarcoma and bone metastases.

The following explains that the analysis device uses a learning model to classify or diagnose lesions in chest X-rays. The analysis device can be implemented as a variety of devices capable of processing data. For example, an analysis device can be implemented as a PC, a server on a network, a smart device, or a chipset with a dedicated program embedded therein.

Machine learning models include decision trees, random forest, KNN (K-nearest neighbor), Naive Bayes, SVM (support vector machine), and ANN (artificial neural network). In particular, the technology described below uses a deep learning model among ANNs to analyze lesions in the humerus in chest X-rays.

Figure 1 is an example of a system 100 for classifying humeral bone tumors using chest X-rays. Figure 1 shows an example in which the analysis device is a computer terminal 130 and a server 140.

The x-ray equipment 110 generates a chest x-ray image for the subject (patient). The x-ray equipment 110 may store the generated chest x-ray image in an electronic medical record (EMR) 120 or a separate database (DB).

In FIG. 1, user A can analyze a chest X-ray image using the computer terminal 130. The computer terminal 130 may receive a chest X-ray image from the x-ray equipment 110 or the EMR 120 through a wired or wireless network. In some cases, the computer terminal 130 may be a device physically connected to the x-ray equipment 110. The computer terminal 130 can constantly preprocess chest X-ray images. The computer terminal 130 can extract the humerus region from the chest X-ray and input it into a previously learned learning model. The computer terminal 130 can classify whether there is a bone tumor in the input humerus based on the value output by the learning model. User A can check the analysis results on the computer terminal 130.

The server 140 may receive a chest X-ray image from the x-ray equipment 110 or the EMR 120. The server 140 can consistently preprocess chest X-ray images. The server 140 may extract the humerus region from the chest X-ray and input it into a pre-trained learning model. The server 140 can classify whether there is a bone tumor in the input humerus based on the value output by the learning model. The server 140 may transmit the analysis results to user A's terminal.

The computer terminal 130 and/or server 140 may store the analysis results in the EMR 120.

Figure 2 is an example of a schematic learning process 200 of a learning model for classifying humeral bone tumors. The learning model building process can be divided into a learning data building process (210) and a model learning process using the learning data (220). The learning data construction process and model learning process can be performed on separate devices. For convenience of explanation, it is explained that the learning device performs the learning model construction process. Learning device refers to a computer device capable of preprocessing medical images and learning models.

The learning device constructs learning data by regularly preprocessing the collected chest X-ray images (210).

The learning device acquires chest X-ray images from an image database (DB). The image DB stores images of normal people and images of patients with bone tumors.

The learning device extracts a region of interest by inputting the entire chest X-ray of a subject from the population into a previously learned segmentation model. The segmentation model is a model trained in advance to distinguish the humerus region in chest X-rays. For example, the segmentation model may be a U-net-based model. The learning device can generate a humerus mask based on the humerus region. Additionally, the learning device can generate a bone tumor mask that indicates the location of the bone tumor in the humerus from the image of the bone tumor patient. The learning DB can store the humerus region and mask (humerus mask and bone tumor mask) extracted from the population's chest X-ray images. The learning device can prepare learning data by processing images of normal people and images of tumor patients through the same process.

The learning device performs a process of building (learning) a learning model using the humerus region and mask (220). The learning device repeatedly performs the learning process using population learning data. The learning model extracts features from the input image and outputs an image that visualizes the humerus area or bone tumor area. Additionally, the learning model can calculate a probability value for the presence or absence of a bone tumor. The learning device updates the parameters of the learning model by comparing the probability value output by the learning model with the label value of the corresponding image that is known in advance. The learning model can binary classify subjects as normal or tumor.

Figure 3 is an example of a learning data construction process or data preprocessing process.

Figure 3(A) is an example of a process 300 for extracting the humerus region from a chest X-ray. Figure 3(A) shows the process of constructing learning data. Figure 3(A) also corresponds to the data preprocessing process for chest X-rays. The data preprocessing process can be performed in the same way in the bone tumor inference process using a learning model. For convenience of explanation, it is explained that the data preprocessing process is performed in the learning device.

The learning device receives a chest X-ray image (310). At this time, the chest X-ray image may be an image of a normal person or a patient with a bone tumor. The learning device can classify the humerus region by inputting the input chest X-ray image into the learned segmentation model (320). Chest x-ray image includes left humerus and right humerus. The learning device sets a square area of a certain size for each of the left humerus and the right humerus based on the center point of the humerus (330). The learning device crops a rectangular area (left humerus patch) of a certain size based on the midpoint of the left humerus and a rectangular area (right humerus patch) of a constant size based on the midpoint of the right humerus (340). The image created by cutting is called a patch. Through this, the learning device can extract the region of interest (superior bone region) from the chest X-ray image. The superior bone region includes the humerus.

As described above, the learning device extracts the left humerus region and the right humerus region from the chest X-ray image. The learning device can prepare data for one direction in order to build a learning model based on one humerus region. For example, the learning device can construct learning data based on the left humerus. Figure 3(B) is an example of constructing learning data based on the humerus region extracted from Figure 3(A). The learning device uses the left humerus area in the chest X-ray image. The learning device reverses the right humerus area left and right to unify the image to the left humerus. The learning device creates a mask (humerus segmentation mask) that can extract the humerus region from the left humerus region and the left humerus region created by inversion. Additionally, the learning device creates a mask (bone tumor mask) that can extract the location of the bone tumor of the humerus based on the left humerus.

Figure 4 is an example of a specific learning process 400 of a learning model for classifying humeral bone tumors. Figure 4 shows a CNN (Convolutional Neural Network) based model as an example. The learning device receives an image of the humerus region among the learning data (410). The learning device inputs the humerus region image into the deep learning model (420). At this time, the CNN may be composed of multiple layers. The convolution layer extracts features from the input image (creating a feature map). CNN does not use a typical fully connected layer (FC) at the end, but has a layer that performs global average pooling. This model corresponds to a model that generates the so-called CAM (Class Activation Map). The learning device multiplies the extracted activation map by the weight of the class through matrix multiplication to perform CAM

can be created. The learning device inputs the humerus region image into a deep learning model and generates a CAM for the image (420).

If the input humerus image is normal, the learning device receives the humerus mask for the corresponding image (430). Alternatively, if the input humerus image has a bone tumor, the learning device receives a bone tumor mask for the image (430).

The learning device multiplies the mask corresponding to the CAM and creates the masked image.

creates . The process of applying a mask to CAM is as shown in Equation 1 below.

means mask. If the humerus image is normal, the mask is a humerus mask, and if the humerus image has a bone tumor, the mask is a bone tumor mask.

means element wise product.

The learning device performs learning using the L2 loss (least square error) function to treat all pixel values of the CAM as 0 for areas excluding the masked area (humerus area or tumor area) in the CAM (440). In other words, lossy L _FPAR processes the image so that all colors are removed from the area excluding the mask in the CAM.

Meanwhile, the last layer of CNN can further use loss L _CLS for classifying bone tumor in addition to the loss function for CAM. L _CLS corresponds to the loss function that guides the model to infer the correct answer.

Ultimately, when learning of the deep learning model is completed, the deep learning model processes pixel values excluding the humerus area or tumor area as 0 according to the characteristics of the input humerus image, thereby creating a CAM in which only the humerus area or tumor area stands out. In addition, the deep learning model also outputs values that classify the input humerus image as normal or bone tumor.

The researcher conducted an experiment to verify the deep learning model (CNN) learned using the above-described method. To test the proposed method, the researcher performed learning and verification using data as shown in Table 1 below.

Dataset	Abnormal (osteosarcoma)	normal
For classification (Abnormal/Normal)	1493 (89 people)	1500 people
For segmentation (Humerus mask)	-	1000 people
For holdout validation (Abnormal/Normal)	100 people	119 people
Total	1593	2619

The researcher used the segmentation mask of the humerus displayed on 1,000 normal chest X-ray images to learn a segmentation model. The researcher used Attention UNet as a segmentation model, and 800 of the 1,000 sheets were used for learning, and the remaining 200 sheets were used for verification. The performance of the segmentation model showed high performance with an average of IOU (Intersection over Union) and Dice of 0.98, as shown in Table 2 below.

	IOU	Dice
score	0.978	0.989

In addition, the researcher used 1,493 chest X-ray images with osteosarcoma and 1,500 chest X-ray images of normal people to learn the deep learning model described above. The 1,493 images were images showing the location of osteosarcoma in 89 people diagnosed with osteosarcoma. The researcher used 100 images of osteosarcoma patients and 119 images of normal subjects for model validation (holdout validation).

The researcher built a CNN-based model with various structures and verified its performance. The researchers used EfficientNet, ResNet, DenseNet, and ShuffleNetV2. The researcher also built a comparison model to compare the performance of the proposed model. Comparison Model 1 is a model that classifies osteosarcoma by receiving only half of the image including the humerus from the chest X-ray. Comparison Model 2 is a model that classifies osteosarcoma by receiving only the image (patch) of the humerus area from the chest X-ray. The proposed model is a model that classifies osteosarcoma by applying an image and a mask cut from only the humerus region (the proposed model described in Figure 4).

The performance of the deep learning model is shown in Table 3 below. The proposed model all showed better performance compared to the comparative models.

	EfficientNet	DenseNet	Resnet	ShuffleNetV2
Comparison model 1	0.82	0.793	0.804	0.785
Comparison model 2	0.843	0.819	0.834	0.833
proposed model	0.873	0.838	0.844	0.835

Figure 5 shows the results of evaluating the performance of a learning model for classifying humeral osteosarcoma. Figure 5 shows the results of generating CAM from the comparative model and the proposed model. In Figure 5, the annotated image is a chest X-ray image and an image with the tumor area annotated in the image. Figure 5(A) is the result of using EfficientNet, and Figure 5(B) is the result of using ShuffleNetV2. Looking at Figure 5, it can be seen that only the proposed model accurately captured the location of osteosarcoma and generated CAM.

Figure 6 is an example of an analysis device for classifying humeral bone tumors. The analysis device 500 corresponds to the above-described analysis devices (130 and 140 in FIG. 1). The analysis device 500 may be physically implemented in various forms. For example, the analysis device 500 may take the form of a computer device such as a PC, a network server, or a chipset dedicated to data processing.

The analysis device 500 may include a storage device 510, a memory 520, an arithmetic device 530, an interface device 540, a communication device 550, and an output device 560.

The storage device 510 can store chest X-ray images generated by x-ray equipment.

The storage device 510 may store a code or program for classifying bone tumors by analyzing chest X-ray images.

The storage device 510 may store a segmentation model that distinguishes a bone tumor area, which is a region of interest, in a chest X-ray.

The storage device 510 can receive a cut humerus image and store a deep learning model that classifies whether a bone tumor exists.

The memory 520 can store data and information generated while the analysis device analyzes the chest X-ray image.

The interface device 540 is a device that receives certain commands and data from the outside. The interface device 540 can receive a chest X-ray image from a physically connected input device or an external storage device. The interface device 540 may analyze the chest X-ray image and transmit the results of classifying the bone tumor to an external object.

Meanwhile, the interface device 540 may receive data or information transmitted via the communication device 550 below.

The communication device 550 refers to a configuration that receives and transmits certain information through a wired or wireless network. The communication device 550 can receive a chest X-ray image from an external object. Alternatively, the communication device 550 may analyze the chest X-ray image and transmit the results of classifying the bone tumor to an external object such as a user terminal.

The output device 560 is a device that outputs certain information. The output device 560 can output an interface required for the data processing process, an input chest X-ray image, a classification result, and an image in which the bone tumor location is visualized.

The calculation device 530 may classify the humerus region by inputting the input chest X-ray image into a segmentation model. The calculation device 530 may generate a humerus patch by cutting only an area of a certain size based on the midpoint of the humerus. For convenience of explanation, we assume that the deep learning model was learned based on the left humerus image. At this time, the calculation device 530 can create a humerus patch by leaving the left humerus intact and flipping the right humerus left and right in the chest X-ray.

The calculation device 530 can input the left humerus patch or the left and right inverted right humerus patch into a deep learning model. The deep learning model generates CAM as described above. The deep learning model extracts features from the input humerus patch and creates a CAM so that the pixels of the CAM excluding the humerus or bone tumor area are 0. In addition, the deep learning model can also output the results of classifying bone tumors based on the humerus patch.

The computing device 530 can control the CAM output from the deep learning model to be output to the output device 560. Additionally, the calculation device 530 may output information on whether the subject is normal or has a bone tumor to the output device 560 based on the probability value for the presence or absence of a bone tumor.

The computing device 530 may be a device such as a processor that processes data and performs certain operations, an AP, or a chip with an embedded program.

Additionally, the medical image analysis method or bone tumor classification method as described above may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a temporary or non-transitory computer readable medium.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), and EPROM (Erasable PROM, EPROM). Alternatively, it may be stored and provided in a non-transitory readable medium such as EEPROM (Electrically EPROM) or flash memory.

Temporarily readable media include Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), and Enhanced SDRAM (Enhanced RAM). It refers to various types of RAM, such as SDRAM (ESDRAM), synchronous DRAM (Synclink DRAM, SLDRAM), and Direct Rambus RAM (DRRAM).

This embodiment and the drawings attached to this specification only clearly show some of the technical ideas included in the above-described technology, and those skilled in the art can easily understand them within the scope of the technical ideas included in the specification and drawings of the above-described technology. It is self-evident that all inferable modifications and specific embodiments are included in the scope of rights of the above-described technology.

Claims (6)

1. A step in which an analysis device receives a chest X-ray image of a subject;

   A step where the analysis device inputs the chest and

   The analysis device inputs the humerus patch into a previously learned deep learning model, and outputs a CAM (Class Activation Map) for the humerus patch and whether or not it is a bone tumor,

   The deep learning model is learned so that the pixel of the CAM is 0 in the area excluding the humerus area in the humerus patch when the subject is normal, and when the subject has a bone tumor, the pixel of the CAM is set to 0 in the area excluding the bone tumor area in the humerus patch. A method for visualizing bone tumors in the humerus using chest X-ray images that are learned so that the pixels of are set to 0.
2. According to paragraph 1,

   The analysis device creates the humerus patch by cutting an area of a certain square size based on the center of the humerus region divided by the segmentation model, and generates the humerus patch after flipping the humerus image on one side from the chest X-ray left and right. A method of visualizing bone tumors in the humerus using chest X-ray images.
3. According to paragraph 1,

   The deep learning model is (i) trained so that the pixel of the CAM is 0 in the area excluding the humerus region in the CAM using the humerus patch of a normal person and a mask representing the humerus position in the learning data, and (ii) the bone tumor in the learning data A method of visualizing a bone tumor of the humerus using a chest
4. An interface device that receives the subject's chest X-ray image;

   A storage device that stores a segmentation model that distinguishes the humerus region in a chest X-ray image and a deep learning model that receives the humerus patch and generates a CAM (Class Activation Map) that visualizes the humerus region or bone tumor region; and

   The input chest Includes a calculation device that calculates CAM and bone tumor,

   The deep learning model is learned so that the pixel of the CAM is 0 in the area excluding the humerus area in the humerus patch when the subject is normal, and when the subject has a bone tumor, the pixel of the CAM is set to 0 in the area excluding the bone tumor area in the humerus patch. An analysis device that classifies bone tumors of the humerus that learns so that the pixel of is 0.
5. According to clause 4,

   The calculation device generates the humerus patch by cutting an area of a certain square size based on the center of the humerus region divided by the segmentation model, and generates the humerus patch after flipping the humerus image on one side from the chest X-ray left and right. An analysis device that classifies bone tumors of the humerus.
6. According to clause 4,

   The deep learning model is (i) trained so that the pixel of the CAM is 0 in the area excluding the humerus region in the CAM using the humerus patch of a normal person and a mask representing the humerus position in the learning data, and (ii) the bone tumor in the learning data An analysis device for classifying bone tumors of the humerus that is learned so that the pixels of the CAM are 0 in the area excluding the bone tumor area in the CAM using the subject's humerus patch and a mask representing the bone tumor location.

Images