KR20240026705A - Bone age prediction system and method - Google Patents

Abstract

The present invention relates to a bone age prediction system and method, wherein a receiver receives an original image of the bone shape, a preprocessor extracts features of the received original image, and a learner learns the extracted features using a deep learning algorithm, In a bone age prediction system that includes a bone age prediction unit that derives the patient's bone age from the original image input based on the learned features, the preprocessor divides the received original image into a plurality of instances, In order to extract features of each segmented instance, feature embeddings for each instance are derived as input to a pre-trained neural network, and weights for the feature embeddings of each instance are derived by inputting the derived feature embeddings to an attention-based algorithm. This is a configuration that generates fused features to predict bone age by calculating the feature embedding and weight of each instance, and derives the relationship between bone resorption of Tc-99m-HDP measured by whole-body bone scintigraphy and the aging process and age-related diseases. If possible, the efficiency of health care to prevent aging and age-related diseases can be improved.

Description

Bone age prediction system and method {BONE AGE PREDICTION SYSTEM AND METHOD}

The present invention relates to a bone age prediction system and method, and to a technology that allows bone age to be estimated using multi-instance learning and an attention network for an original image.

Bone age estimation (BAE) has several purposes. First, it can be used to investigate some physical problems in a child's growth. This operation can represent the biological structure of children's bones with radiological images obtained by irradiating X-ray images to the human body, but is not effective for people over 18 years of age. Additionally, BAE can be used to analyze and early predict the aging process and bone-related diseases.

Recently, a DL (Deep Learning)-based image analysis model was developed to predict bone age, which is a basic process for evaluating disease status. A LeNet-5 network based on CNN (Convolutional Neural Networks) was proposed that predicts bone age by receiving an input image of size 32 x 32 by resetting the size instead of 512 x 512. Support Vector Regression (SVR) can also be used to aggregate disparate features for the task of predicting bone age. Therefore, the combination of CNN and SVR-based models improves performance efficiency for BAE tasks when the data contains heterogeneous features.

As an example using different models, the BoNet architecture designed ad-hoc CNN for BAE utilizes deformation layers to handle non-rigid deformation of bone, and the low-complexity CaffeNet-based CNN model has a huge number of edges connecting neurons ( edge) and at the same time the neuron value is fixed, showing better performance in small-sized input images.

Additionally, we focused on using transfer learning on BAE with pre-training on the ImageNet dataset using starting blocks for the classification model, using a Google-Net model with 22 hidden layers for the task of bone age classification. There are research results that improve age prediction performance.

However, conventional models for predicting bone age require a lot of time to learn or reduce the size of images with large amounts of data because the images or images for learning are composed of approximately 900 x 3000 pixels, so the amount of data to be processed is large. The occurrence of data loss in the process remains a problem that needs to be solved. Additionally, when using full-body images, it is difficult to predict bone age because the model cannot focus on some important areas.

Republic of Korea Patent No. 10-2308257 (2021.10.05.) Republic of Korea Patent No. 10-2330216 (2021.11.25.)

The present invention is a bone age prediction system and method that divides the original image into several tiles, considers them as an instance set, and derives bone age through three main modules in order to minimize data loss occurring in the process of reducing image size. can be provided.

A bone age prediction system according to one aspect of the present invention includes a receiving unit that receives an original image of the bone shape; a preprocessor that extracts features of the received original image; A learning unit that learns the extracted features using a deep learning algorithm; and a bone age prediction unit that derives the patient's bone age from the input original image based on the learned features, wherein the preprocessor divides the received original image into a plurality of instances, and each of the divided instances. In order to extract features, feature embeddings for each instance are derived as input to a pre-trained neural network, and the derived feature embeddings are input to an attention-based algorithm to derive weights for the feature embeddings of each instance. Calculate feature embeddings and weights to generate fused features to predict bone age.

Preferably, the original image may be whole body bone scintigraphy.

Preferably, the preprocessor may divide the original image into a preset size.

Preferably, the preprocessor may derive instances excluded from feature extraction among the segmented instances.

Preferably, the preprocessor may input each instance into a neural network composed of a linear layer calculated with at least one of a hyperbolic tangent (tanh) function and a sigmoid (Sigm) function.

Preferably, the preprocessor applies a Sigsoftmax function to the output of at least one linear layer to derive at least one weight for each feature, wherein the weight is in the range of 0 to 1, and the sum of each weight is This could be 1.

Preferably, the preprocessor may assign different weights to each feature of the instance and derive important instances according to each assigned weight.

A bone age prediction method performed in a bone age prediction system according to another aspect of the present invention includes a reception step of receiving an original image of whole body bone scintigraphy at a receiver; Dividing the original image received from the preprocessor into a plurality of instances; Derivation of feature embeddings in which feature embeddings are derived by inputting each segmented instance from the preprocessor into a neural network that is calculated using at least one of the hyperbolic tangent (tanh) function and the sigmoid (Sigm) function. step; Weight derivation in which weights for the feature embeddings of each instance are derived as input to an attention-based algorithm that calculates the feature embeddings derived from the preprocessor using the Sigsoftmax function at the output stage to merge at least one linear layer. step; A fusion feature extraction step in which the preprocessor calculates feature embeddings and weights of each instance to generate fused features for predicting the patient's bone age; A learning step in which the learning unit learns the generated features using a deep learning-based algorithm to generate label data for bone age for each feature of each instance to build a prediction model; And a prediction step in which the patient's bone age is derived from the input original image based on the learned features in the bone age prediction unit.

According to the present invention, it is possible to derive the relationship between bone absorption of Tc-99m-HDP measured by whole-body bone scintigraphy and the aging process and age-related diseases, thereby improving the efficiency of health care to prevent aging and age-related diseases. You can.

1 is a configuration diagram of a bone age prediction system according to an embodiment.
Figure 2 is a schematic diagram showing the entire process of predicting bone age according to an embodiment.
Figure 3 is a diagram showing whole body bone scintigraphy according to one embodiment.
Figure 4 is a flowchart showing a method for predicting bone age according to an embodiment.

Hereinafter, the bone age prediction system and method according to the present invention will be described in detail with reference to the attached drawings. In this process, the thickness of lines or sizes of components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the operator's intention or custom. Therefore, definitions of these terms should be made based on the content throughout this specification.

The purpose and effect of the present invention can be naturally understood or become clearer through the following description, and the purpose and effect of the present invention are not limited to the following description. Additionally, in describing the present invention, if it is determined that a detailed description of known techniques related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

1 is a configuration diagram of a bone age prediction system according to an embodiment.

As shown in FIG. 1, the bone age prediction system according to one embodiment includes a receiving unit 100, a preprocessing unit 300, a learning unit 500, and a bone age predicting unit 700.

The receiving unit 100 may receive an original image of the bone shape. Here, the original image may be whole body bone scintigraphy (10), but is not necessarily limited to the above-described scintigraphy (10). For example, the original image may be an X-ray image.

The preprocessor 300 extracts features of the received original image. Here, the preprocessor 300 divides the received original image into a plurality of instances 20, and inputs each instance ( Derive feature embeddings for 20), derive weights for feature embeddings of each instance 20 by inputting the derived feature embeddings into an attention-based algorithm, and calculate feature embeddings and weights for each instance 20. Thus, fused features for predicting bone age can be created.

Additionally, the preprocessor 300 can segment the original image into a preset size and derive instances 20 excluded from feature extraction among the segmented instances 20. The preprocessor 300 may input each instance 20 into a neural network composed of a linear layer calculated with at least one of the hyperbolic tangent (tanh) function and the sigmoid (Sigm) function. , At least one weight for each feature is derived by applying the Sigsoftmax function to the output of at least one linear layer, where the weight ranges from 0 to 1, and the sum of each weight may be 1.

Additionally, the preprocessor 300 may assign different weights to each feature of the instance 20 and derive important instances 20 according to each assigned weight.

The learning unit 500 can learn the extracted features using a deep learning algorithm.

The bone age prediction unit 700 can derive the patient's bone age from the original image input based on the learned features.

Figure 2 is a schematic diagram showing the entire process of predicting bone age according to an embodiment.

As shown in Figure 2, bone age estimation (BAE) using whole-body bone scintigraphy (10) can consist of three interconnected networks to be trained, each of which will be composed of a feature extraction module, an attention module, and a prediction module. You can.

First, the original image can be divided into several instances (20) of size N × N. These instances 20 can be provided to the first module, which is InceptionV3 (no upper layers), pre-trained as a backbone network for feature extraction. During the training process, a beck of k instances (20) is fed to the feature extractor, which is used to generate k attention weights by the attention module using a neural network with two linear layers with tanh(.) and sigm(.) functions. Obtain k feature embeddings. The Sigsoftmax function is applied to the outputs of the two linear layers so that the attention weights are in the range (0, 1) and sum to 1. These attention weights are then combined with the feature embeddings to obtain the bag representation. The bag representation is further processed by a prediction module, which includes two fully connected (FC) layers and a final layer of single neurons that predict the patient's age through linear activation.

The original size of the image data is about 900 During the experiment, training the model using a bag containing all 30 instances (20) meant that the images contained a lot of background areas, so for each image we calculated the percentage of background values and used 10 instances where most pixels were background. An exclusion area (11) can be set for the instance (20). The preprocessor can improve learning performance by extracting features only from the instances 20 in the remaining extraction area 12 other than the instances 20 designated as the exclusion area 11.

Figure 3 is a diagram showing whole body bone scintigraphy according to one embodiment.

As shown in FIG. 3, in whole body bone scintigraphy 10 according to an embodiment, unlike general image classification that assumes that the image belongs to one class (label), partially different classes may exist. In this case, the multi-instance (20) model can be applied. The problem of weakly annotated data commonly arises in the field of medical imaging, where full-slide medical images have millions of pixels, and gigapixel images are too large for detailed annotation, making detailed annotation time-consuming and expensive. Therefore, these massive-sized images can be divided into multiple segments to train a deep learning model. In the Matrox Image Library (MIL), these images can be considered as a bag (set) with multiple instances (20), and we can process a model with a set of instances (20) and output a single label for the entire bag.

However, in the MIL problem, the instance 20 can be expressed as an input data function that does not require complex processing. Some tasks, such as computer vision or language processing, require additional feature extraction processes. Therefore, by parameterizing all procedures of MIL using a novel network, the entire process can be learned end-to-end through backpropagation. MIL pooling methods, which are max pooling and average pooling, have some disadvantages in that they are predefined and cannot be trained. To solve this difficulty, a neural network can be used to assign a weighted average to the instances 20, and Attention MIL pooling can be done using feature embeddings ( ) and attention weights ( ) can be combined and expressed in the following equation.

[Equation 1]

[Equation 2]

Feature embedding of Equation 1 ( ) can be expressed in Equation 2, and in Equation 1, W, V, and U are parameters, is element-wise multiplication, tanh(.) is a hyperbolic tangent function, and sigm(.) is a sigmoid function. The hyperbolic element tangent (tanh(.)) is used to retrieve the relationship between instances 20 by scaling the shape values in the range (-1,1), corresponding to both negative and positive values. However, using only one layer with the tanh(.) function may limit the final expression for complex relational problems. In this case, a gate mechanism can be used to handle this problem as described.

The first normalizes the features to values between 1 and 1 using the elementwise hyperbolic tangent (tanh(.)) and examines the similarities and differences between instances (20). The output of this linear layer is then multiplied element-wise with another linear layer using a sigmoid function that calculates an attention weight between 0 and 1 for each instance (20). The attention weights are then combined with the feature embeddings and MIL pooling is applied to create the final bag representation. Therefore, the model can improve training performance by focusing more on important cases. In other words, we can discover important instances (20) that have a greater impact on the final prediction. Additionally, attention-based MIL pooling can overcome the shortcomings of traditional pooling methods because it is more flexible and trainable.

The attention approach using softmax in Equation 2 has a bottleneck in capacity representation for the model, so Linear-Softmax has limitations in expressing probability distributions when the output space is much larger than the size of the feature embedding. Therefore, the sigsoftmax function can be used to solve this capacity limitation problem, which can be expressed as the following equation.

[Equation 3]

In Equation 3, attention-based MIL pooling assigns importance weights to instances 20 before merging them into the final bag representation. Accordingly, the prediction model can improve training performance by focusing more on the highly important instance 20 among the weighted instances 20.

Figure 4 is a flowchart showing a method for predicting bone age according to an embodiment.

As shown in Figure 4, the bone age prediction method according to one embodiment includes a reception step (S100), an instance segmentation step (S200), a feature embedding derivation step (S300), a weight derivation step (S400), and a fusion feature extraction step ( S500), a learning step (S600), and a prediction step (S700).

In the reception step (S100), the original image of whole body bone scintigraphy 10 is received by the receiver 100.

In the instance division step (S200), the original image received from the preprocessor 300 is divided into a plurality of instances 20 (Instance).

In the feature embedding derivation step (S300), the preprocessor 300 converts each of the divided instances 20 into a linear layer calculated with at least one of the hyperbolic tangent function and the sigmoid function. Feature embeddings are derived as input to the constructed neural network.

In the weight derivation step (S400), the feature embeddings derived from the preprocessor 300 are input to each instance (20 ) weights for feature embeddings are derived.

In the fusion feature extraction step (S500), the preprocessor 300 calculates the feature embedding and weight of each instance 20 to generate fused features for predicting the patient's bone age.

In the learning step (S600), a prediction model is constructed by learning the generated features using a deep learning-based algorithm to generate label data for bone age for each feature of each instance 20.

In the prediction step (S700), the patient's bone age for the input original image is derived based on the learned features.

Although the present invention has been described in detail through representative embodiments above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. will be. Therefore, the scope of rights of the present invention should not be limited to the described embodiments, but should be determined not only by the claims described later, but also by all changes or modified forms derived from the claims and the concept of equivalents.

100: Receiving unit 300: Preprocessing unit
500: Learning unit 700: Bone age prediction unit
10: Scintigraphy 20: Instance
11: Exclusion area 12: Extraction area

Claims (8)

A receiving unit that receives the original image of the bone shape;
a preprocessor that extracts features of the received original image;
A learning unit that learns the extracted features using a deep learning algorithm; and
In the bone age prediction system including a bone age prediction unit that derives the patient's bone age from the input original image based on the learned features,
The preprocessor,
Divide the received original image into multiple instances,
In order to extract features of each of the divided instances, feature embeddings for each instance are derived as input to a pre-trained neural network,
By inputting the derived feature embeddings into an attention-based algorithm, a weight for the feature embeddings of each instance is derived,
A bone age prediction system that generates fused features for predicting bone age by calculating feature embeddings and weights of each instance.
According to claim 1,
A bone age prediction system, wherein the original image is whole body bone scintigraphy.
According to claim 1,
A bone age prediction system wherein the preprocessing unit divides the original image into preset sizes.
According to claim 1,
The preprocessor extracts features from the divided instances. A bone age prediction system characterized by deriving excluded instances.
According to claim 1,
The preprocessor inputs each instance into a neural network composed of a linear layer operated by at least one of the Hyperbolic Tangent (tanh) function and the Sigmoid (Sigm) function. Bone age prediction system. .
According to claim 1,
The preprocessor applies the Sigsoftmax function to the output of at least one linear layer to derive at least one weight for each feature,
The bone age prediction system is characterized in that the weight ranges from 0 to 1, and the sum of each weight is 1.
According to claim 1,
A bone age prediction system wherein the preprocessor assigns different weights to each feature of the instance and derives important instances according to each assigned weight.
In the bone age prediction method performed in the bone age prediction system,
A receiving step in which an original image of whole body bone scintigraphy is received at a receiving unit;
Dividing the original image received from the preprocessor into a plurality of instances;
Derivation of feature embeddings in which feature embeddings are derived by inputting each segmented instance from the preprocessor into a neural network that is calculated using at least one of the hyperbolic tangent (tanh) function and the sigmoid (Sigm) function. step;
Weight derivation in which weights for the feature embeddings of each instance are derived as input to an attention-based algorithm that calculates the feature embeddings derived from the preprocessor using the Sigsoftmax function at the output stage to merge at least one linear layer. step;
A fusion feature extraction step in which the preprocessor calculates feature embeddings and weights of each instance to generate fused features for predicting the patient's bone age;
A learning step in which the learning unit learns the generated features using a deep learning-based algorithm to generate label data for bone age for each feature of each instance to build a prediction model; and
A bone age prediction method comprising a prediction step in which the patient's bone age is derived for the original image input based on the learned features in the bone age prediction unit.