CN107871316B - Automatic extraction method of hand bone interest region based on deep neural network - Google Patents

Abstract

A deep neural network-based automatic extraction method for the region of interest of hand bone in X-ray film, removes the part of the black background embedded text on both sides of the original hand bone X-ray film image, and uniformly brightens and removes the original hand bone X-ray film image. Noise operation; sample and train the model M1 to obtain the hand bone X-ray image without text Output2; normalize the size of Output2 to obtain Output3; sample and train the model M2, for the intersection of the hand bone, background, and hand bone background in Output3 Partial judgment; based on the model M2 to judge the image sliding window in Output3, according to the judgment value, the hand bone marker map Output4 is obtained; based on Output3 and Output4, the image Output5 with only the hand bone is obtained; Output5 is optimized to obtain the final hand bone interest area. The invention can automatically acquire the region of interest of the hand bone in the X-ray film.

Description

Automatic extraction method of hand bone interest region based on deep neural network

technical field

The invention relates to the field of medical image analysis and machine learning, in particular to a method for extracting regions of interest in human hand bone X-ray images, and belongs to the field of deep learning-based medical image analysis.

Background technique

Skeletal age, referred to as bone age, is determined by the degree of calcification of a child's bones. Bone age can more accurately reflect the developmental level of people at all ages from birth to full maturity. Not only that, but also in the analysis and diagnosis of endocrine diseases, developmental disorders, nutritional disorders, hereditary diseases and metabolic diseases. important role. Radiologists measure a child's bone age by comparing X-rays of the child's hand with their age-appropriate standard. The technology is stable and has been around for decades. With the improvement of parents' health awareness, the number of children undergoing bone age testing is increasing day by day, but the status quo of medical institutions has not changed: there are not many licensed doctors in the pediatric imaging department, and the reading efficiency is not high.

With GPU-accelerated deep learning technology, automatic detection of bone age through artificial intelligence has become possible. Deep learning requires a large number of hand bone X-ray images for training to achieve good detection results. However, the original hand bone X-ray image has a lot of useless information, such as text, noise, etc. In the original hand bone X-ray image, due to the inconsistent placement of each person's hands and the uneven illumination of the machine, X-ray images are not uniform. The position and brightness of the hand in the light image are not fixed, these factors will lead to poor learning effect of the detection model, resulting in large judgment errors. Therefore, for bone age prediction based on deep learning, it is crucial to accurately obtain hand bone regions in X-ray films and treat these hand bone regions as good samples that the detection model can learn from. However, manual labeling of hand parts is not only inefficient, but also the labeling results of different labelers will be different.

SUMMARY OF THE INVENTION

In order to overcome the shortcomings of low efficiency, large error and low precision of the existing X-ray film hand bone interest region extraction methods, the present invention proposes a deep neural network-based X-ray system with high efficiency, small error and high precision. The automatic extraction method of the region of interest of the hand bone in the light film can not only automatically obtain the hand bone region in the X-ray film, but also automatically remove noise and adjust the brightness.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

A method for automatically extracting a region of interest of a hand bone in an X-ray film based on a deep neural network, comprising the following steps:

Step 1: For the grayscale image of the original hand bone X-ray film, remove the black background embedded text on both sides of the image, so that most of the text is removed from the original image;

The second step is to perform a brightening operation on the grayscale image of the original hand bone X-ray film. In this step, the overall brightness of the image is evaluated first, and the image with insufficient brightness is subjected to a brightening operation. After the brightening, a denoising operation is performed to obtain an image. The set is called Output1;

Step 3, sample and train the model M1, which is used to remove the text near the hand bone and on the hand bone in the X-ray film in Output1, and obtain a grayscale image of the hand bone X-ray film Output2 without text;

Step 4: Normalize the size of all images in Output2. In order to keep the height and width consistent, first fill in the black background on both sides. When the width ratio of the image is larger than the height, the operation is to cut inward on both sides, and then Then reduce the image to (512,512), and call the new image set Output3;

Step 5: Sampling and training the model M2, which is used to judge the intersection of the hand bone, the background, and the hand bone background in Output3, and the size of the hand bone background is 16*16;

Step 6, with the model M2, judge the sliding window of the image in Output3, add the judged value to each pixel, and then each pixel according to the size of the value of the different judgement types obtained by itself, put the hand bone part. The pixel value of the hand bone is set to 255, and the pixel value of the background part is set to 0, so as to obtain the hand bone binary map, which is called Output4;

Step 7, compare the hand bone binary labeling map of Output4, and obtain the image with only the hand bone area on the basis of Output3, but at the same time, there will still be background impurities in the image, so it is necessary to perform a calculation of the maximum connected area here. Remove impurities to get Output5;

Step 8: Because the aperture around the image in Output5 is judged to be hand bone tissue, and it is connected to the largest connected area; since the length of the aperture is much larger than the width of the hand bone, therefore, for each image in Output5 The bottom part is used for difference comparison to remove the halo in the image and get the final hand bone region of interest.

So far, the description of the operation flow is completed.

Further, in the first step, the method of removing the embedded text in the black background is as follows: first convert the image into a numerical array, and then start to detect the two left and right columns toward the middle. Due to the particularity of pure white text and pure black background, if there is If the number of non-pure black is more than 10% of the entire column, it can be judged that this is the non-black background inlaid text part, then the previous part will be cut off.

计算该影像的整体亮度。这里考虑像素值大于120的像素点，对于不同的Aug值，用不同的参数进行提亮。Further, in the second step, the overall brightness evaluation process is as follows: set the resolution of an original image O to be M×N, and the value of each pixel point is t _ij , through the formula

Calculates the overall brightness of the image. Here, pixels with pixel values greater than 120 are considered. For different Aug values, different parameters are used to brighten.

Further, in the third step, for the training of the model M1, the sample collection process is as follows: intercepting 100*100 samples containing letters in the original grayscale image as positive samples; then intercepting samples of the same size that do not contain letters at all, As a negative sample; the process of building a two-dimensional convolutional neural network is:

Step 3.1 The input image goes through the Conv2D convolution layer to extract local features, and the input size is 100*100*1, followed by the relu activation function layer and the Maxpooling pooling layer.

Step 3.2 extracts three Conv2D convolution layers with different sizes, and the activation function layer and pooling layer structure are consistent with 3.1.

Step 3.3 connects the above 4-layer convolutional layer with the next fully connected layer through a Flatten layer.

Step 3.4 The above features pass through the first fully connected layer, and the internal sequence includes the Dense layer, the relu activation layer, and the Dropout layer to prevent overfitting. Next is the second fully connected layer, the internal sequence includes the Dense layer, and the sigmoid activation layer to get the output.

Among them, the SelectiveSearch (selective search) method is used to find letters in the image and judge, because of the particularity of letters, the letter L can be found by this method, and it is precisely positioned to a size of 100*100. Since other letters are mainly concentrated in the upper right corner of the image, and the text in the upper right corner will not affect the judgment of the hand bone area, after finding L and removing it, the upper right corner area is filled according to the average value of the surrounding background values.

In the fifth step, the training process of the model M2 is as follows: after removing the text interference in the fourth step, the model is used to distinguish the background, the hand bone, and the hand-bone-background intersection area. Sliding windows are used to sample these three categories respectively, which are defined as 0, 1, and 2, which correspond to the background, the hand-bone-background intersection area, and the hand bone respectively. The process of constructing a two-dimensional convolutional neural network model is as follows:

Step 5.1 The input image is extracted through a two-dimensional convolution layer to extract local features, and the input size is 16*16*1, followed by the relu activation function layer and the Maxpooling pooling layer;

Step 5.2 Extract a layer of Conv2D two-dimensional convolution layers with different sizes, and the activation function layer and Maxpooling layer structure are consistent with 5.1;

Step 5.3 Connect the above convolutional layer and the next fully connected layer through a Flatten layer;

Step 5.4 The above features pass through the first fully connected layer, the internal order includes the Dense layer, the relu activation layer, the Dropout layer to prevent overfitting, followed by the second fully connected layer, the internal order includes the Dense layer, and the softmax activation layer ( The output is three categories), and the output result is obtained.

In the step 6, the process of sliding window judgment is as follows:

Step 6.1 Perform a sliding window on the input image, 16*16, and the step size is 1;

Step 6.2 For the 16*16 patch, use the model M2 to judge, and the obtained value x adds 1 to the x value of the 16*16 pixels in the patch. It can be seen that each pixel has a process of counting the number of different x values. , define val[512][512][3] to count the number of different x values for each pixel;

Step 6.3 Define an output array result. For each point in the result, in the corresponding val array, only compare the number of types 0 and 2. If the number of types 2 is large, the corresponding point of the result is filled with 255, which is white. Otherwise, it is 0, black;

Since the sliding window adds the statistics corresponding to the pixels, the complexity is O(n×n×m×m). Since we only need a single-point query, the tree array is used to reduce the complexity, and the statistical complexity is reduced to O (n×n×log ² (n)), the single-point query complexity is log ² (n).

In the step 8, the process of difference comparison is:

Step 8.1: The image is first converted into a numerical array. Since the light and shadow exist at the bottom, only the cross section (patch) with a certain height upward from the bottom is operated, and the height is determined according to the average height of the light and shadow;

Step 8.2: Perform white statistics on each line in the patch, and keep the most white lines and the least white lines;

Step 8.3: For the two reserved rows, compare them column by column, one white and one black are different, and note the different quantity t;

Step 8.4: When t is greater than the set difference threshold, it is judged that there is light and shadow, so the part between the two lines is discarded, and if it is not greater than the threshold, it will remain unchanged;

Step 8.5: Do the calculation of the maximum connected area again. Since the part between the light and shadow and the hand bone is discarded, the hand bone area will be retained this time, and the light and shadow part will also be removed.

The technical idea of the present invention is as follows: using a deep neural network to detect the text information in the X-ray image, and using the deep neural network to classify the contents of different regions in the X-ray image, the obtained model can automatically and quickly extract the skinnyness of the hand area of interest.

In the process given in the present invention, the first model M1, for the most conspicuous letters L (representing the left hand), R (representing the right hand) and some residual text information in the initial X-ray image, is trained by sampling to make it match other The background is distinguished, so as to realize the task of removing letter information from the original image, which lays an important foundation for the subsequent extraction of the region of interest of the hand bone. The purpose of the second model M2 is to obtain the hand bone labeling map. The model distinguishes the interesting part and the uninteresting part in the input image by different labels. The present invention uses white and black to label. Since the text has been removed in the model M1, at this stage, the model is mainly used to distinguish three categories, the background, the hand bone, and the hand-bone-background intersection area. By sampling the sliding window of the input image, and then counting the detection results on each pixel, the marker map is obtained.

Compared with the traditional method of manually extracting the region of interest of the hand bone, the present invention has the following advantages: 1. The efficiency of obtaining the region of interest of the hand bone is greatly improved. 2. Unified processing can reduce errors caused by human operation. 3. More accurate results can be obtained than manual extraction. 4. Different tasks in different stages, the layered operation process reduces the risk of paralysis of the entire process caused by extreme pictures, and is easier to maintain and improve.

Description of drawings

Figure 1. Flow chart of extracting the region of interest of hand bone in X-ray film based on deep neural network.

FIG. 2 is a schematic structural diagram of a model M1 for removing characters.

Figure 3 is a schematic diagram of the structure of the model M2 used to construct hand bone marker mapping.

Figure 4 is a schematic diagram of the structure of the convolution module in model M1 and model M2.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

Referring to Figures 1 to 4, for the sampling work of Output1, the sliding window is mainly used to sample pictures with text information, especially the large letter L. By training the model M1, the detection of whether there is text information in the picture can be completed almost 100%. When Output1 is input into the model, a selective search (SelectiveSearch) is used to detect large letters as objects, and for other characters existing in the corners, a unified coverage strategy is adopted. This saves the time of sliding the original large image, and can effectively remove text information.

In the present invention, the image is normalized to a uniform size after removing the text information, which will reduce the time complexity of the subsequent acquisition of the marker map of the region of interest of the hand bone. For the acquisition of tag mapping, here is a detailed description:

1. Model M2 is able to classify 16*16 samples, which are divided into three categories: background, hand bone, and hand-bone-background intersection area.

2. Through the sliding window of Output3, the step size is 1, and a total of 500*500 samples are to be detected.

3. The detection results of each sample will be counted on the 16*16 pixel points of the sample for each category of detection results. So an array of 512*512*3 is needed for statistics. 0, 1, and 2 correspond to the background, the intersection area of the hand bone and the background, and the hand bone, respectively. Statistics are necessary because assuming that the previous sample is detected as the background and the next sample is detected as the intersection area, all pixels that differ by this step will eventually be considered as the background, so that the boundaries in the intersection area are divided. According to the statistical data of the three categories in the pixel, if the number of 0 categories is large, the pixel is set to 0 (black), otherwise it is 255 (white). In the end we can get the hand bone region (ROI) to be white and the background noise to be all black.

4. Due to the high time complexity of the algorithm, we use a tree-like array for optimization, and expand the regular tree-like array into a two-dimensional tree-like array.

5. After the marker map is obtained, it is inevitable that there will be some white noise. Here, the white noise is removed by calculating and retaining the maximum connected area. In addition, the light and shadow of the hand bone area and the bottom of the image sometimes overlap, and the light and shadow are very similar to the hand bone tissue, so that the model cannot accurately distinguish them. The specific method is: first convert the target area into a data array, calculate the row with the most white points and the row with the least white point, and then compare them column by column, either all of them are white or all black, otherwise they are regarded as difference. If the difference is greater than the threshold, it is considered to be caused by light and shadow. Discard the area between the two lines, and then do the calculation of the maximum connected area again to remove the noise caused by light and shadow.

Example: The hand bone X-ray image used contains an age range of 0 to 18 years, with a total of 944 samples. Take the 632 samples as the training set, train the model M1 and the model M2, and use the remaining 312 samples as the test set. The validation set and training set coincide. The construction and testing process of the two models are described below.

Model M1:

Step 1.1, build a deep convolutional neural network, the specific structure is shown in Figure 2.

Step 1.1.1: This convolutional neural network consists of 4 convolutional modules, a Flatten layer and two fully connected layers

Step 1.1.2: In the convolution layer, the size of the convolution kernel is 1*3, the input size is (1, 100, 100), and the number of convolution kernels increases with the depth of the network, which are 6, 12, 24, and 48 in turn. Each convolution module also includes a relu activation layer, and a Maxpooling pooling layer with a pool_size of (1, 2).

Step 1.1.3: Before the fully connected layer, there will be a flatten layer to flatten the output of the convolutional layer.

Step 1.1.4: In the fully connected layer, the first fully connected layer has a Dropout layer to prevent overfitting, the parameter is 0.5, and the subsequent fully connected layers are output through the sigmoid activation layer.

Step 1.2, data sampling and model training

Step 1.2.1: The X-ray images of the hand bone are all grayscale images, and the number of channels is 1. 500 images were sampled, each image was randomly sampled 99 samples plus a sample of the letter L. Then connect all the samples into a numpy array, the first dimension is the number of samples, thus a four-dimensional array of (50000, 1, 100, 100) is obtained as the training set; the test set is processed in the same way as (10000, 1, 100, 100). The validation set and training set coincide.

Step 1.2.2: The model adopts the batch training method. The number of samples in each batch of the training set generator and the verification set generator is 120, and a total of 200 rounds of training are used. The logarithmic loss function is used, and the rmsprop algorithm is used for optimization. The model only keeps the model with the highest accuracy.

Step 1.3, Model Testing

For the role of the model M2, please refer to the operation process for details, and will not be repeated here.

Model M2:

Step 2.1, build a deep convolutional neural network, the specific structure is shown in Figure 2.

Step 2.1.1: This convolutional neural network consists of 2 convolutional modules, a Flatten layer and two fully connected layers

Step 2.1.2: In the convolution layer, the size of the convolution kernel is 1*3, the input size is (1, 16, 16), and the number of convolution kernels increases with the depth of the network, 6 and 24 in turn. Each convolution module also includes the activation layer of relu, and the Maxpooling layer, and the pool_size is (1, 2).

Step 2.1.3: Before the fully connected layer, there will be a flatten layer to flatten the output of the volume base layer.

Step 2.1.4: In the fully connected layer, the first fully connected layer inputs the previous 24 outputs at 96 input points, and has a Dropout layer to prevent overfitting, the parameter is 0.05, and the subsequent fully connected layers pass softmax The activation layer output of , the output is three categories.

Step 2.2, data sampling and model training

Step 2.2.1: The X-ray images of the hand bone are all grayscale images, and the number of channels is 1. Sliding window sampling is performed on 500 images, 16*16 sliding window, step size is 32, for backgrounds that are too unique, step size is 8, and this part of the sample will be doubled to increase its performance in model training. Weights. As a result, a four-dimensional array of (100000, 1, 16, 16) is obtained as the training set, where the first dimension is the number of samples; the test set is processed in the same way as (20000, 1, 16, 16). The validation set and training set coincide.

Step 2.2.2: The model adopts the batch training method. The number of samples in each batch of the training set generator and the validation set generator is 120, and a total of 2000 rounds of training are used. The loss uses the multi-class logarithmic loss function, and the optimizer selects adam . The model only keeps the model with the highest accuracy.

Step 2.3, Model Testing

After the operations of the above steps, the extraction of the region of interest of the hand bone in the X-ray image of the hand bone can be realized.

The above-mentioned specific description further describes the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above-mentioned descriptions are only specific embodiments of the present invention, which are used to explain the present invention and are not intended to be used for The protection scope of the present invention is limited, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (7)

1. an automatic extraction method of X-ray hand bone region of interest based on deep neural network, is characterized in that: described method comprises the following steps: Step 1: For the grayscale image of the original hand bone X-ray film, remove the black background embedded text on both sides of the image, so that most of the text is removed from the original image; The second step is to perform a brightening operation on the grayscale image of the original hand bone X-ray film. In this step, the overall brightness of the image is evaluated first, and the image with insufficient brightness is subjected to a brightening operation. After the brightening, a denoising operation is performed to obtain an image. The set is called Output1; Step 3, sample and train the model M1, which is used to remove the text near the hand bone and on the hand bone in the X-ray film in Output1, and obtain a grayscale image of the hand bone X-ray film Output2 without text; Step 4: Normalize the size of all images in Output2. In order to keep the height and width consistent, first fill in the black background on both sides. When the width ratio of the image is larger than the height, the operation is to cut inward on both sides, and then Then reduce the image to (512,512), and call the new image set Output3; Step 5: Sampling and training the model M2, which is used to judge the intersection of the hand bone, the background, and the hand bone background in Output3, and the size of the hand bone background is 16*16; Step 6, with the model M2, judge the sliding window of the image in Output3, add the judged value to each pixel, and then each pixel according to the size of the value of the different judgement types obtained by itself, put the hand bone part. The pixel value of the hand bone is set to 255, and the pixel value of the background part is set to 0, so as to obtain the hand bone binary map, which is called Output4; Step 7, compare the hand bone binary labeling map of Output4, and obtain the image with only the hand bone area on the basis of Output3, but at the same time, there will still be background impurities in the image, so it is necessary to perform a calculation of the maximum connected area here. Remove impurities to get Output5; Step 8. Since there is a phenomenon that the aperture around the image in Output5 is judged as hand bone tissue, and it is connected to the largest connected area, since the length of the aperture is much larger than the width of the hand bone, the bottom of each image in Output5 is determined. Part of the difference is compared, so as to remove the aperture in the image, and get the final hand bone interest area. 2. The deep neural network-based automatic extraction method for the region of interest of the hand bone in the X-ray film as claimed in claim 1, wherein in the step 1, the method for removing the embedded text part in the black background is: first convert the image into Numerical array, the two most left and right columns start to be detected in the middle. Due to the particularity of pure white text and pure black background, if the number of non-black backgrounds that exist exceeds 10% of the entire column, it can be judged that the non-black background is embedded here. part, then the previous part will be completely cut off. 3. The deep neural network-based automatic extraction method for the hand bone region of interest in X-ray films as claimed in claim 1 or 2, wherein: in the step 2, the overall brightness evaluation process is: set the resolution of an original image O The rate is M×N, and the value of each pixel is t _ij , through the formula

Calculate the overall brightness of the image, consider pixels with pixel values greater than 120, and use different parameters to brighten for different Aug values. 4. The deep neural network-based automatic extraction method for the region of interest of hand bones in X-ray films as claimed in claim 1 or 2, characterized in that: in the step 3, for the training of the model M1, the sample collection process is: intercepting the original ash The 100*100 samples containing letters in the degree image are used as positive samples; then the samples of the same size that do not contain letters at all are taken as negative samples; the process of constructing a two-dimensional convolutional neural network is as follows: Step 3.1 The input image is extracted through the Conv2D convolution layer to extract local features, and the input size is 100*100*1, followed by the relu activation function layer and the Maxpooling pooling layer; Step 3.2 extracts three layers of Conv2D convolution layers with different sizes, and the activation function layer and pooling layer structure are consistent with step 3.1; Step 3.3 Connect the above 4-layer convolutional layer and the next fully connected layer through a Flatten layer; Step 3.4 The above features pass through the first fully connected layer, the internal sequence includes the Dense layer, the relu activation layer, the Dropout layer to prevent overfitting, followed by the second fully connected layer, the internal sequence includes the Dense layer, and the sigmoid activation layer, get the output result; The selective search method is used to find letters in the image and judge. Because of the particularity of the letters, the letter L can be found by this method, and it can be accurately located in the size of 100*100; since other letters are mainly concentrated in the upper right corner of the image, and The text in the upper right corner will not affect the judgment of the hand bone area, so after finding L and removing it, the upper right corner area is filled according to the average value of the surrounding background values. 5. The deep neural network-based method for automatically extracting regions of interest in X-rays of hand bones as claimed in claim 1 or 2, characterized in that: in the step 5, the training of the model M2, after removing the word interference through the step 3 , this model is used to distinguish background, hand bone and hand bone background intersection area; use sliding window to sample these three categories respectively, defined as 0, 1, 2 three categories, corresponding to background, hand bone background intersection area, hand bone ; The process of building a two-dimensional convolutional neural network model is: Step 5.1 The input image is extracted through a two-dimensional convolution layer to extract local features, and the input size is 16*16*1, followed by the relu activation function layer and the Maxpooling pooling layer; Step 5.2 Extract a layer of Conv2D two-dimensional convolution layers with different sizes, and the activation function layer and Maxpooling layer structure are consistent with 5.1; Step 5.3 Connect the above convolutional layer and the next fully connected layer through a Flatten layer; Step 5.4 The above features pass through the first fully connected layer, and the internal sequence includes the Dense layer, the relu activation layer, and the Dropout layer to prevent overfitting; followed by the second fully connected layer, the internal sequence includes the Dense layer, and the sotfmax activation layer, get the output. 6. The method for automatically extracting the region of interest of X-ray hand bone based on deep neural network as claimed in claim 5, it is characterized in that: in described step 6, the process of sliding window judgment is: Step 6.1 Perform a sliding window on the input image, 16*16, and the step size is 1; Step 6.2 For the 16*16 patch, use the model M2 to judge, and the obtained value x adds 1 to the x value of the 16*16 pixels in the patch. It can be seen that each pixel has a process of counting the number of different x values. , define val[512][512][3] to count the number of different x values for each pixel; Step 6.3 Define an output array result. For each point in the result, in the corresponding val array, only compare the number of types 0 and 2. If the number of types 2 is large, the corresponding point of the result is filled with 255, which is white. Otherwise, it is 0, black. 7. The deep neural network-based automatic extraction method for the hand bone region of interest in X-ray films as claimed in claim 1 or 2, wherein in the step 8, the process of difference comparison is: Step 8.1: The image is first converted into a numerical array. Since the light and shadow exist at the bottom, only the cross-sectional patch with a certain height upward from the bottom is operated, and the height is determined according to the average height of the light and shadow; Step 8.2: Perform white statistics on each line in the patch, and keep the most white lines and the least white lines; Step 8.3: For the two reserved rows, compare them column by column, one white and one black are different, and note the different quantity t; Step 8.4: When t is greater than the set difference threshold, it is judged that there is light and shadow, so the part between the two lines is discarded, and if it is not greater than the threshold, it will remain unchanged; Step 8.5: Do the calculation of the maximum connected area again. Since the part between the light and shadow and the hand bone is discarded, the hand bone area will be retained this time, and the light and shadow part will also be removed.

Images