EA045294B1 - METHOD FOR DETERMINING THE SEVERITY OF SKIN DISEASE BASED ON THE PERCENTAGE VALUE OF THE BODY SURFACE AREA COVERED BY LESIONS - Google Patents

Abstract

The invention proposes an image processing method that automatically calculates body surface area (BSA) estimates using machine learning methods. A Felzenschwalb image segmentation algorithm is used to identify proposed regions in each of a plurality of images in a training set. The images in the training set are resegmented, and each proposed region in each of the plurality of resegmented images in the training set is then classified as "lesion" or "non-lesion." A convolutional neural network (CNN) is then trained using the manually classified proposed regions in each of the plurality of images in the training set. The trained CNN is then used on control images to calculate BSA metrics.

Description

Prerequisites for creating the invention

Assessments of the severity of diseases such as psoriasis involve calculating the percentage of body surface area that is covered by lesions and inflammation (ie, the BSA (Body Surface Area) score). In the following, lesions and inflammation are collectively called lesions. BSA is the measured or calculated surface area of a person's body.

Psoriasis is an autoimmune skin disease that appears as red and inflamed areas that appear different from healthy, normal skin. An important component of disease severity measurements for psoriasis is monitoring what percentage of the body's surface area is covered by inflamed areas, called lesions. For plaque psoriasis, the two main measures of disease severity are the BSA and PASI (Psoriasis Surface Area and Severity Index), both of which involve the calculation of a percentage that is used to monitor disease progression and treatment effect (A Bozek, a. Reich (2017) Reliability of three psoriasis assessment tools: Psoriasis Surface Area and Severity Index, Body Surface Area and Physician Global Assessment Adv Clin Exp Med 2017 Aug;26(5):851-856 doi: 10.17219/acem/69804 ). Nowadays, these percentages are often assessed in a doctor's office by a dermatologist or nurse. One of the main problems associated with existing indicators of posoriasis is that they are imprecise and crude estimates, subject to subjective influence. Additionally, the process of calculating percentages to obtain an overall PASI score is labor-intensive and time-consuming. Another clinical need is that there is currently no ability to measure BSA for guttate psoriasis, in which the affected areas of the body are large and covered with many inflammatory lesions ranging from 2 to 10 mm, making measurement difficult by vision.

Accordingly, there is a need for a more objective and quantitative way to monitor skin inflammation using computational methods. The present invention satisfies such a need.

Summary of the invention

An image processing method is proposed that automatically calculates the BSA score using machine learning methods. The Felzenschwalb image segmentation algorithm is used to identify suggested regions in each of the multiple images of the training set. The training set images are resegmented and then each of the proposed regions in each of the plurality of resegmented training set images is classified as lesioned or unlesioned. A convolutional neural network (CNN) is then trained using manually classified proposed regions in each of the multiple images in the training set. The trained CNN is then used on the reference images to calculate BSA scores. Also included in the present invention is a device (or computer system) driven by computer instructions used in connection with the method, such as a computer device or medium for performing a method known in the art.

Short description

The patent or application file contains at least one color graphic material. Copies of this patent or patent application publication with color graphics may be obtained upon filing of an appropriate application and payment of a fee.

The above brief description, as well as the following detailed description of preferred embodiments of the invention, will be better understood when read in conjunction with the accompanying drawings. To illustrate the invention, the drawings show currently preferred embodiments. However, the invention is not limited to the precise systems and means provided. The graphic materials show the following.

In fig. Figure 1 illustrates psoriasis image segmentation using the Felzenswalb, Quickshift, SLIC and Compact watershed methods.

In fig. Figure 2 illustrates a psoriasis image correctly segmented using the Felzenswalb method.

In fig. Figure 3 illustrates three different types of imprecisely segmented psoriasis images: undersegmented, oversegmented, and full image segmented.

In fig. Figure 4 illustrates how the CNN filter improves the Felzenswalb segmentation results.

In fig. 5A illustrates image undersegmentation.

In fig. 5B illustrates image resegmentation.

In fig. 6 is a flow diagram of a computer-assisted method for determining the severity of a skin disease based on the percentage of BSA that is covered by lesions, in accordance with one preferred embodiment of the present invention.

- 1 045294

In fig. 7A and 7B are schematic diagrams of the system software and hardware for implementing FIG. 6.

Detailed Description of the Invention

Certain terminology herein is used for convenience only and should not be construed as limiting the present invention.

I. General description.

In fig. 6 is a flow diagram of a method using computer technology to determine the severity of a skin disease (eg, psoriasis) based on the percentage of BSA that is covered by the lesions. In fig. 7A and 7B are schematic diagrams of the system software and hardware for implementing FIG. 6.

With reference to FIG. 6, 7A and 7B the method is performed as follows.

Step 600: Perform image segmentation on the plurality of body surface area training set images using the Felzenswalb segmentation algorithm (FSA) and output the proposed regions in each of the plurality of training set images. Each of the plurality of images of the training set of body surface areas contains a skin disease. The image segmentation step is performed by the processor 700 shown in FIG. 7A.

Step 602: Resegment each of the plurality of images in the training set.

This step is also performed by processor 700.

Step 604: Manually classify each of the proposed regions in each of the plurality of resegmented images of the training set as lesioned or unlesioned. This step is performed by one or more human classifiers 702, shown in FIG. 7A.

Step 606: Train the neural network using the manually classified proposed regions in each of the plurality of images in the training set. This step is performed by the neural network 704 shown in FIG. 7A.

Step 608: Perform image segmentation on the body surface area reference image that contains the skin disease using the Felzenswalb segmentation algorithm and output regions in the body surface area reference image. The image segmentation step is performed by the processor 700 shown in FIG. 7B. Processor 700' may be the same processor as 700 or a different processor.

Step 610: Re-segment the reference image. This step is also performed by processor 700'.

Step 612: Input regions of the resegmented reference image into the trained neural network, designated 704' in FIG. 7B because it is the same neural network as neural network 704 in FIG. 7A, except that she is now trained.

Step 614: Use the trained neural network 704' to identify and filter out non-lesional regions from the re-segmented reference image, with the remaining regions of the re-segmented reference image being classified as lesion regions.

Step 616: Calculate the percentage of BSA in the reference image that is covered by lesions using the areas of the classified lesion regions of the re-segmented reference image, as well as the areas identified as non-lesional regions of the re-segmented reference image. This step is performed in processor 700. Processor 700 may be the same as processor 700 or processor 700' or other.

II. Detailed description.

The detailed description of the invention below describes the experimental process that led to the present invention and explains approaches that have been more successful than others.

The present invention solves the problem of calculating the surface area of a body during image segmentation. Image segmentation has become one of the cornerstone issues in computer vision and is defined as the process of dividing digital images into multiple segments, thereby organizing image data into meaningful pieces of data. A number of image segmentation methods have been explored that can effectively calculate the body surface area of a patient with psoriasis. Preferred embodiments of the invention claimed herein use the Felzenschwalb image segmentation algorithm [3] and convolutional neural networks (A. Krizhevsky, I. Sutskever, G.E. Hinton (2012) Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing (NeurIPS) , 2012) as a false positive filter to improve Felzenschwalb segmentation results.

Several image processing techniques were tested to automatically calculate the BSA score using machine learning techniques applied on 117 images of guttate and plaque psoriasis downloaded from the Internet. Compared to some other unsupervised segmentation methods, the Felzenschwalb image segmentation algorithm showed the highest percentages of correct segmentations of lesion regions compared to non-lesion regions with 56% of images having good segmentation while 44%

- 2,045,294 images are over- or under-segmented. To improve the segmentation results, a convolutional neural network (CNN) based on the visual geometry group (VGG) architecture was implemented to further filter out false-positive lesions from the proposed regions of the Felzenschwalb algorithm. The training data for the CNN consisted of human curated datasets of lesioned and unlesioned regions output from the Felzenschwalb algorithm. The CNN was achieved with a 90% 5-fold cross-validation accuracy score in classifying between lesioned and non-lesioned. This CNN filter, used in conjunction with Felzenschwalb's algorithm, accurately segmented 77% of the training data, or 67 of 86 guttate psoriasis images. This method is useful for digitizing disease severity measurements and remote monitoring of skin diseases such as psoriasis for patients and doctors.

Image data for psoriasis.

To collect the dataset, approximately 300 images of guttate psoriasis and 100 images of chronic plaque psoriasis were collected from Google images. Inaccurate and/or erroneous image data were filtered out, leaving a final data set of 86 guttate psoriasis images and 31 chronic plaque psoriasis images.

Psoriasis image segmentation methods.

Five different image segmentation algorithms were tested, all implemented using the Scikit-learn library and the OpenCV library in Python. The five algorithms tested were Felzenschwalb, Quickshift, SLIC, Compact watershed, and the Otsu threshold algorithm (for a review, see D. Liu, B. Soran, G. Petrie, and L. Shapiro. A review of computer vision segmentation algorithms. Lecture notes , 53, 2012). After visual inspection of example image pairs, it became apparent that Felzenszwalb's method (P.F. Felzenszwalb and D.P. Huttenlocher (2004) Efficient Graph-Based Image Segmentation. International Journal of Computer Vision 59(2), 167-181) provided the most promising long-range segmentation.

In fig. Figure 1 shows the output images of four different segmentation algorithms. The yellow boundaries indicate the segmentation of the estimated (suggested) region by the algorithm. After these tests, it was decided to segment all images using the Felzenswalb algorithm.

Psoriasis segmentation and binary classification filtering algorithm

To improve Felzenschwalb's segmentation results, a new algorithm based on a convolutional neural network filter was used, which removes false positive segmentation results. This algorithm is inspired by and has similarities to region-based convolutional neural networks (R-CNNs), which are currently becoming famous in the art, in solving problems associated with image segmentation. The algorithm works as follows.

1. Train the neural network to distinguish between areas with and without lesions among the proposed areas from the Felzenschwalb segmentation algorithm.

2. Resegment the images by increasing the k-value parameter to the optimal threshold value (k = 250).

3. Use a neural network to filter out the non-lesional regions suggested by Felzenschwalb segmentation.

Re-segmentation of the image in step 2 ensures that most of the lesion regions are included within the segmentation. The pre-trained neural network in step 1 is then used to filter out redundant non-lesional regions or false positives to distinguish non-lesional regions from lesioned regions in stage 3.

Training data.

The proposed regions emerging from the Felzenschwalb segmentation algorithm (with k-value set to 250) were used to resegment the image, thereby including as many true positive lesion regions as possible. Approximately 30 images of guttate psoriasis were then selected to generate approximately 3000 images of the proposed areas. Each of the 3000 proposed regions was then manually classified (human) as either affected or unaffected. To test the accuracy of this data set, the process was repeated three times. Some of these unlesioned areas were easily identified, including large areas covering large areas of skin, black background areas, loops, noise, and the like. Other areas were more difficult to distinguish between lesioned and non-lesioned areas, including areas that had shadowing, poor lighting, sutures, etc. A large source of error is thought to be attributed to the inconsistency in the data set in these more difficult regions to classify lesion versus non-lesional regions. This may be another explanation for why the binary classification results are still not in close proximity to the results reported in the Modified National Institute of Standards and Technology (MNIST) or Canadian Institute for Advanced Research (CIFAR) datasets.

Experiments on neural networks with binary classification.

To train the neural network models, a procedure similar to that described below was used.

- 3 045294

First, all input proposed regions were preprocessed by resizing all images to a certain constant pixel size using the cubic interpolation method from the onencv library. Various parameters of the neural network model were then changed, including model architecture (dense neural network vs. convolutional neural network, hidden layer sizes, batch normalization), parameters (learning rate), and image input pixel sizes (4 x 4, 8 x 8, 16 x 16, 32 x 32, 64 x 64). Finally, all models were tested with a training test data split factor of 80–20 to determine the accuracy, log loss, and root mean squared error estimates of the respective models. This means that 80% of the data was used to build the model, and 20% of the remaining data (which was not visible to the model) was used to evaluate the predicted performance of the model.

The final CNN model that was selected and based on the VGG architecture did not use batch normalization, had smaller hidden layers (both in the number of hidden layers and the width of the hidden layers), had a learning rate of 1e-4, and used the Adam optimizer. (Adam is an adaptive learning rate optimization algorithm that was designed specifically for training deep neural networks). To ensure the accuracy of the final model, the model was tested using 5-fold cross-score and an average 5-fold training accuracy score of 94% and an average 5-fold validation accuracy score of 90% were obtained.

Conditions that improve accuracy results.

Three approaches were used that significantly improved the binary classification accuracy of the neural network.

The first approach was a cubic interpolation of the shape of the input images with a constant size of 16 x 16 pixels. Initially, when the image size was set to 64 x 64, the classification accuracy at extremely low speeds was 60%. As the image size decreases, it was found that a standard small dense neural network would provide better and better classification performance until reaching the optimal image size of 16 x 16. The reason for this may be that most of the proposed regions were around 16 x 16 in size. Thus Thus, interpolating more information may result in the region receiving erroneous information, and interpolating less information may result in the loss of important information.

The second approach was to use convolutional neural networks as opposed to dense neural networks. It is well known that convolutional neural networks outperform dense neural networks in image classification tasks for a variety of reasons.

The third approach was that less is more when creating convolutional neural network model architectures. For example, if the dimensions of the hidden layers were reduced, faster training and more accurate validation estimates were achieved. In particular, when reducing the latent size of the first dense layer, it is especially important to reduce a significant number of parameters. This may be because parameter redundancy can prevent a hole from appearing in the model and cause redundant effects. This redundancy issue can be best seen by the nearly 10% loss in accuracy between training and testing validation estimates in large VGG models.

Conditions that did not affect or degrade segmentation results.

Several approaches did not improve accuracy results. In particular, three approaches had negligible or negative effects on the models.

Firstly, it was believed that adding layers of batch normalization, as seen in VGG models, improves the results, but as shown in some papers (S. Santurkar, D. Tsipras, A. Ilyas, A. Madry (2018) How Does Batch Normalization Help Optimization? Advances in Neural Information Processing (NeurIPS), 2018), batch normalization does not appear to improve classification accuracy in all cases.

Second, adjusting the learning rate did not improve performance. After testing rates of 0.01, 0.001, 0.0003, and 0.0001, it was found that the learning rate resulted in a negligible difference in performance between the learning rate settings of 0.001, 0.0003, and 0.0001.

Third, a deeper and wider network did not improve the accuracy of the model. Apparently, higher-density neural network models and convolutional neural network models are worse by even up to 10% in cross-validation testing than the simple small VGG models that have been studied. This seems to confirm that the fundamental principle of agile development is simplicity.

Results and discussion.

i. Advantages and disadvantages of image segmentation algorithms.

After testing a small sample of images on five different algorithms, it became clear that Felzenswalb's algorithm had more superior results than the other five unsupervised

- 4 045294 algorithms, as shown in FIG. 1. One of the important characteristics of this method is its ability to preserve detail in low variability regions of images while ignoring detail in high variability regions. Additionally, it is fast (< 1 s for a 512 x 512 image) with a runtime of O(n log n), where n is the number of pixels. Based on these observations, the Felzenschwalb method was chosen as the main method used for imaging psoriasis.

ii. Felzenschwalb image segmentation evaluation measure and results.

In fig. Figure 2 shows an example of good image segmentation. Good image segmentation is defined as any image segmentation that does not miss obvious significant lesion-containing parts that would need to be classified by a conventional clinician.

In fig. Figure 3 shows three examples of inaccurate image segmentation. The first image is an example of a re-segmented area. To reduce this problem, the k-value parameter should be reduced. The second image is an example of an under-segmented area. To mitigate this problem, the k-value parameter should be increased. The last example is an instance of large areas classified as lesions. To mitigate this, the k-value parameter should be increased. In fig. 5A and 5B also show examples of undersegmentation and oversegmentation.

The original Felzenschwalb algorithm was able to segment 49 out of 88 input images with good segmentation, 30 out of 88 input images were undersegmented, and 9 out of 88 input images were oversegmented. At the end of each segmentation result, BSA score calculations were obtained.

iii. Neural networks based on filtering results.

Different neural network models were tested with different parameters, architectures and input sizes. Test accuracies were calculated for each model. Setting lower image dimensions, choosing VGG-based convolutional neural networks and a learning rate of 1e-4 produces the best model with the highest testing accuracy of 0.9.

iv. Convolutional neural network and Felzenschwalb filter image segmentation results.

After implementing Felzenschwalb image segmentation filtered with a convolutional neural network, 67 out of 86 images achieved good segmentation, 16 out of 86 images undersegmented, and 3 out of 86 images oversegmented. Two input images were removed from the original Felzenschwalb image set due to inaccurate representations of guttate psoriasis. An example of a convolutional neural network with improved segmentation results can be seen in the images of Fig. 4.

In the image on the left shown in FIG. 4, the convolutional neural network was made to filter out large regions on the left side of the hand, all white background regions, as well as many regions with little noise, which were manually classified as unlesioned regions. With improved training data and more extensive training, the neural network model can achieve even better filtering results and can be used as an effective complement to the Felzenschwalb segmentation algorithm to remove false positive regions without lesions.

Conclusions.

As a result, it was found that the Felzenschwalb image segmentation algorithm provided a good initial diagnosis algorithm, which can effectively calculate the surface area of the patient's body injury. It was also found that convolutional neural networks can classify with high accuracy between putative lesioned and unlesioned regions from Felzenschwalb segmentation, given a good training dataset. Combined with these two results, Felzenschwalb's image segmentation algorithm combined with a convolutional neural network was shown to have a more accurate initial diagnosis by computing image segmentation and thus estimating body surface area for psoriasis disease.

The methods described above for calculating BSA can be used to create a system for calculating a digitized psoriasis disease score. For example, by training similar convolutional neural networks with 5 input images of the anterior body, back, anterior leg, posterior leg, and head regions to automate a complete PASI scoring system, it can output a PASI severity index in the range of 0-72. Such computer systems, for example, were recently proposed in (S. Fink, L. Uhlmann, S. Kutter, et al. (2018) Automatic, computer-controlled PASI measurements using digital image analysis compared with conventional calculating clinicians: a study protocol for comparative analysis, single center, study BMJ Open 2018;8:e018461.doi:10.1136/bmjopen-2017-018461), can help doctors create better, faster and more informed decisions when diagnosing and managing skin diseases such as psoriasis.

Those skilled in the art will appreciate that changes can be made to the embodiments described above without departing from the general concept of the invention. Therefore, it should

-

Claims (10)

It is understood that the present invention is not limited to the specific embodiments described, but is intended to cover modifications within the spirit and scope of the present invention. CLAIM 1. A method using computer technology for determining the severity of a skin disease based on the percentage of body surface area (BSA, Body Surface Area) that is covered with lesions, comprising: (a) performing image segmentation into the plurality of images of the BSA training set using a segmentation algorithm, wherein each of the plurality of images of the BSA training set contains a skin disease, wherein the image segmentation performed using the segmentation algorithm outputs proposed regions in each of the plurality of images of the training set; (b) classification of each of the proposed areas as affected or unaffected; (c) training a neural network using the classified proposed regions in each of the plurality of images in the training set; (d) performing image segmentation on the BSA reference image that contains the skin disease using a segmentation algorithm, wherein the image segmentation performed using the segmentation algorithm outputs regions in the BSA reference image; (e) inputting the reference image region into the trained neural network; (f) using a trained neural network to identify and filter out non-lesional regions from the reference image, with the remaining regions of the reference image being classified as lesion regions; and (g) calculating the percentage of BSA in the control image that is covered by lesions using the areas of classified lesion areas of the control image as well as the areas identified as non-lesional areas of the control image. 2. The method according to claim 1, wherein both the training set images and the control image are re-segmented. 3. The method according to claim 1 or 2, wherein the neural network is a convolutional neural network. 4. The method according to any of the previous paragraphs, in which, before step (c) of training, the plurality of image regions of the training set are resized to an image interpolation size of approximately 16 x 16 pixels, and before step (e), the control image regions are resized to an image interpolation size of approximately 16 x 16 pixels. 5. The method according to any of the previous paragraphs, in which the Felzenswalb segmentation algorithm is used. 6. The method as claimed in any one of the preceding claims, wherein step (b) includes the following steps: resegmenting each of the plurality of images of the training set and manually classifying each of the proposed regions in each of the plurality of resegmented images of the training set as having a lesion or not. 7. The method as claimed in any one of the preceding claims, further comprising the step of re-segmenting the reference image before inputting the region of the re-segmented reference image into the trained neural network. 8. The method of any one of the preceding claims, wherein both the training set images and the reference image are re-segmented by increasing the k-value parameter to about 250. 9. The method as claimed in any one of the preceding claims, wherein the skin disease is psoriasis. 10. A computer system containing a processor and a carrier of computer instructions, upon execution of which by the processor the method according to any one of claims 1 to 9 is performed. -