EP4670116A1

EP4670116A1 - DETERMINATION OF THE SEVERITY OF GINGIVITIS

Info

Publication number: EP4670116A1
Application number: EP24706419.9A
Authority: EP
Inventors: Reevu MAITY; Sreedha B; Ravindra Balasaheb Patil; Venkata Rama Srinivas GUTTA
Original assignee: Koninklijke Philips NV
Current assignee: Koninklijke Philips NV
Priority date: 2023-02-22
Filing date: 2024-02-19
Publication date: 2025-12-31
Also published as: WO2024175551A1; CN120752669A

Abstract

The subject-matter of the present disclosure relates to a computer-implemented method of determining a severity of gingivitis from an image showing a gum. The computer-implemented method comprises :receiving the image showing the gum; for each unit area of the gum, from amongst a plurality of unit areas of the gum, determining a colour; clustering, using a clustering algorithm, the plurality of unit areas according to the respective determined colours; calculating a distance of the colour of each cluster from a reference colour; calculating an overall distance value for the gum based on the distances of each cluster from the reference colour and the respective size of each cluster; and determining a severity of gingivitis based on the overall distance.

Description

DETERMINING A SEVERITY OF GINGIVITIS

FIELD

[01 ] The subject-matter of the present disclosure relates to computer-implemented methods of determining a severity of gingivitis from an image showing a gum, a computer- implemented method of generating training examples of images of gums, a computer- implemented method of training a machine learning model to segment an image showing a gum to extract the gum from the image, and a transitory, or non-transitory, computer- readable medium.

BACKGROUND

[02] Gingivitis is a common early form of gum disease affecting the gingiva, which is the region across the gumline and between the crowns of the teeth. Some common symptoms involve inflammation, swelling and irritation of the gingiva. Typically, there are two types of gingivitis, namely plaque induced gingivitis and non-plaque induced gingival lesions. Plaque is a sticky film of bacteria that accumulates on teeth, if proper oral hygiene is not maintained regularly. In plaque induced gingivitis, plaque forms due to the interaction of starch and sugar in food with the bacteria present in the mouth. The plaque hardens over time to form tartar which accumulates bacteria and acts as a protective shield for the bacteria resulting in irritation along the gum line. Eventually, the untreated irritation can lead to inflammation of gingiva and this symptom is one of the primary causes of gingivitis. Non-plaque induced gingival lesions can result from fungal infections and allergic reaction to certain foreign bodies such as dentures.

[03] The primary cause for development of gingivitis in gums is due to poor oral hygiene. A healthy lifestyle involving brushing the teeth twice daily and regular dental check-ups can prevent and in most cases, reverse the ailment. The color of healthy gum is typically pale pink, and it is firmly fitted with the teeth. However, when the gums of an individual are infected with gingivitis, it turns dusky or dark red color. In addition, the gums become swollen, tender and bleed on probing or brushing the teeth. Gingivitis is also characterized by edema, ulceration and thickening of the gingival tissue. If left untreated, the symptoms can lead to a more serious gum disease called periodontitis and eventual tooth loss. Research in gum diseases have shown the bacteria responsible for periodontitis can enter the blood stream through the gum tissue and thereby affects the lungs, heart and other critical organs.

[04] Gingivitis is diagnosed by examining if there is any inflammation in the gums. The dentist also may inspect the medical history of the patient, including whether the patient is diabetic or addicted to smoking. H.Loe and J.Silness, Periodontal disease in pregnancy I. Prevalence and severity. Acta odontologica scandinavica, 21 (6), 533-551 (1963), describes a gingival index score which indicates the stage or seventy of gingivitis. The score measures the gum health status in a scale of 0-3, with 0 indicating healthy gum and a score of 3 suggesting severe gingivitis characterized by bleeding and edema.

D. M. Alalharith et al., A deep learning-based approach for the detection of early signs of gingivitis in orthodontic patients using faster region-based convolutional neural networks, International Journal of Environmental Research and Public Health, 17(22), 8447 (2020), proposed an Al based approach for detecting the inflammation of gingiva. Their model comprises of two faster region-based convolutional neural networks (R-CNN) with ResNet-50 backbone. The first model detects the teeth, and second model detects the inflammation of gingiva. The College of Dentistry at Imam Abdulrahman bin Faisal University collected a gum dataset of 134 images. In each image, the distal, middle, and mesial gingival of the maxillary central incisors are annotated as shown in Figs. 1 (a) to (c). More specifically, in Figure 1 , the predicted inflamed and non-inflamed bounding boxes in distal (marked as 1), middle (marked as 2), and mesial (marked as 3) regions. In Figure 1 (a), inflamed and noninflamed region are correctly detected. In Figure 1 (b), non-inflamed distal region of the tooth p is incorrectly detected as inflamed. In Figure 1 (c), distal region of tooth r is not detected, non-inflamed middle region of tooth s is incorrectly detected as inflamed.

[05] The study focused on detecting gingivitis in a specific region in the upper gum which is the gumline around the maxillary central incisors. A Faster R-CNN model is trained to predict whether the distal, middle and mesial regions are inflamed. The work of Alalharith et al. has two major drawbacks: a) the Al model predicts gingivitis in a very specific region in the upper gum. The model cannot detect inflammation in other regions of upper and lower gums, b) the Al model cannot predict the stage of gingivitis or severity of inflammation in the gums, which can aid the user to avail appropriate treatment options. A. Rana et al., Automated segmentation of gingival diseases from oral images, in IEEE Healthcare Innovations and Point of Care Technologies, pp.144-147 (2017), developed an autoencoder with convolutional neural network (CNN) architecture to predict the inflamed regions in the gingiva. The oral dataset is collected from dental patients in Nashik, India aged between 18-90 years. In each image, the plaque is represented with yellow and orange shades and gingivitis is indicated by magenta and red shades. A dental expert annotated the ground truth mask of the inflamed region in each image. The dataset comprises of 400 images and the corresponding mask annotation of inflamed regions in the gums. The autoencoder based CNN model is trained to learn the mapping between the input image and ground truth gingivitis segmentation. In Fig. 2, the trained classifier predicts a pixel-wise segmentation of gum regions with gingival inflammation. More specifically, in Figure 2(a), each row shows an image showing a gum sample from a particular individual. Figure 2 (b) shows a ground truth label of gum inflammation by a dental expert. Figure 2(c) shows an inflamed region predicted by Al model. Figure 2 (d) shows prediction errors indicated by TP, FP and FN.

[06] One of the main drawbacks of the above approach is the oral dataset comprises of images with highly localized regions in the gum, which makes it difficult to train an Al model for predicting the inflamed segments in the entire region of both upper and lower gums. Another disadvantage is the model cannot predict the degree of inflammation in gums.

[07] It is an aim of the subject-matter of the present disclosure to improve on the prior art.

SUMMARY

[08] According to a first aspect of an embodiment of the present invention, there is provided a computer-implemented method of determining a severity of gingivitis from an image showing a gum, the computer-implemented method comprising: receiving the image showing the gum; for each unit area of the gum, from amongst a plurality of unit areas of the gum, determining a colour; clustering, using a clustering algorithm, the plurality of unit areas according to the respective determined colours; calculating a distance of the colour of each cluster from a reference colour; calculating an overall distance value for the gum based on the distances of each cluster from the reference colour and the respective size of each cluster; and determining a severity of gingivitis based on the overall distance. [09] Advantageously, by using the overall distance to determine the severity of gingivitis, it is possible to determining the severity of gingivitis in a continuous manner. In other words, the difference in severity between images can be assessed at a more granular level. In addition, no metadata relating to identifying the user is required since an embodiment of the invention will work on a single image showing the gum.

[10] In an embodiment, the computer-implemented method further comprises sending the determined severity of gingivitis to a display for displaying the severity of gingivitis to a user.

[11 ] In an embodiment, the clustering algorithm is a k-means clustering algorithm.

[12] In an embodiment, the calculating a distance of the colour of each cluster from a reference colour comprises: assigning a colour space value to the colour of the cluster; and determining a distance between the colour space value of the colour of the cluster and a colour space value of the reference colour, wherein optionally the colour space values are hexadecimal numbers. Using a colour space is a particularly efficient way of labelling the colours for use by the various machine learning algorithms.

[13] In an embodiment, the calculating the distance of the colour of each cluster from a reference colour comprises: calculating a Euclidian distance of the colour of each cluster from the reference colour. The Euclidian distance proved to be the most useful in comparing different colours in comparison to other approaches that were attempted.

[14] In an embodiment, the overall distance value is a weighted average distance, WAD, defined by: WAD = Sies/i ■ d_it wherein is a fraction of an area of an i-th cluster compared to a total area of the plurality of unit areas of the gum, di is the distance of the colour of the /-th cluster from the reference colour, S is a set of indices of colour clusters present in the gum image, and Sies/i = 1- Using the WAD is more computationally efficient than SWAD (see below) because fewer parameters are required.

[15] In an embodiment, the computer-implemented method further comprises: generating a gingival score for each cluster based on the distance, wherein the overall distance value is a selected weighted average distance, SWAD, defined by: SWAD = - iesfi ’ ^ di, wherein is a fraction of an area of an i-th cluster compared to a total area of the gum, d_t is the distance of the colour of the i-th cluster from the reference colour, e^Ci is a weight where Ci is the gingival score for the i-th cluster, S is a set of indices of colour clusters present in the gum image, and Sies/i = 1-

[16] Using the SWAD provides more accurate diagnosis of local gingivitis because each cluster is additionally weighted according to its own gingival score.

[17] In an embodiment, each unit area of the plurality of unit areas is a pixel and wherein the determining the colour comprises determining a value of the pixel using a red, green, blue, colour model or a light, red, green, blue, and yellow, colour model. The term light may be replaced with lightness to conform to the terminology of a CIELAB colour space. Pixel level unit areas makes colour determination easier and more computationally efficient.

[18] In an embodiment, the computer-implemented method further comprises: prior to determining a colour of each unit area, segmenting the image to extract only the gum using a machine learning model.

[19] Extracting only the gum from the image reduces the risk of the clusters being affected by other features of a mouth, e.g. a tongue, which may also be varying shades of red and pink.

[20] In an embodiment, segmenting the image to extract only the gum from the image using the machine learning model comprises: segmenting the image to extract an upper gum region of interest of the gum using a first machine learning model of the machine learning model; and segmenting the image to extract a lower gum region of interest of the gum using a second machine learning model of the machine learning model.

[21 ] In an embodiment, the computer-implemented method further comprises: filtering the clusters to remove one or more clusters having a distance of the colour from the reference colour that is greater than a threshold distance, wherein the threshold distance corresponds to a colour associated with a healthy gum.

[22] It is more computationally efficient to remove clusters having colours associated with healthy gums before calculating the overall distance rather than after

[23] In an embodiment, the computer-implemented method further comprises: determining an overall gingival classification as: severe gingivitis if the overall distance is less than a first threshold distance; moderate gingivitis if the overall distance is greater than the first threshold distance and less than a second threshold distance, where the second threshold distance is greater than the first threshold distance; and mild gingivitis if the overall distance is greater than the second threshold distance and less than a third threshold distance, where the third threshold distance is greater than the second threshold distance.

[24] According to an aspect of the present invention, there is provided a computer- implemented method of generating training examples of images of gums for use in training a machine learning model to segment images of gums to extract gums therefrom, the computer-implemented method comprising: receiving, by a cycle generative adversarial model, a plurality of images labelled as healthy gums, and a plurality of images labelled as gums with gingivitis; training the cycle generative adversarial model to transfer an image of the plurality of images labelled as healthy gums into an image showing a gum with gingivitis using one or more of the plurality of images labelled as gums with gingivitis as a real sample; and storing the transferred images, the received pluralities of images labelled as healthy gums, and gums with gingivitis, as training examples.

[25] It is often easier to find examples of healthy gums than gums with gingivitis. Therefore, generating examples of gums with gingivitis balances the training set

[26] According to an aspect of the present invention, there is provided a computer- implemented method of training a machine learning model to segment an image showing a gum to extract the gum from the image, the computer-implemented method comprising: receiving, by the machine learning model, training examples of images of gums; receiving, by the machine learning model, a plurality of gum masks, each gum mask of the plurality of gum masks being paired with a training example of an image showing a gum from the training examples of images of gums; running the machine learning model to generate a gum mask from each image of the plurality of images of gums; and modifying a parameterisation of the machine learning model to reduce an error between the gum masks generated by the machine learning model and the respective paired gum mask.

[27] In an embodiment, the computer-implemented method further comprises: generating the training examples of images of gums using the computer-implemented method of any preceding aspect or embodiment. [28] According to an aspect of the present invention, there is provided a transitory, or non- transitory, computer-readable medium, having instructions stored thereon that when executed by a computer, cause the computer to perform the computer-implemented method of any preceding aspect or embodiment.

[29] These and other aspects of the present invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

BRIEF DESCRIPTION OF DRAWINGS

[30] The embodiments of the present invention may be best understood with reference to the accompanying figures, in which:

[31 ] Figures 1 (a) to (c) show images of gums where inflamed and non-inflamed regions are classified using prior art methods;

[32] Figures 2 (a) to (d) shows images of gums where inflamed and non-inflamed regions are classified using prior art methods;

[33] Figure 3 shows a flow chart briefly summarising a training pipeline for predicting gingival index in gums, according to one or more embodiments;

[34] Figure 4 shows another flow chart briefly summarising a method of inference for gingivitis detection, according to one or more embodiments;

[35] Figure 5 shows a flow chart briefly summarising training Al models used in the flow chart of Figure 4, according to one or more embodiments;

[36] Figure 6 shows several images of gums including healthy and pathological samples from a gum dataset, according to one or more embodiments;

[37] Figure 7(a) shows two images of gums that are genuine healthy samples arranged in a column, Figure 7(b) shows two images of gums that are synthetic gingivitis gum samples generated by CycleGAN from corresponding healthy gum images in 7(a), according to one or more embodiments;

[38] Figure 8(a) shows an original image showing a gum, Figure 8(b) shows a mask of an upper gumline extracted from the original image showing the gum of Figure 8(a), Figure 8(c) shows a mask of a lower gumline extracted from the original image showing the gum of Figure 8(c), according to one or more embodiments;

[39] Figure 9 shows a table showing colour clusters extracted in RGB and LAB colour spaces with k-means clustering, according to one or more embodiments;

[40] Figure 10(a) shows a colour associated with a colour space Hex number #800000, Figure 10(b) shows a colour associated with a colour space Hex number #FFA093;

[41 ] Figure 11 shows a table showing colour clusters extracted in RGB and LAB spaces after colour filtration, according to one or more embodiments;

[42] Figure 12 shows a colour associated with a colour space Hex number #AD1 D1 D;

[43] Figure 13 shows a gingival index chart, as known from the prior art;

[44] Figure 14 shows a table including gingival thresholds based on distance D (in units) with respect to reference colour #AD1 D1 D, where categories severe, moderate, mild, and normal correspond to the respective gingival status in the gingival index chart of Figure 13, according to one or more embodiments;

[45] Figure 15 shows a table including gingival index predictions for healthy and pathological samples in RGB space, according to one or more embodiments;

[46] Figure 16 shows a table including gingival index predictions for healthy and pathological samples in LAB space, according to one or more embodiments;

[47] Figure 17 shows a flow chart summarising a computer-implemented method of determining a severity of gingivitis from an image showing a gum, according to one or more embodiments;

[48] Figure 18 shows a flow chart summarising a computer-implemented method of generating training examples of images of gums for use in training a machine learning model to segment images of gums to extract gums therefrom, according to one or more embodiments;

[49] Figure 19 shows a flow chart summarising a computer-implemented method of training a machine learning model to segment an image showing a gum to extract the gum from the image, according to one or more embodiments; and [50] Figure 20 shows a schematic diagram of a device.

DESCRIPTION OF EMBODIMENTS

[51 ] At least some of the example embodiments described herein may be constructed, partially or wholly, using dedicated special-purpose hardware. Terms such as ‘component’, ‘module’ or ‘unit’ used herein may include, but are not limited to, a hardware device, such as circuitry in the form of discrete or integrated components, a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks or provides the associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processors. These functional elements may in some embodiments include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination. In particular, the features of any one example embodiment may be combined with features of any other embodiment, as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the term “comprising” or “comprises” means including the component(s) specified but not to the exclusion of the presence of others.

[52] The methods described herein of training and inference may be provided as instructions stored on a transitory, or non-transitory, computer-readable medium that, when executed by a processor, cause the processor, to perform the corresponding method. The methods may thus be computer-implemented methods.

[53] A hybrid Al method is designed based on state-of-the-art computer vision and unsupervised machine learning algorithms to predict the gingival indices of upper and lower gums of a patient with a high accuracy in one-shot. For predicting the gingival index, the criteria developed by Loe and Silness is followed. A smartphone can be used to capture an image of gums which is then provided to the hybrid Al method for analysis and gingival index prediction.

[54] With reference to Figure 3, a training pipeline includes the following steps. Firstly, a training dataset 20 is provided. Image augmentation using CycleGAN occurs at 22 to balance the dataset. This is because there are many more images of healthy gums available than images of gums with gingivitis. Next, a regional of interest segmentation with DeepLab is identified at 24. At 26, a machine learning module, which may be a clustering algorithm, detects gum inflammation. At 28, a gingival index is predicted.

[55] With reference to Figure 4, a method of inference is achieved using the following modules.

[56] One module is an Image Augmentation module 32: Typically, in healthcare datasets, the number of pathological samples is far fewer than healthy samples. Fine-tuned a pretrained CycleGAN Al model on human gum images to augment the dataset with samples in the pathological class. The model generates synthetic pathological samples from genuine healthy gum images which can further enhance the quality of masks predicted by an Al model for more accurate analysis of gingivitis.

[57] Another module is a ROI Segmentation module 34: Fine-tuned a pretrained DeepLab Al model for detecting the ROI in the gums. The ROI comprises of image pixels representing the gums, across the gumlines and in between the crowns of teeth.

[58] Another module is a Color Detection module 34: Learn the colors present in the masks of upper and lower gums with k-means clustering algorithm. Healthy gums look pink and firm, while gums with gingivitis tend to be red and swollen.

[59] Another module is a Color Filtration module 36: Filter colors which have reddish tone in clusters predicted in previous step with an appropriate feature.

[60] Finally, a novel metric is proposed which predicts the gingival indices at 28 of upper and lower gums based on filtered colors in the same. In contrast to previous works which focuses on highly localized regions in gums (as shown in Fig. 1 and Fig. 2), embodiments of the current invention predict the health of upper and lower gums by considering the entire gumline region in the same. They are also the first to predict the severity of gingivitis in gums. [61 ] In Fig.3, the training pipeline is shown involving both deep learning and classical machine learning models for predicting the gingival index in gums. In Fig. 4, the inference flowchart is presented which highlights how the proposed method processes an input gum image and predicts the gingival indices for both upper and lower gums.

[62] It should be noted that there are two branches in Figure 4. Those branches correspond to performing embodiments of the invention in RGB and LAB colour spaces, as will be described in more detail below.

[63] With reference to Figure 5, simple yet effective method is proposed for accurate gingival index prediction of both upper and lower gums. The proposed approach consists of three algorithms: a) CycleGAN image augmentation, b) DeepLab mask segmentation, and c) color detection with k-means clustering. The proposed method leverages state-of-the-art Al models for image augmentation and semantic segmentation tasks, and a novel metric for classifying the gingival indices of upper and lower gums.

[64] In Fig. 5, a workflow of training and fine-tuning Al models is shown.

[65] In more detail, the workflow of Figure 5 includes receiving pre-trained Al models at 40, fine-tuning the Al models on a custom dataset at 42, saving the best Al models at 44, then using those Al models in production at 46. In production, a user gum image is received at 48, and a gingival prediction is made at 50 using the Al models in production at 46.

[66] In the above workflow, pre-trained CycleGAN and DeepLab Al models are fine-tuned on a custom human gum image dataset. Further details of fine-tuning Al models and custom human gum dataset are discussed below.

[67] A. Gum dataset

[68] With reference to Figure 6, an embodiment of the current invention fine-tunes state-of- the-art Al models on a custom human gum dataset for predicting gingival index. The gum dataset comprises of 150 publicly available human gum images shared by various medical websites. All images were procured with a web scraper. The dataset is manually prepared to include images from patients in different age groups and socio-demographic backgrounds. Both healthy 52 and pathological 54 samples of different degrees of severity are included in the dataset. However, the dataset is imbalanced with 110 healthy and 40 pathological samples. [69] Some challenges with training Al model on gum dataset are mentioned below.

[70] Gumlines are not properly visible for accurate analysis and predictions. Environment lighting conditions, particularly dark backgrounds. Light reflected from saliva on gums. Highly unequal number of samples in respective classes. Pathological sample has advanced to periodontitis.

[71 ] B. Image augmentation with CycleGAN

[72] With reference to Figures 7(a) and (b), primary challenges with medical datasets are the availability of limited data and class imbalance between healthy and pathological samples. Limited datasets can lead to overfitting of deep neural networks which can affect the scoring performance in test data. Image or data augmentation provides a suite of techniques that can improve the quality and size of training datasets to build robust deep learning models. The embodiment leverages the potential of generative adversarial network (GAN) which has been widely employed for data augmentation in medical datasets.

[73] A generative adversarial network learns to generate new data with approximately the same distribution as the training dataset. In other words, a GAN trained on original images can generate new synthetic images which a human can find to be authentic. A common Al use case in healthcare and medicine is image-to-image translation where the task is to determine the prominent features of one image domain and predict how these features can be translated into another image domain. In healthcare datasets, there are primarily two image domains: healthy and pathological. Since healthcare datasets are highly skewed towards the healthy class, Al algorithms for image-to-image translation can be used to generate synthetic pathological samples from genuine healthy image data.

[74] CycleGAN is a special class of GAN that can perform image-to-image translation in absence of paired images. These models can translate images from one domain to another without the need of a one-to-one mapping between source and target domains. A dataset of source and target images (the source and target images need not be correlated) can be used to train CycleGAN. The embodiment avails CycleGAN to generate synthetic samples with gingivitis from images in the healthy domain, thus enhancing the size and quality of the gum dataset. The original gum dataset comprises of 110 healthy and 40 pathological samples. A pretrained CycleGAN (e.g. horse2zebra or PyTorch) model is fine-tuned on the gum dataset and generate 70 synthetic samples with gingivitis. In Fig. 7a, genuine healthy gum samples are shown. The corresponding synthetic pathological samples with similar characteristics as genuine gingivitis samples are shown in Fig. 7b.

[75] C. Mask detection of gums

[76] With reference to Figures 8 (a) to (c), region of interest (ROI) mask detection is a computer vision task where an Al model maps each pixel of an image to a specific class. Mask detection models is widely used in training Al enabled autonomous vehicles and medical image diagnostics. In Fig. 8a, a sample from the gum dataset is shown while Fig. 8b and Fig. 8c represent the corresponding masks of upper and lower gumlines predicted by an Al model. The mask of an object comprises of pixels belonging to the same category or class.

[77] For training an Al model, the original gum dataset is divided into training and validation sets. Next, an open-source package by K. Wada Labelme, is used to annotate the ROI of upper and lower gumlines in the training and validation sets. The Labelme package aids the user to manually annotate the image pixels corresponding to a specific class.

[78] The original gum dataset is used to fine-tune a pretrained DeepLabv3 PyTorch model with an FCN-ResNet101 backbone for learning the mask of upper gumline 56 in each image. A second DeepLab model is trained to learn the mask of lower gumline 58. The fine-tuned DeepLab models achieved IOU scores of 0.60 and 0.62 respectively at 0.95 threshold on the validation gum dataset. The augmented gum dataset (with synthetic pathological samples generated by CycleGAN) is then used to fine-tune the DeepLab models and improved IOU scores of 0.66 and 0.67 at 0.95 threshold are achieved on the same validation dataset. The above results empirically prove that image augmentation with CycleGAN can enhance the predictive performance of an Al model on test data.

[79] At inference time, a method according to one or more embodiments, the machine learning models, or mask detection models, receive an image showing a gum, and prior, to determining a colour of each unit area, segments the image to extract only the gum. The segmenting the image to extract only the gum from the image using the machine learning model, with reference to Figures 8(a) and 8(b) may comprise: segmenting the image to extract an upper gum 56 region of interest of the gum using a first machine learning model of the machine learning model; and segmenting the image to extract a lower gum 58 region of interest of the gum using a second machine learning model of the machine learning model.

[80] D. Color detection

[81 ] In the previous section, the masks of upper and lower gums were obtained with a deep learning model. The next step involves determining the colors and corresponding tones present in the masks of gums. Healthy gums typically look pink and firm, while gums with gingivitis tend to be red and swollen. The primary objective in this section is detecting those colors with tones ranging from light pink to dark red in each mask.

[82] In this way, the method comprises, for each unit area of the gum, from amongst a plurality of unit areas of the gum, determining a colour. Each unit area of the plurality of unit areas may be a pixel wherein the colour of each pixel may be determined by determining a value of a pixel using a red, green, blue (RGB) colour model or a light/lightness, red, green, blue, and yellow, colour model. As will be appreciated from below, the method may also comprise assigning a colour space value to the colour of the cluster, and optionally the colour space values are hexadecimal numbers.

[83] Color space maps are mathematical functions which projects colors to a co-ordinate space. Since humans have trichromatic color vision with highest sensitivities at red, green and blue wavelengths, most color spaces have three dimensions. The most common color space is RGB or red-green-blue which is used by modem computers and mobile applications to display and store color images. CIELAB is another color space where the three channels are luminance (black to white), a (green to red) and b (blue to yellow). The CIELAB color space has been designed to estimate human vision, and it is convenient for predicting small differences in color. For the sake of brevity, CIELAB will hereafter be referred to as LAB.

[84] K-means clustering is an unsupervised machine learning algorithm which groups n unlabelled observations into k pre-determined clusters. A cluster refers to a set of observations aggregated together because of similarity in certain features. Each observation is assigned to a cluster with the nearest mean or cluster center. An embodiment of the current invention leverages k-means clustering to detect distinct color tones, present in upper and lower gum masks. The black background color in masks is redundant for gingival index prediction and is removed for further analysis. The next step involves setting up appropriate thresholds to extract different shades of healthy and inflamed gums. In Fig. 10, the hex color thresholds or bounds #800000 and #FFA093 are shown. The thresholds guarantee only those color shades within the bounds will be considered for analyzing the gum color shades with k-means clustering.

[85] In Figure 9, colors of similar shade and tone in gum masks are clustered together based on their RGB and LAB features. The colors in pie charts represent shades and tones of different clusters present in heathy and inflamed gum masks in both RGB and LAB color spaces. The clusters are labelled by the corresponding hex colors adjacent to a pie chart. The numerical value representing each cluster in a pie chart indicates the percentage of area or pixels in the upper or lower gum mask represented by the particular cluster. The numerical values in the clusters can aid in determining the spatial spread of inflammation in gums.

[86] In Figure 9, colors of similar shade and tone in gum masks are clustered together based on their RGB and LAB features. The colors in pie charts represent shades and tones of different clusters present in heathy and inflamed gum masks in both RGB and LAB color spaces. The clusters are labelled by the corresponding hex colors adjacent to a pie chart. The numerical value representing each cluster in a pie chart indicates the percentage of area or pixels in the upper or lower gum mask represented by the particular cluster. The numerical values in the clusters can aid in determining the spatial spread of inflammation in gums.

[87] E. Colour filtration

[88] The method comprises clustering, using a clustering algorithm, the plurality of unit areas according to the respective determined colours.

[89] The main objective of k-means clustering algorithm is obtaining color clusters representing different shades of healthy and inflamed gums. Healthy gum colors comprise of varying pink shades and tones, while inflamed gums are represented by red colors of varying shades. The next step involves color filtration of healthy gum color clusters from RGB and LAB space pie charts obtained in the previous section. The clusters representing healthy gum colors in upper and lower masks are filtered out from the pie charts obtained in the previous section and the updated pie charts consists of color clusters representing various shades and tones of inflamed gums. [90] Several color space features are considered for implementing color filtration with the aim of clearly identifying healthy gum color clusters from their inflamed counterparts in pie charts. The redness (R) component of RGB color space ranges from 0 to 255 where the RGB co-ordinates of pure red color is (255, 0, 0). The value of R in a hex color code is posited to be a definitive feature for distinguishing between various pink and red shades. However, further investigation revealed that the redness feature is not optimal since various pink and red color shades can have similar R values. For example, the hex color #FF1493 (255, 20, 147) with a dark pink shade and hex color #FF0000 (255, 0, 0) which represents pure red colour have same R value.

[91 ] The next feature considered is the yellowness of a color cluster. In RGB color space, pure yellow (255, 255, 0) is a secondary color formed by the combination of 2 primary colors - red and green. Therefore, the yellowness feature of a color is correlated with the redness of the same. CMYK (Cyan, Magenta, Yellow, Key) is another color space based on primary colors which are red, green and blue. Each of C, M, Y and K components vary between 0% to 100%. For pure yellow color, the Y value is 100% and is posited to be the feature which can distinguish between pink and red shades. However, the yellowness feature could not precisely identify several pink shade color clusters from their red counterparts. For example, #F811AB (248, 17, 171) with a pink shade has the same Y value of 30% in CMYK space as the hex color #810835 (129, 8, 53) represented by a variant of red shade.

[92] Another feature considered is the lightness of a color cluster. HSL is a relative color space and an alternative representation of RGB space. HSL refers to hue, saturation, and lightness of a color. The lightness of a color vary from 0 to 1 , with 1 representing the lightest tone and vice versa. Healthy gums often have a light pink shade compared to inflamed gums which have a darker shade. Hence, it is natural to posit that the lightness of a healthy gum should be greater than an inflamed gum. However, the lightness of certain pink color shades can be similar as those of multiple red color shades which makes it unsuitable for distinguishing healthy and inflamed gum colors. For example, hex color #E21 EBD (226, 30, 189) which has a dark pink shade has a lightness of 0.5 which is similar to that of pure red color #FF0000 (255, 0, 0).

[93] As will be appreciated from the following description, the method may comprise determining a distance between the colour space value of the colour of the cluster and a colour space value of the reference colour. Given the limitations of the above-mentioned features, a distance-based color feature is considered for distinguishing healthy and inflamed gums. The Euclidean distance of a color representing a cluster of interest is computed with respect to a reference color in both RGB and LAB spaces. In other words, the method comprises calculating a Euclidian distance of the colour of each cluster from the reference colour. The reference color considered for the illustrated embodiment is #A91 D1 D as shown in Fig. 12. Willi Geller, a renowned dentist, provided a color chart for detecting gingivitis in gums. However, the hex color codes corresponding to the stage of gingivitis is not mentioned in the chart. The hex color code which is closest to an advanced stage of gingivitis gum color as indicated by Willi Geller is #A91 D1 D. In this embodiment, the hex color #A91 D1 D is considered to represent the gum color of an advanced stage of gingivitis and the predicted severity of all stages of gingivitis is benchmarked against the hex color #A91 D1 D in RGB and LAB spaces. The current hex color code benchmark is subjective and alternative benchmarks can be considered that is best suitable for a particular use case.

[94] The Euclidean distance-based feature analysis led to the observation that the distance of healthy gum color clusters is greater than inflamed gum clusters with respect to the reference color #A91 D1 D in RGB and LAB spaces. A detailed investigation in RGB space indicated that with high probability, the count of healthy gum color shades being more than 80-100 units away from the reference color is much higher than the count of inflamed gums at a similar distance. Similarly, a greater number of healthy gum shades in LAB space is more than 40-50 units away than their inflamed gum counterparts. In an embodiment, the above empirical observations are considered for setting the distance thresholds of 80 units in RGB space and 40 units in LAB space to distinguish between healthy and inflamed gum color shades. The color filtration step removes those clusters in a pie chart that are beyond 80 units and 40 units with respect to the reference color in RGB and LAB spaces. Based on RGB and LAB thresholds, the remaining clusters in a filtered pie chart represent inflamed gum color shades present in gum masks. In Figure 11 , the filtered pie charts with inflamed gum clusters are shown for upper and lower gums in RGB and LAB spaces. In other words, the method may comprise filtering the clusters to remove one or more clusters having a distance of the colour from the reference colour that is greater than a threshold distance, wherein the threshold distance corresponds to a colour associated with a healthy gum. [95] F. Gingival index classification

[96] With reference to the description below, the method comprises calculating an overall distance value for the gum based on the distances of each cluster from the reference colour and the respective size of each cluster, and determining a severity of gingivitis based on the overall distance.

[97] The gingival index (Gl) score introduced by Loe and Silness is a measure of the stage of gingivitis - normal or healthy, mild, moderate, and severe - as shown in Figure 13. The Gl score ranges between 0 and 3, where a score of 0 indicates a healthy gum and a score of 3 indicates severe gingivitis characterised by swelling, inflammation, and bleeding of gums. The color filtration method introduced in the previous section removes clusters representing healthy gum color shades (Gl score = 0). The aim of the present section is to classify the gingival index category (Gl score = 1 , 2 or 3) of pathological upper and lower gums represented by remaining clusters in the filtered pie chart. A rule based gingival index classification method in RGB and LAB spaces is proposed in this section.

[98] Based on empirical observations of RGB and LAB space distances of clusters in a filtered pie chart with respect to the reference color #A91 D1 D and the stage of gingivitis in upper or lower gum, the optimal gingival index thresholds for accurately classifying the gingival index of a gum sample is proposed in Figure 14. Note that the gingival thresholds for determining the severity of gingivitis can also be treated as hyperparameters. The machine learning method of hyperparameter optimization can be leveraged to efficiently learn the gingival thresholds provided the gum dataset is labelled by a domain expert with corresponding gingival indices.

[99] In the previous section, the color filtration method produces a modified pie chart comprising of inflamed gum color clusters. The numerical value representing each cluster indicates the percentage of pixels or area assigned to the particular cluster. The numerical value can further be used to determine the fraction of pixels or area in the gum mask assigned to the i-th cluster in the filtered pie chart. Let denote the distance of the i-th cluster with respect to the reference color #A91 D1 D. Let S denote the set of indices of inflamed gum clusters in a modified or filtered pie chart representing a gum mask after color filtration. Then the weighted average distance, WAD, (which may be the overall distance) of the set of clusters in S with respect to the reference color is given by:

[100]

[103] where Sies/i = 1. At a qualitative level, the WAD distance indicates the stage or severity of gingivitis in gums. The gingival threshold buckets shown in Figure 14 and the distance WAD are analyzed to determine the severity of gingivitis. However, there is a drawback with the weighted average distance which might affect the final gingival prediction score of a gum sample. The WAD distance gives an equal weight to each cluster in the set S irrespective of the score (severe: 3, moderate: 2, mild: 1 and normal: 0) and does not make any distinction between a cluster with a higher score and a cluster with a lower score. Note that the gingival score of each individual cluster can be obtained from Figure 14. At an intuitive level, even a highly localized gum area affected by severe inflammation could be diagnosed with moderate to severe gingivitis which might not be indicated by WAD.

[104] To overcome the limitations of WAD, a new metric is proposed for accurately predicting the gingival score or index of a gum sample. The selected weighted average distance, SWAD, (which may be another type of overall distance value) places higher weights on clusters with a higher gingival score and the vice versa. Let c_t denote the gingival score of the i-th cluster based on the thresholds shown in Figure 14. In this way, the method may comprise generating a gingival score for each cluster based on the distance. The selected weighted average distance is then given by:

[105]

[108] where the normalization factor N = jesfj ’ ^eCJ- Note that ^ct ^{= 0} is not considered in the above equation since clusters representing healthy gums are removed after color filtration and hence, they are not included in the set S. In the above equation, an additional weight of e^Ci is assigned to the i-th cluster in S. Qualitatively, the SWAD distance can capture the severity of gingivitis in gums at a more fine-grained level when compared with WAD. Based on the SWAD distance computed for clusters in the filtered pie chart, the gingival index is predicted for upper and lower gums.

[109] With further reference to Figures 13 and 14, the overall gingival classification may be determined as severe gingivitis if the overall distance is less than a first threshold distance; moderate gingivitis if the overall distance is greater than the first threshold distance and less then a second threshold distance, where the second threshold distance is greater than the first threshold distance; and mild gingivitis if the overall distance is greater than the second threshold distance and less than a third threshold distance, where the third threshold distance is greater than the second threshold distance.

[110] The method may further comprise sending the determined severity of gingivitis to a display for displaying the severity of gingivitis to a user. Such a display is shown in Figure 20, which has been shown purely for illustrative purposes. The display 72 may be part of a mobile device, e.g. a smart phone. The device 70 may also include a camera 74, which may be the camera that captured the image used to perform the inference method of determining a level of gingivitis.

[111] G. Results

[112] In Figure 15 and Figure 16, the predicted gingival indices of some healthy and gingivitis affected samples in the gum dataset are shown in RGB and LAB spaces. The pie charts displayed in the tables are obtained after implementing the color filtration method. The first four images in the tables represent gingivitis affected gums while the final image is a healthy gum sample. Note that lower the value of SWAD distance, the predicted stage of gingivitis would be more advanced as compared to a higher SWAD distance. The SWAD distance is not defined for clusters representing healthy gum colors. Hence, by default, the SWAD distance would be undefined for healthy gums.

[113] Both RGB and LAB space based gingival index predictions can be considered to make an informed decision regarding the gum health status. The proposed method can aid a patient to gauge the severity of gingivitis and track gum health on a regular basis during and after treatment. [114] With reference to Figure 17, a computer-implemented method of determining a severity of gingivitis from an image showing a gum, according to one or more embodiments, can be summarised to include the steps of: receiving S100 the image showing the gum; for each unit area of the gum, from amongst a plurality of unit areas of the gum, determining S102 a colour; clustering S104, using a clustering algorithm, the plurality of unit areas according to the respective determined colours; calculating S106 a distance of the colour of each cluster from a reference colour; calculating S108 an overall distance value for the gum based on the distances of each cluster from the reference colour and the respective size of each cluster; and determining S110 a severity of gingivitis based on the overall distance.

[115] With reference to Figure 18, a computer-implemented method of generating training examples of images of gums for use in training a machine learning model to segment images of gums to extract gums therefrom, is summarised according to one or more embodiments. The computer-implemented method comprises the steps of: receiving S200, by a cycle generative adversarial model, a plurality of images labelled as healthy gums, and a plurality of images labelled as gums with gingivitis; training S202 the cycle generative adversarial model to transfer an image of the plurality of images labelled as healthy gums into an image showing a gum with gingivitis using one or more of the plurality of images labelled as gums with gingivitis as a real sample; and storing S204 the transferred images, the received pluralities of images labelled as healthy gums, and gums with gingivitis, as training examples.

[116] With reference to Figure 19, a computer-implemented method of training a machine learning model to segment an image showing a gum to extract the gum from the image is summarised, according to one or more embodiments as a computer-implemented method comprising: receiving S300, by the machine learning model, training examples of images of gums; receiving S302, by the machine learning model, a plurality of gum masks, each gum mask of the plurality of gum masks being paired with a training example of an image showing a gum from the training examples of images of gums; running S304 the machine learning model to generate a gum mask from each image of the plurality of images of gums; and modifying S306 a parameterisation of the machine learning model to reduce an error between the gum masks generated by the machine learning model and the respective paired gum mask. [117] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[118] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method of determining a severity of gingivitis from an image showing a gum, the computer-implemented method comprising: receiving the image showing the gum (S100); for each unit area of the gum, from amongst a plurality of unit areas of the gum, determining (S102) a colour; clustering (S104), using a clustering algorithm, the plurality of unit areas according to the respective determined colours; calculating (S106) a distance of the colour of each cluster from a reference colour; calculating (S108) an overall distance value for the gum based on the distances of each cluster from the reference colour and the respective size of each cluster; and determining (S110) a severity of gingivitis based on the overall distance.

2. The computer-implemented method of Claim 1 , further comprising sending the determined severity of gingivitis to a display (72) for displaying the severity of gingivitis to a user.

3. The computer-implemented method of Claim 1 or Claim 2, wherein the calculating a distance of the colour of each cluster from a reference colour comprises: assigning a colour space value to the colour of the cluster; and determining a distance between the colour space value of the colour of the cluster and a colour space value of the reference colour, wherein optionally the colour space values are hexadecimal numbers.

4. The computer-implemented method of any preceding claim, wherein the calculating the distance of the colour of each cluster from a reference colour comprises: calculating a Euclidian distance of the colour of each cluster from the reference colour.

5. The computer-implemented method of any preceding claim, wherein the overall distance value is a weighted average distance, WAD, defined by: wherein is a fraction of an area of an 7-th cluster compared to a total area of the plurality of unit areas of the gum, d_L is the distance of the colour of the /-th cluster from the reference colour, S is a set of indices of colour clusters present in the image showing the gum, and iesfi = 1-

6. The computer-implemented method of any of Claims 1 to 4, further comprising: generating a gingival score for each cluster based on the distance, wherein the overall distance value is a selected weighted average distance, SWAD, defined by: wherein is a fraction of an area of an 7-th cluster compared to a total area of the gum, d_t is the distance of the colour of the 7-th cluster from the reference colour, e^c; is a weight where c_t is the gingival score for the 7-th cluster, S is a set of indices of colour clusters present in the image showing the gum, and Xies/i = 1-

7. The computer-implemented method of any preceding claim, wherein each unit area of the plurality of unit areas is a pixel and wherein the determining the colour comprises determining a value of the pixel using a red, green, blue, colour model or a light, red, green, blue, and yellow, colour model.

8. The computer-implemented method of any preceding claim, further comprising: prior to determining a colour of each unit area, segmenting the image to extract only the gum using a machine learning model.

9. The computer-implemented method of Claim 8, wherein segmenting the image to extract only the gum from the image using the machine learning model comprises: segmenting the image to extract an upper gum region of interest of the gum using a first machine learning model of the machine learning model; and segmenting the image to extract a lower gum region of interest of the gum using a second machine learning model of the machine learning model.

10. The computer-implemented method of any preceding claim, further comprising: filtering the clusters to remove one or more clusters having a distance of the colour from the reference colour that is greater than a threshold distance, wherein the threshold distance corresponds to a colour associated with a healthy gum.

11. The computer-implemented method of any preceding claim, further comprising: determining an overall gingival classification as: severe gingivitis if the overall distance is less than a first threshold distance; moderate gingivitis if the overall distance is greater than the first threshold distance and less then a second threshold distance, where the second threshold distance is greater than the first threshold distance; and mild gingivitis if the overall distance is greater than the second threshold distance and less than a third threshold distance, where the third threshold distance is greater than the second threshold distance.

12. A computer-implemented method of generating training examples of images of gums for use in training a machine learning model to segment images of gums to extract gums therefrom, the computer-implemented method comprising: receiving (S200), by a cycle generative adversarial model, a plurality of images labelled as healthy gums (52), and a plurality of images labelled as gums with gingivitis (54); training (S202) the cycle generative adversarial model to transfer an image of the plurality of images labelled as healthy gums into an image showing a gum with gingivitis using one or more of the plurality of images labelled as gums with gingivitis as a real sample; and storing (S204) the transferred images, the received pluralities of images labelled as healthy gums, and gums with gingivitis, as training examples.

13. A computer-implemented method of training a machine learning model to segment an image showing a gum to extract the gum from the image, the computer-implemented method comprising: receiving (S300), by the machine learning model, training examples of images of gums; receiving (S302), by the machine learning model, a plurality of gum masks, each gum mask of the plurality of gum masks being paired with a training example of an image showing a gum from the training examples of images of gums; running (S304) the machine learning model to generate a gum mask from each image of the plurality of images of gums; and modifying (S306) a parameterisation of the machine learning model to reduce an error between the gum masks generated by the machine learning model and the respective paired gum mask.

14. The computer-implemented method of Claim 13, further comprising: generating the training examples of images of gums using the computer-implemented method of Claim 12.

15. A transitory, or non-transitory, computer-readable medium, having instructions stored thereon that when executed by a computer, cause the computer to perform the computer-implemented method of any preceding claim.