WO2024102062A1

WO2024102062A1 - Method for melanoma screening and artificial intelligence scoring

Info

Publication number: WO2024102062A1
Application number: PCT/SE2023/051143
Authority: WO
Inventors: Gyorgy Marko-Varga; Peter Horvath; Istvan BALAZS NEMETH; Krisztian KOOS; Gábor HOLLANDI
Original assignee: Malskin Ab
Priority date: 2022-11-11
Filing date: 2023-11-10
Publication date: 2024-05-16

Abstract

The invention pertains to a method for providing a neural network for skin anomaly classification, the method comprising: obtaining a data set comprising image data of a skin segment comprising a suspected skin anomaly, obtaining ground truth data, such as an assessment/diagnosis by a physician using the seven-point checklist scoring system, associated with the data set, and training the skin anomaly classification function based on the data set and ground truth data. The invention further provides a neural network ensemble for skin anomaly classification, and a method of dermoscopic screening of a skin anomaly defect, with melanoma tumour classification from skin images, using said neural network ensemble for skin anomaly classification.

Description

METHOD FOR MELANOMA SCREENING AND ARTIFICIAL INTELLIGENCE SCORING

Field of the Invention

This invention pertains in general to the field of skin mole identification and characterization. More particularly the invention relates to a method for characterizing skin moles that may be melanoma. Furthermore, the present invention pertains to a method of determining the malignity of a mole.

Background of the Invention

Melanoma is a type of skin cancer that develops from the pigment-producing cells known as melanocytes. Melanomas typically occur in the skin, and in women, they most commonly occur on the legs, while in men, they most commonly occur on the back. About 25% of melanomas develop from moles. Therefore, new skin moles, or changes in a mole that can indicate melanoma. Such changes include increase in mole size, irregular edges, change in color, itchiness, or skin breakdown.

Melanoma is the most dangerous type of skin cancer. Globally, in 2012, it newly occurred in 232,000 people. In 2015, 3.1 million people had active disease, which resulted in 59,800 deaths. One problem is that melanoma may lead to spread (metastasis) of the disease. Most people are cured if spread has not occurred, however, five-year survival rates in the United States were 65% when the disease has spread to lymph nodes, and 25% among those with distant spread. Thus, it is of utmost importance to capture melanoma early. Even so, in most countries, screening for melanoma is not mandatory.

Screening means testing people for early stages of a disease, before they see concrete symptoms of melanoma. Typically, a skin cancer specialist or nurse examine the skin. A specialist is trained to look out for moles that may be starting to become cancerous.

It is known that mole classification is very difficult. However, much of the cancer decisions are based on the mole image, generally from a dermatoscope, providing a morphological scope of the underlying tumor, and its specific classification, and as such it is vital for the patient to get a correct classification in order to provide a optimal treatment.

To further complicate matters, novel Melanoma diagnosis and malignancy testing is often based upon a manual interrogation of the mole(s), covering the body with a variety of localizations. Finally, the tumor may grow in different tissue, making the physical properties appear very different between tumors.

For screening investigations to be useful the tests: need to be reliable at picking up cancers, overall must do more good than harm to people taking part, and must be something that people are willing to do.

Thus, screening both has to pick up all suspicious moles without removing too many (non-mel anoma) moles. To further complicate matters, different individuals usually have different normal appearance of the skin, also dependent on ethnicity, age and life style. Thus, it may be hard for a specialist meeting a patient for the first time, to be able to spot what is normal or abnormal for that unique individual.

Thus, there is a need for methods to help screen individuals in order to capture melanoma early.

Summary of the Invention

Accordingly, the present invention preferably seeks to mitigate, alleviate or eliminate one or more of the above-identified deficiencies in the art and disadvantages singly or in any combination and solves at least the above mentioned problems by providing a method for cancer characterization, wherein is provided a method for providing a neural network for skin anomaly classification, the method comprising: obtaining a data set comprising image data of a skin segment comprising a suspected skin anomaly, obtaining ground truth data, such as an assessment/diagnosis by a physician using the seven-point checklist scoring system, associated with the image data, and training the skin anomaly classification function based on the data set and ground truth data.

Also is provided a method for providing a neural network ensemble for skin anomaly classification, the method comprising: at least 2, such as at least 4, preferably at least 5, more preferably at least 8, neural networks, said neural networks being ensembled to return a single classification probability score.

Further, a neural network ensemble is provided configured to classifying data as either of the two classes benign and malignant phenotype on a continuous scale from 0 to 1.

Also, is provided a method of dermoscopic screening of a skin anomaly, with melanoma tumor classification from skin images, comprising the steps of a) from at least one macro image of a subject’s skin, comprising a high-resolution overview of a larger skin area of the subjects, performing a preliminary prediction, wherein a detection algorithm determines each mole by an image processing method, for each determined mole, a machine learning model, an image processing algorithm or a neural network according to claim 4 detects and selects moles more likely to show a malignant pattern, and provides a list of the selected moles with their coordinates, and b) for at least one, preferably for each, of the selected moles selected in step a), from a high resolution micro image of the selected mole, or cluster of moles, performing a classification of the malignity for each of the moles using the neural network ensemble according to claim 6 or 7, wherein the method provides an output displaying the moles selected in step a) together with the classification of malignancy, on a continuous scale from 0 to 1 from benign to malignant phenotype of step b), together with mole coordinates.

Further, a system for dermoscopic screening of a skin anomaly, comprising a macro-camera, a micro-camera, a controller and computation unit, a graphical output device, and a database and/or storage device is provided.

Also, is provided a method for dermoscopic screening of a skin anomaly using the method and system for dermoscopic screening, wherein the macro-camera is used to capture at least one macro image of the skin surface of the patient, the controller and computation unit performs a preliminary prediction and provides a list of selected moles deemed more likely to show a malignant pattern together with their coordinates on the graphical output device, the micro-camera is used to capture a micro image per mole of at least one, preferably all, of the selected moles, and the controller and computation unit performs a classification of the malignity of each mole for which a micro image has been captured and provides an output on the graphical output device showing a list of the selected moles, their classification of malignancy and a mole identifier or mole coordinates.

Brief Description of the Drawings

These and other aspects, features and advantages of which the invention is capable of will be apparent and elucidated from the following description of embodiments of the present invention, reference being made to the accompanying drawings, in which

Fig- 1 shows an overview of the method of dermoscopic screening of a skin anomaly, with melanoma tumor classification from skin images, in accordance with the invention. The overview is from image capture of the patient to final melanoma clinical status after excision of two spots. Possible software algorithm analysis steps are shown, including identification/analysis of moles at various body locations, assessment and diagnostic read-out;

Fig- 2 shows and overview of a system for using the method of dermoscopic screening of a skin anomaly, with melanoma tumor classification from skin images, in accordance with the invention. (A) The system comprises a macro-camera, a microcamera (dermascope), a controller and computation unit (computer), a graphical output device (monitor), and a database and storage device (cloud). The graphical output device is as an exmple shown in (B) to output macro image with selected spots, micro images of selected spots and software assessment output;

Fig- 3 shows and overview of a method of dermoscopic screening of a skin anomaly, with melanoma tumor classification from skin images, in accordance with the invention. In the method, a preliminary prediction of at least one macro image of a subject's skin is performed, where moles are detected, assessed and an output is created, followed by an assessment of selected moles using high resolution micro images of the selected moles, resulting in an output of the malignity for each of the assessed moles;

Fig- 4 shows an overview of training of a skin anomaly classification function based on the data set and ground truth data using the method of the invention. In (A) a data set comprising image data of a skin segment comprising a suspected skin anomaly and corresponding ground truth data, such as an assessment/diagnosis by a physician using the seven-point checklist scoring system, is obtained, and the skin anomaly classification function is trained based on the data set and ground truth data. In (B), the images are first selected for a characteristic or processed (such as resized to a specific resolution), whereby the training would be weighted by the prevalence of these criteria in the data sets;

Fig. 5 shows an overview of using a neural network ensemble for skin anomaly classification, wherein in A), five neural networks, preferably each having been trained for a different selected characteristic, are ensembled to return a single classification probability score. In B), additionally, images may be processed before the assessment, and the ensembled single classification probability score may be weighted based on neural network characteristics, image processing of patient meta data.

Fig. 6 shows a macro image of a patient and detected moles marked by bounding box;

Fig. 7 shows an example of cropped micro-images of captured moles, and Fig- 8 ; shows the output of the method using the 8-channel mAIskin algorithm with micro-image of mole, identifier and prediction of malignancy.

Description of embodiments

The following description focuses on an embodiment of the present invention applicable to a method for cancer characterization which addresses several of the problems with the current methods.

In the text, the terms skin anomaly, moles and birthmarks may be used interchangeably, referring to moles on the body surface.

In the routine dermatological practice, the patient has a nude body allowing gross, naked-eye inspection of the existing moles on the whole body surface. In general, mole can have any coloral change, but nearly skin-colored macules, papules or plaques are also noticed. At the gross exam, the topography of the moles are identified, also the macroscopically clearly different moles (in size, color, or shape) from others are already initially marked (gross topography identification and initial preanalysis, marking). The order of the regions are the trunk, head and neck region, limbs (also palms and soles), and genital area.

After the gross inspection, marking, and initial pre-analysis, there is a systematic manual dermoscopic inspection of the moles in order of the body regions. By dermoscope 1. analysis of the magnified patters (statical cross sectional pattern analysis) of each moles, 2. at the same time also the systematic comparison among the examined moles is performed. By examining them in order the similar phenotypes indicating the subsequent movement of the process. If different dermoscopic patterns are noticed, jump from the previous lesions and comparison can be made together with marking. The pattern analysis is made by the dermoscopic guidelines (ABCDE-rule, Assymetry, Border, Color, Diameter, Elevation and 7-points checklist: 3 points indicate suspicious for melanoma):

Major Criteria Score

Atypical pigment network 2

Gray -blue areas 2

Atypical vascular pattern 2

Minor Criteria Score

Radial streaming (streaks) 1

Irregular diffuse pigmentation (blotches) 1 Irregular dots and globules 1 Regression pattern 1

In case of melanoma suspicion, excision should be made for histopathologic clarification. The latest can provide the ultimate feedback for the clinician to assess the diagnostic efficacy, further therapeutic and follow up strategies.

The regular mole check in usually 1 event/year, however, multiple atypical mole syndrome may indicate 3-6 monthly follow up. Similarly, certain dysplastic moles which have an moderately atypical pattern but do not step through the threshold criteria of excision, shorter follow up periods should be made.

Typically, this analysis a skin cancer specialist trained to look out for moles that may be starting to become cancerous. However, it is very tedious work for and requires a lot of concentration.

Thus, in the invention is provided a method which enables automated detection, recognition and classification of moles.

In the method of the invention, the input is images of the patient taken at a macro (larger area) or micro (individual mole) level. The method provides an output of selected (non-benign) moles, together with a probability score corresponding to the malignity of the moles. An example in accordance to the invention can be seen in figure 1.

A simple system for collecting images is illustrated in figure 2A. Such a system should comprise at least a macro-camera, a micro-camera (dermascope), a controller and computation unit (computer and software), a graphical output device (monitor), and a database and/or storage device (here the cloud). A graphical output device is shown in (B), displaying output macro image with selected spots, micro images of selected spots and software assessment output.

Also, is provided a method for dermoscopic screening of a skin anomaly using the method and system for dermoscopic screening. Such a method can be seen in Figure 3. Here, the macro-camera is used to capture at least one macro image of the skin surface of the patient, the controller and computation unit performs a preliminary prediction and provides a list of selected moles deemed more likely to show a malignant pattern together with their coordinates on the graphical output device. The micro-camera is used to capture a micro image per mole of at least one, preferably all, of the selected moles, and the controller and computation unit performs a classification of the malignity of each mole for which a micro image has been captured. Also provided is an output on the graphical output device showing a list of the selected moles, their classification of malignancy and a mole identifier or mole coordinates.

The classification of the moles between the benign and malignant phenotypes may be a continuous scale from 0 to 1; this can also be expressed on a percentage scale. A simple example of such a classification is shown in figure 1 or 2B.

Preferably, before or during examination, metadata for the patient is collected. Metadata may include non-image based characteristics, such as age, gender, location of birthmark, previous medical history (did they have melanoma in the family). If meta data is provided, this may be linked to their associated images whereby interoperability between them is ensured. Such information, like evidence of risk-factors, may be help provide a better prediction when characterizing spots.

During an examination, overview images, so-called macro images showing larger skin regions of the patient's body are taken with a high resolution camera. Usually, at least four macro images are taken to cover the larger skin regions of the patient's body.

Ideally, the macro images should be captured with good light conditions straight above the patient, to ensure easier correlation between the position of the moles on the patient and in the macro image.

The macro image(s) preferably comprises at least 1 000 pixels per cm² of imaged skin surface, preferably at least 2000 pixels, such as at least 4000 pixels, per cm² of imaged skin surface. Thus, a high-resolution image of 7680 x 4320 (8K UHD) might be used to image a macro image of a large part of a torso of a patient.

The mole detection may be based on a convolutional neural network to recognize and overlay the moles on the images.

Preferably, the macro prediction is appended with a classification step, which further filters the samples to be examined.

During classification, a melanoma probability score is assigned to each mole detected during processing, based on this the detected objects can be categorized medically.

This may be done by for instance a deterministic or stochastic algorithm which generates a score for each mole, indicative of if the mole is likely to show a malignant pattern.

Preferably, the score is generated using a neural network for skin anomaly classification of the malignity for the mole, or a neural network ensemble classification, further described under micro-scan analysis below. From these overview images (i.e. macro images), a preliminary prediction, i.e. macro-prediction, is performed, during which spots are identified and highlighted.

In this preliminary prediction, a detection algorithm determines each mole by an image processing method. Since the image processing method only has to detect skin defects, several different algorithms work for this initial detection. The mole detection may be based on a convolutional neural network to recognize and overlay the moles on the images.

Preferably, the macro prediction is appended with a classification step, which further filters the samples to be examined. During classification, a melanoma probability score is assigned to each mole detected during processing, based on this the detected objects can be categorized medically.

Preferably, the score is generated using a neural network for skin anomaly classification of the malignity for the mole, or a neural network ensemble classification, which is further described under micro-scan analysis below.

If several macro images are captured, the result of the macro prediction can be used to recognize and overlay moles that appears on more than one macro image, to ensure that each mole/spot gets a unique identifier. This may also be used to overlap several macro images to create one image. These overview images may thus serve as a base layer for creating a skin surface map. This skin surface map helps with the identification and positioning of the moles of the skin surface during later examination.

Detected moles are preferably visually displayed on a screen with a bounding box (its position is saved in absolute or relative coordinates from the corner points and the centroid of the selected area). In figure 6 is shown an example of a macro image of a patient and detected moles marked by bounding box.

All moles over a cut-off score are selected for further examination. The algorithm selects moles that are likely to show a malignant pattern and for these provide the macro-image coordinates of the mole. The collected data about these moles can help focusing for further subsequent moves in the so-called “micro scan” process.

Thus, the goals of the pre-selection step on macro images are: filter out obviously benign moles (this keeps specificity high) only the suspicious or malignant moles need to be examined with a dermoscope and must be involved in the subsequent processing steps. Therefore image processing is faster, because the unselected moles are not processed any further, and the medical examination is also faster and more comfortable, because the doctor doesn't need to check every moles with a dermoscope.

Sequentially the “macro-scan” analysis, the next step is the “micro-scan” analysis considering the skin examination of the patient. During a further examination, high resolution (dermoscopy) images are taken of the individual moles selected during the macro-prediction. The moles are easily found based on the macro scan provided coordinates and cropped image of the mole.

In the micro-scan, high-resolution dermoscopy images are captured, with preferably only one object, or a cluster of close-by objects, on the image which is the mole to be examined. The device may be supplemented with a polarizer.

The high-resolution micro image preferably comprises at least 1 000 000 pixels per cm² of imaged skin surface, preferably at least 3 000 000 pixels per cm² of imaged skin surface.

Capturing a micro image of each mole/spot presents several advantages. The high resolution imoage ofcourse provides a lot more detail than the cropped macro image. Another advantage is that the camera and lens can be selected to provide a favorable depth of field (DOF), which, under good light conditions, may help to capture mole details not just on the surface of the skin, but also features just under the skin surface, thereby possibly obtaining more characteristics of the mole.

From these detailed mole images (i.e. micro images), the method of the invention performs a micro-prediction.

It was hypothesized that machine learning could be used to aid in the mole, or skin defect, characterization, using artificial intelligence (Al) to provide support to physicians and help relieve their workload.

In the method of the invention, a method for providing a neural network for skin anomaly classification was developed. The method comprised obtaining a data set comprising image data of a skin segment comprising a suspected skin anomaly. Also, ground truth data - that is an assessment/diagnosis by a physician - associated with the data set should be obtained. Thereby, a skin anomaly classification function could be trained based on the data set and ground truth data.

In the invention, it was found that using a very accurate ground truth data set during training lead to very good characterization results by the neural network. Thus, for the ground truth dataset, the ABCD rules according to Soltz et al. (1994) and the seven-point checklist scoring system based on Argenziano et al. (1998) are used. For qualitative analysis, after the manner of mole severity scores, the naevus is evaluated from 1 to 4, whereas 1- no suspicion for malignancy, 2 - dysplastic naevus, 3 - suspicion for malignancy, 4 - malignant melanoma. Based on the ABCD rules, the moles are examined depending on their asymmetry (A), the evaluation of border sharpness score (B), the determination of color (C), the dermoscopic structures (D) such as dots, globules, structureless areas, network and branched steaks. Furthermore, the diameter of moles (D), more than 6 mm diameter large naevus can be suspicious to malignant transformation. The seven-point checklist scoring system is a sensitive dermoscopic algorithm to help the clinicians to differentiate between benign melanocytic lesion and melanoma. It has major criteria such as atypical pigment network, blue-white veil and atypical vascular pattern; and minor criteria, whereas the dermatology photo is dominated by the irregularity like the presence of irregular streaks, irregular dots/globules, irregular blotches and regression structures.

Thus, the training data preferably comprises image data of a skin segment comprising a suspected skin anomaly and ground truth data, such as an assessment/diagnosis by a physician using the seven-point checklist scoring system, associated with the image data.

Using a convolutional neural network, it was found that both high sensitivity and specificity could be obtained. In a convolutional neural network, the neural network learns how to identify the classes through consequent feature map representations extracted from the input image by convolutional layers.

As seen in example 2, a convolutional neural network trained with a very accurate data set comprising 25311 images lead to an AUC - “area under the curve” of an ROC ( “receiver operating characteristic”) curve, which plot the sensitivity along the y-axis and (1 - specificity) along the x-axis) - of about 0.9, where an AUC of 1 is able to perfectly classify observations into classes while a model that has an AUC of 0.5 does no better than a model that performs random guessing. This was a very good result, and showed that the approach could indeed be used for mole characterization.

Figure 4 shows an overview of training of a skin anomaly classification function using the method of the invention, where the training is based on the data set and ground truth data. In (A) a data set comprising image data of a skin segment comprising a suspected skin anomaly and corresponding ground truth data, such as an assessment/diagnosis by a physician using the seven-point checklist scoring system, is obtained, and the skin anomaly classification function is trained based on the data set and ground truth data. In (B), the images are first selected for a characteristic or processed (such as resized to a specific resolution), whereby the training would be weighted by the prevalence of these criteria in the data sets

In the invention, it was found that instead of training one neural network with a larger amount of images, one could train several neural networks with sub-batches of images. It was found that the individually trained neural networks had slight differences when characterizing spots.

Also, spots can be divided into subgroups depending on their characteristics, or for other data criteria, such as skin types or skin tone tones (which will affect how the spots looks like).

It was found that one could use different data sets with focused on certain characteristics, such as mole characteristics or skin characteristics, whereby the training would be weighted by the prevalence of these criteria in the data sets, and that such training provided slightly different results from the neural network.

It was hypothezised that this could be used as an advantage, like having several different specialists that could provided expert opinion with regards to a specific mole. Thus, in the invention was thus developed a method for providing a neural network ensemble for skin anomaly classification. Instead of training a single neural network, the method comprises training at least 2, such as at least 4, preferably at least 5, more preferably at least 8, neural networks, wherein the neural networks being ensembled to return a single classification probability score.

Further, was provided a neural network ensemble wherein at least two neural networks are provided, the two neural networks may be based on data sets having different selected characteristics.

An example of such a a neural network ensemble for skin anomaly classification is shown in figure 5, wherein in A), five neural networks, preferably each having been trained for a different selected characteristic, are ensembled to return a single classification probability score. In B), additionally, images may be processed before the assessment, and the ensembled single classification probability score may be weighted based on neural network characteristics, image processing of patient meta data.

It was surprisingly found that using such an approach, the AUC for 2-4 ensemble models was as high as 0.91 - 0.94. Using 6 models, the AUC was as high as 0.93 - 0.94. Finally, when using 8 models weighted in the ensemble, the AUC was a consistent 0.95 - 0.96. This provided a charactering that was not only sensitive, but also specific enough to be used not only to find spots of malignant phenotype, but also to avoid too many false positives. The final score is an ensemble of the predictions of the multiple neural network models, each sensitive for distinct characteristics of moles via unique training approaches including number of epochs, learning rate, augmentation. These may also include metadata (which is the relevant previous medical information of the patient), therefore the decision of several independent "doctors" is simulated. The algorithm weights the individual model predictions and combines them to a single final score which provides an overview of the mole's malignity.

As can be seen in Example 1, a neural network could be trained using resized micro images having a low size, such as between 40000 pixels to 500000 pixels, such as for example having sizes of from 256x256 to 512x512.

It was found that a network trained using these smaller images generally provided higher prediction probabilities, and malignant cases were predicted to have considerably higher scores, while benign moles also had higher scores but not with such a large difference as malignant moles. Thus, such training provided models more sensitive to malignant samples, which in an ensamble can be used to prevent potential melanomas from being undetected because the final score was increased enough to fall into the suspicious category even if the other models predicted lower scores (below the grey zone).

Another training module that may be used is a hair augmentation module. During hair augmentation different hair strings are added from real images to training images that don't have hair on them in order to help mole recognition when the mole is occluded by hair. We combine strings with different arc and length, and place them on the image with random rotation and offset.

The neural network ensemble is configured to classify data as either of the two classes benign and malignant phenotype on a continuous scale from 0 to 1. Preferably, the neural network ensemble is further configured to provide data output processed at a processing stage as output data of the neural network skin anomaly classification.

That is, that the individual classification network output a classification score each, then these are combined to a single score which is the output of the neural network ensemble. The combination uses weighted averaging and machine learning methods.

Thus, the ensemble prediction score is the combination of the multiple classification model results, wherein one final score is calculated for each image from the multiple classification results. The model predictions may be weighted based on factors such as input image size, metadata information and/or if any processing, such as hair augmentation was used in training.

The final combined score is thus calculated, possibly by using a machine learning algorithm, from the weighted average of the classification models' predictions, and may optionally including a metadata (previous medical information of the patient, e.g. age, sex, location of birthmark etc).

To aid the clinician during the following micro scan step, moles with a higher score (suspicious or malignant) may be displayed on a screen with a different color for visualization (e.g. the benign samples represented with green color, the malignant with red, the suspicious moles with a color between green and red based on the prediction score).

Once it was realized how effective the micro-scan classification was, it was hypothesized that the macro scan classification might benefit from using neural networks trained using micro images. However, the cropped macro images are smaller with lower resolution than the micro images.

Since the cropped macro-images have a smaller image size, a neural network trained using resized micro images having a low size (as described above) seemed to yield especially good results when classifying cropped macro-images. Thus, the images for training may be resized micro images or cropped macro-images.

In one example of the invention, the classification step of the macro prediction may include the following steps:

- the moles selected during the preliminary prediction are resized to for instance 256x256, i.e. we scale the images to the image sizes used in the main classification.

The whole image is examined during image processing, one model makes the prediction on each crop, and the prediction is also between 0 and 1.

Preferably, but not mandatory, the neural network used for the classification step has been trained with an image size of between 512x512 to 256x256.

It was found that such an approach yielded very good results. Also, the threshold for selecting spots as possibly malignant can be adjusted from the value defined for the main classification on dermatoscopic images in order to avoid false negatives. A lower threshold value might result in a higher likelihood of false positives, however, this is preferable since it reduces the risk of borderline spots not being assessed during the micro scan. Thus, the neural networks from the micro image classification may be used during the classification step of the macro-scan. Similarly, an ensemble prediction score, combining multiple classification model results, may also be used for macro prediction.

In the invention, a the method of dermoscopic screening of a skin anomaly, with melanoma tumour classification from skin images, of the invention, preferably comprises the steps of a) from at least one macro image of a subject’s skin, comprising a high- resolution overview of a larger skin area of the subjects, performing a preliminary prediction, wherein a detection algorithm determines each mole by an image processing method, a machine learning model, an image processing algorithm or neural network detects and selects moles more likely to show a malignant pattern, and the model or algorithm provides a list of images of the selected moles with their coordinates to enable a micro-scan of the selected moles, b) for at least one, preferably for each, of the selected moles selected in step a), using a high resolution image of a selected mole, or cluster of moles, as data set, performing a classification of the malignity for each of the moles using a neural network ensemble (as described above), wherein the method provides an output displaying the moles selected in step a) together with the classification of malignancy, on a continuous scale from 0 to 1 from benign to malignant phenotype of step b), together with mole coordinates.

One main advantage of the invention is that instead of relying on the experiences of individual clinicians (which may vary) it collects and refines the knowledge in a database, making the classification better and better, the more samples that are processed. Figure 8 shows the output of the method using the 8-channel mAIskin algorithm with micro-image of mole, identifier and prediction of malignancy.

Further, the ensamble prediction helps avoid prediction bias due to properties of the patient skin or micro image quality, since several predictions trained for different situations are combined or weighted together. This also makes the ensamble prediction less sensitive to prediction outliers.

In both the macro- and micro analysis methods, a number of modules may be used. This is an advantage, since it means that additional modules can be added to the method if needed.

The preliminary prediction of step a) may comprises: For all moles detected by the detection algorithm, setting a bounding box around each mole and cropping the images of each boundary box to 256x256 pixels.

Further, if a neural network is used in step a), it may have been trained using images of malignant moles having an image size of from 1024x1024 to 256x256 pixels.

In the both the macro- and micro processing, a quality control module may used. The quality control module performs an analysis of the input images to verify that they meet the image requirements for mole detection. This normally includes a check that the image comprises an object that is in focus, with enough contrast and sharpness to enable a prediction. This is of vital importance, since if too poor images are used, the prediction results may not be correct. Images that do not pass the quality control will be flagged as unusable and highlighted.

Images should preferably have appropriate lighting and focus settings, and should not be blurry or overexposed. Preferably, images are taken using the same lighting conditions for all images; like with the same device or camera system. Image should be standardized in order to reach the optimal execution performance during image processing.

The quality control module may use focus detection, elimination of areas without information, removal of image noises, discard of dark corner regions from dermatoscopic images.

Also, for the micro images, a pre-processing module may crop the image of the mole and resize it to a different scale or size. As can be seen in Example 1, an image size of 1024x1024 is usually sufficient for the micro image characterization. This image size also helps increase the speed of the processing. Figure 7 shows an example of cropped micro-images of captured moles.

Another module that may be used is a template matching module. This module performs the image matching used between modules. The module performs an examination and storage of moles position and provides positioning information for moles selected in the macro-scan to simplify micro scanning.

Once micro scanning has been performed, the template matching module performs mole pairing between micro and macro scan and provides a unique identifier for a paired moles. The module performs expansion and storage of mole pairs and lists and manages stored images with time stamp.

As can be seen in Examples 3 to 5, benchmarking tests using the method of dermoscopic screening of the invention resulted in a very large overlap between the data set and the prediction. Thus, the method greatly helps to reduce the burden for the physician. Instead of having the physician looking at each individual skin anomaly (a very tedious work for the and requires a lot of concentration), the method of the invention finds and selects spots that need further investigation from a small number of macro-screen overview images. From micro-images of the selected spots, the method of the invention further performs a classification of the malignity for each of the moles, using ensamble prediction. This helps avoid prediction bias due to properties of the patient skin or micro image quality, since several predictions trained for different situations are combined or weighted together. The provided patient overview and prediction output greatly helps the physician when is gives the physician a great support when determining if any spots need further therapeutic and follow up strategies.

Examples

1 - Model training

Trained on two datasets (ISIC 2019-2020, ISIC-2020) Both datasets were processed with 4 different image sizes 256x256, 384x384, 512x512, and 768x768.

The models were only ensembled at the final step.

Only a few configurations (trained models) used hair augmentation Hair augmentation is independent on the image size ISIC-2019-2020: 256x256

ISIC-2019-2020: 384x384

ISIC-2019-2020: 512x512

The images were input to the model training in sizes smaller than the original image size. It was found that:

1) using original image sizes didn’t improve classification accuracy

2) the small images resulted in faster training, saving gpu memory

3) observations on smaller image sizes (e.g. 256x256) generally provided higher prediction probabilities, and malignant cases were predicted to have considerably higher scores, while benign moles also had higher scores but not with such a large difference as malignant moles. It was found that these models were more sensitive to malignant samples, these models were included in the final ensembled score with higher weights than the models trained on larger image sizes, and that this could be used to prevent potential melanomas from being undetected because the final score was increased enough to fall into the suspicious category even if the other models predicted lower scores (below the grey zone).

The advantage is that this promotes well being: if the patient returns for a follow up examination in 3-6 months and the given mole is predicted a higher score, it suggests a suspicious case.

The disadvantage is that it may increases false positive rate and decreases the specificity

Example 2 - Ensemble Model training

Optimal prediction precision on the ISIC test dataset was found to have:

1) High sensitivity and low false negative rate (as a trade off false positive rate and specificity can be higher).

2) Preferably, a minimum of 4 models are needed (trained on 4 characics, such as 4 image sizes).

3) It was found that classification performance was improved without serious increase in computational resources/time when applying 8 models.

Observations on even larger image size e.g 1024x1024 showed that:

1) Applying 5 image sizes including 1024x1024 did not increase classification accuracy or even decreased it

2) Further increase in image size didn’t improve performance.

3) Applying a larger number of models with the first 4 image sizes might increase classification accuracy at the cost of considerably longer training and prediction times and computational resources.

4) It is likely that additional/different models might be required when the experimental training data is extended.

Using 8 models ensembled to return a single classification probability score, to classifying data as either of the two classes benign and malignant phenotype on a continuous scale from 0 to 1 , therefore the ensemble of 8 models is the classification unit, the following observations were reached using our current models on the ISIC test dataset: AUC with 1-2 model(s):

• 0.89 - 0.93

• strongly dependent on training dataset, image size and training configurations

• an optimal image size could not be selected

AUC with 2-4 models:

• 0.91 - 0.94 (only trained on ISIC-2019-2020)

AUC with 6 models:

• 0.93 - 0.94

• already a high number of models, still depends on training dataset, image size and training configurations

AUC with 8 models:

• 0.95 - 0.96

• all image size configurations included and their models weighted in the ensemble

3 - Data benchmarking using Data Ensamble model and ISIC-2019 dataset number of images: 25 331 sex: 53% male, 47% female age_approx: average age: ~ 45 year anatom_site: 38% torso, 20% lower extremity ... diagnosis: nv-51%, mel-18%, other-31% target: 82% benign (20 809), 18% malignant (4 522)

4 - Data benchmarking using Data Ensamble model and ISIC-2020 dataset number of images: 33 126 sex: 52% male, 48% female age_approx: average age: ~ 45 anatom_site: torso 51%, lower extremity 25% diagnosis: target:98% benign (32 542), 2% malignant (584) 5 - Data benchmarking using Data Ensamble model and ISIC-2019-2020 database number of images: 58 457 sex: 52% male, 48% female age_approx: avg: ~45 years target: 91% benign, 9% malignant

Although the present invention has been described above with reference to (a) specific embodiment s), it is not intended to be limited to the specific form set forth herein. Rather, the invention is limited only by the accompanying claims and, other embodiments than the specific above are equally possible within the scope of these appended claims, e.g. different ... than those described above.

In the claims, the term "comprises/comprising" does not exclude the presence of other elements or steps. Furthermore, although individually listed, a plurality of means, elements or method steps may be implemented by e.g. a single unit or processor.

Additionally, although individual features may be included in different claims, these may possibly advantageously be combined, and the inclusion in different claims does not imply that a combination of features is not feasible and/or advantageous. In addition, singular references do not exclude a plurality. The terms "a", "an", “first”, “second” etc do not preclude a plurality. Reference signs in the claims are provided merely as a clarifying example and shall not be construed as limiting the scope of the claims in any way.

Claims

1. A method for providing a neural network for skin anomaly classification, the method comprising: obtaining a data set comprising image data of a skin segment comprising a suspected skin anomaly, obtaining ground truth data, such as an assessment/diagnosis by a physician using the seven-point checklist scoring system, associated with the image data, and training the skin anomaly classification function based on the data set and ground truth data.

2. The method according to any one of claims 1 to 2, wherein the neural network is a convolutional neural network.

3. The method according to any one of claims claim 1 to 2, wherein the neural network is trained using a data set, comprising a selected characteristic chosen from the group comprising of skin anomaly type, melanoma type, early stage melanoma phenotype, late stage melanoma phenotype, or a specific skin tone.

4. The method according to any one of claims claim 1 to 3, wherein the neural network is trained using a data set, comprising lower resolution image data, such as images having between 30000 to 500000 pixels, such as 50000 to 300000 pixels.

5. The method according to any one of claims claim 1 to 4, wherein the neural network is trained using a data set processed by a hair augmentation module, wherein during hair augmentation different hair strings from real images are added to data set images without hair on them, in order to help mole recognition when the mole is occluded by hair.

6. A method for providing a neural network ensemble for skin anomaly classification, the method comprising: at least 2, such as at least 4, preferably at least 5, more preferably at least 8, neural networks provided by the method according to any one of claims 1 to 5, said neural networks being ensembled to return a single classification probability score, classifying data as either of the two classes benign and malignant phenotype on a continuous scale from 0 to 1.

7. A neural network ensemble configured to classifying data as either of the two classes benign and malignant phenotype on a continuous scale from 0 to 1 according to the method of claim 6, wherein the neural network ensemble is further configured to provide data output processed at a processing stage as output data of the neural network skin anomaly classification.

8. A neural network ensemble according to claim 7, wherein at least two neural networks are provided, wherein the at least two neural networks are based on data sets having different selected characteristic according to claim 3 and/or claim 4.

9. A method of dermoscopic screening of a skin anomaly, with melanoma tumor classification from skin images, comprising the steps of a) from at least one macro image of a subject’s skin, comprising a high- resolution overview of a larger skin area of the subjects, performing a preliminary prediction, wherein a detection algorithm determines each mole by an image processing method, for each determined mole, a machine learning model, an image processing algorithm or a neural network according to claim 4 detects and selects moles more likely to show a malignant pattern, and provides a list of the selected moles with their coordinates [to enable a micro-scan of the selected moles for obtaining a high resolution image of each selected mole], and b) for at least one, preferably for each, of the selected moles selected in step a), from a high resolution micro image of the selected mole, or cluster of moles, performing a classification of the malignity for each of the moles using the neural network ensemble according to claim 6 or 7, wherein the method provides an output displaying the moles selected in step a) together with the classification of malignancy, on a continuous scale from 0 to 1 from benign to malignant phenotype of step b), together with a mole identifier or mole coordinates.

10. A method of dermoscopic screening of a skin anomaly according to claim 9, wherein the macro image(s) comprise(s) at least 10 %, such as at least 40 %, such as at least 75 % of the skin area of the subject.

11. A method of dermoscopic screening of a skin anomaly according to any one of claims 9 to 10, wherein the images in step a) and/or step b) are first processed by a quality control module, to verify that they meet the image requirements.

12. A method of dermoscopic screening of a skin anomaly according to any one of claims 9 to 11, wherein the preliminary prediction of step a) comprises:

For all moles detected by the detection algorithm, setting a bounding box around each mole and cropping the images of each boundary box and resizing to a image size about 50000 to 10000 pixels, preferably 256x256 pixels.

13. A method of dermoscopic screening of a skin anomaly according to any one of claims 9 to 12, wherein the neural network of step a) has been trained using images of malignant moles having an image size of from between 1024x1024 to 256x256 pixels.

14. A method of dermoscopic screening of a skin anomaly according to any one of claims 9 to 13, wherein the images in step a) and/or step b) are processed by a template matching module, wherein the template matching module performs mole pairing between micro and macro scan and provides a unique identifier for each paired mole.

15. A method of dermoscopic screening of a skin anomaly according to any one of claims 9 to 14, wherein a template matching module performs image matching between modules, wherein the module performs an examination and storage of moles position, and provides positioning information for moles selected in the macro-scan to enable and simplify micro scanning.

16. A method of dermoscopic screening of a skin anomaly according to any one of claims 9 to 15, wherein the high-resolution macro image of step a) comprises at least 2 000 pixels per cm² of imaged skin surface, preferably at least 4000 pixels per

17. A method of dermoscopic screening of a skin anomaly according to any one of claims 9 to 16, wherein the high-resolution micro image of step b) comprises at least 1 000 000 pixels per cm² of imaged skin surface, preferably at least 3 000 000 pixels per cm² of imaged skin surface.

18. A system for dermoscopic screening of a skin anomaly according to any one of claims 9 to 17, comprising a macro-camera, a micro-camera, a controller and computation unit, a graphical output device, and a database and/or storage device.

19. A method for dermoscopic screening of a skin anomaly using the method according to any one of claims 9 to 17 and a system for dermoscopic screening according to claim 18, wherein the macro-camera is used to capture at least one macro image of the skin surface of the patient, the controller and computation unit performs a preliminary prediction and provides a list of selected moles deemed more likely to show a malignant pattern together with their coordinates on the graphical output device, the micro-camera is used to capture a micro image per mole of at least one, preferably all, of the selected moles, and the controller and computation unit performs a classification of the malignity of each mole for which a micro image has been captured and provides an output on the graphical output device showing a list of the selected moles, their classification of malignancy and a mole identifier or mole coordinates.