FR3117643A1 - SYSTEMS AND METHODS FOR COUNTING, LOCATING AND VISUALIZING ACNE - Google Patents

Abstract

Systems and methods for counting, localizing and visualizing acne Systems, methods, and techniques for locating, counting, and visualizing acne are provided. An image is processed using a model trained to identify objects. The model may be a deep learning network (e.g. a convolutional neural network) configured for object classification with a detection focus on small objects. The image can be a frontal or profile facial image, processed end-to-end. The model identifies and localizes different types of acne. Occurrences are counted and visualized, such as by annotation of the source image. An example of an annotation is an overlay identifying a type and location of each occurrence. Accounts by type of acne allow the establishment of a score. A product and/or service may be recommended in response to acne identification (e.g. type, location, count and/or score).

Description

SYSTEMS AND METHODS FOR COUNTING, LOCATING AND VISUALIZING ACNE

Domain

The present application relates to the fields of computer image processing, convolutional neural networks and dermatology and more particularly to systems and methods for counting, locating and displaying acne and to electronic commerce systems and methods using them.

Context

Skin conditions such as acne often affect the face and other areas of the body can also have such conditions. In the case of acne, sebum or dead skin cells clog a follicle causing blemishes of various types. Although common among teenagers, it also occurs in people of other age groups. With a given facial image, an acne localization task aims to detect the possible presence of acne in the facial portrait.

Acne localization is useful in downstream applications related to dermatology and image visualization.

Summary

To this end, the subject of the invention is a method comprising:

- analyzing a source image to determine respective locations of acne occurrences; And

- visualization of acne occurrences on the source image for display;

- in which the source image is analyzed using a model based on parametric surfaces configured to:

- the partitioning of the source image into parametric skin surfaces,

- detection by focusing on small objects in the source image, and

- determining occurrences of acne and at least one type of acne in images based on detection by focusing on small objects in the source image; And

- in which the parametric surfaces are determined in accordance with a skin mask.

According to particular embodiments, the method comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations:

- the model is a deep learning neural network model configured for object classification and localization and configured to process the image on a pixel level, working end-to-end to directly detect an acne location without cropping the source image;

- the method is further configured for generating respective anchor boxes providing location information for each of the detected acne occurrences, wherein an anchor box form factor, to be used to define one anchor boxes, is calculated using k-means clustering from identified acne occurrences in an image dataset;

- the visualization of the occurrences of acne indicates a respective location for each of the occurrences on the source image as well as a respective type of acne for each of the occurrences on the source image;

- the visualization of acne indicates a respective type of acne for each of the occurrences on the source image;

- the method further comprises determining a count of acne occurrences;

- the method further comprises obtaining a recommendation for a specific product and/or service to treat the occurrences of acne;

- the method further comprises communicating with an electronic commerce system to purchase the product and/or service;

The invention also relates to a method comprising:

- analyzing a source image to determine respective locations of acne occurrences by focused detection on small objects in the source image; and detecting at least one type of acne in the source image;

- generating and providing an acne score in response to a count of occurrences;

- generation of a recommendation for a specific product and/or service to treat acne occurrences; And

- the generation of one or more occurrences of a virtual visualization including the occurrences of acne on the source image.

The invention also relates to a computer system comprising a circuit configured to provide:

- an interface for receiving a source image and returning an annotated source image that visualizes acne occurrences determined by a model configured to process the source image;

- a recommendation component configured to recommend a product and/or service to specifically treat at least some of the acne occurrences; And

- an electronic commerce transaction component to facilitate a purchase of the product and/or service;

where the model is configured for:

- detection by focusing on small objects in images; And

- the detection of at least one type of acne in images.

Systems, methods, and techniques are provided for locating, counting, and visualizing acne in accordance with embodiments. An image is processed using a model for the identification (classification) of objects. In one embodiment, the model is a convolutional neural network (CNN) configured for object classification with a detection focus on small objects. In one embodiment, the image is a facial image, in frontal or profile mode, processed end-to-end by the CNN, without cropping. In one embodiment, the model identifies and locates different types of acne. Acne occurrences are counted (e.g. by type) and visualized, e.g. by annotating the source image. An example annotation is an overlay identifying a type and location of each occurrence. Accounts by type of acne allow the establishment of a score. In one embodiment, a product and/or service may be recommended in response to the identification of acne (e.g. type, location, count and/or score). In one embodiment, a purchase is facilitated.

In one embodiment, a method is provided comprising: analyzing a source image to determine respective locations of acne occurrences; and visualizing acne occurrences on the source image for display; wherein the source image is analyzed using a model configured to: detect at least one type of acne in images; and detection by focusing on small objects in the images.

In one embodiment, the model is a deep learning neural network model configured for object classification and localization.

In one embodiment, the model is configured to process the image on a pixel level, working end-to-end to directly detect an acne location without cropping the source image.

In one embodiment, the model generates respective anchor boxes providing location information for each of the detected acne occurrences.

In one embodiment, an anchor box shape factor, to be used to define one of the anchor boxes, is computed using k -means clustering from occurrences of acne identified in an image dataset.

In one embodiment, the model is based on parametric surfaces and the method includes providing parametric surfaces of skin from the source image to the model for processing to detect occurrences of acne.

In one embodiment, the model is based on parametric surfaces and the method includes providing parametric surfaces of skin from the source image to the model for processing to detect occurrences of acne, and wherein the surfaces parametric parameters are determined in accordance with a skin mask.

In one embodiment, the visualization of the acne occurrences indicates a respective location for each of the occurrences on the source image.

In one embodiment, the acne visualization indicates a respective type of acne for each of the occurrences in the source image.

In one embodiment, the at least one type of acne includes one or more of retentional acne, inflammatory acne and pigmentary acne.

In one embodiment, the method includes determining a count of acne occurrences.

In one embodiment, the method includes obtaining a recommendation for a specific product and/or service to treat occurrences of acne.

In one embodiment, the method includes communicating with an e-commerce system to purchase the product and/or service.

In one embodiment, the source image comprises a facial image in frontal mode or in profile mode.

In one embodiment, the method includes acquiring the source image.

In one embodiment, a method is provided comprising: analyzing a source image to determine respective locations of acne occurrences; and generating and providing an acne score in response to a count of the occurrences; in which the source image is analyzed using a model configured for: detection by focusing on small objects in images; and detecting at least one type of acne in images.

In one embodiment, the acne score is in response to one or more of location, account, and type of acne.

In one embodiment, the method includes generating a recommendation for a specific product and/or service to treat acne occurrences.

In one embodiment, the method includes visualizing occurrences of acne on the source image.

In one embodiment, there is provided a computing device comprising circuitry configured to perform a method according to any of the embodiments.

In one embodiment, a computer system is provided comprising circuitry configured to provide: an interface for receiving a source image and returning an annotated source image that visualizes occurrences of acne determined by a model configured to process the source image ; wherein the model is configured for: detection by focusing on small objects in images; and detecting at least one type of acne in images.

In one embodiment, the computer system is configured to provide: a recommender component configured to recommend a product and/or service to specifically treat at least some of the acne occurrences; and, an electronic commerce transaction component to facilitate a purchase of the product and/or service.

[Figs. 1A, 1B and 1C] Figures 1A, 1B and 1C are facial images annotated using a visualization to show occurrences of acne by type and location, in accordance with one embodiment.

[Fig. 2] Fig. 2 is an illustration of an example of a skin mask based on facial landmarks for use when processing a facial image in accordance with one embodiment.

[Fig. 3] Figure 3 is an illustration of a facial image with acne visualization and mask visualization in accordance with one embodiment.

[Fig. 4] Figure 4 is an electronic commerce network diagram, in accordance with one embodiment, showing a client computing device configured to detect occurrences of acne in a source facial image and obtain a recommendation for a product and/or a service recommendation and to purchase these.

[Fig. 5] Figure 5 is a block diagram of a computing device in accordance with one embodiment.

[Fig. 6] Fig. 6 is a flowchart of operations according to one embodiment.

detailed description

In accordance with one embodiment, there is shown and described a model configured to process a source image to detect occurrences of acne. The model is configured to determine respective locations for acne occurrences, and thereby perform an acne location task. In one embodiment, the model is configured to return object frames or coordinates. In one embodiment, the model is also configured to classify occurrences into particular types of acne. In one embodiment, the source image is a full facial image.

In one embodiment, the model is a deep learning model. In one embodiment, the model involves focusing on small objects.

In one embodiment, the model is end-to-end and operates on a pixel level, meaning the model processes an entire image (e.g. of a face) and directly performs acne detections on the image. image without any cropping.

In the following description, various embodiments of a model for processing source images are described, where a first group of embodiments ("first model embodiments" of a "first type of model") are based on a modified YOLO object detection deep learning network and a second group of embodiments ("second model embodiments" of a "second type of model") are based on a deep learning network based on parametric surfaces. (See J. Redmon and A. Farhadi, YOLOv3: An Incremental Improvement, arXiv:1804.02767, April 8, 2018 URL: arxiv.org/abs/1804.02767 , hereinafter referred to as "YOLOv3").

It is understood that various features described in association with an embodiment of one of the model types are applicable to an embodiment of the second model type. By way of example, but not limitation, features of locating acne, detecting by type of acne, counting acne (e.g., by type), and viewing acne are applicable to both types. of model. In embodiments, the first and second model embodiments are useful for processing a full facial image, whether frontal or in profile, without cropping. In embodiments, the first and second model embodiments are useful for processing a complete facial image, guided by a skin mask generated from the face image. While some of these features relate to the operations of the model embodiments themselves such as acne detection features (e.g. location and classification), other features such as counting, viewing, etc are operations associated with a process provided as output from a model. These and other features are common among model embodiments.

Early model embodiments

Figures 1A, 1B and 1C show examples of visualization of acne following analysis of respective source images using the model, in accordance with one embodiment, where three types of acne are detected and viewed. Types include retentional acne, inflammatory acne, and pigmentary acne. The types of acne that can be detected depend on the training data used when training the model. This means that the model itself has an expansion capability to learn and detect other types of acne if applicable training data is provided. It is understood that in one embodiment fewer types of acne may be trained for detection.

As shown, in accordance with one embodiment, the model also supports acne detection on three facial views (profile/left side, profile/right side, and frontal modes) of the input. The model, in this embodiment, is viewpoint-invariant and can process images from any of the viewpoints/modes.

Figures 1A, 1B and 1C are simulated black and white (BW) images 100, 102 and 104 showing annotated source images visualizing occurrences of acne, in accordance with one embodiment. Images 100, 102 and 104 are converted from original color images and edited for patent compliance purposes, even though color images are used, in the embodiment, in practice so as to present a more faithful image of the subject. Image 100 is a left side (profile) view, image 102 is a right side (profile) view, and image 104 is a front view.

In the embodiment, the three-view annotation intentionally focuses on specific parts of the face on each view. For example, the frontal view 104 annotates acne around the mouth and in the center of the image because it is more visible on this view. Similarly, side views 100 and 102 annotate acne around the temples where frontal view 104 does not. In the embodiment, the training data is defined (marked) in this way for the respective views and the model is trained to identify opposite data at these locations accordingly.

In the embodiment, occurrences are marked with a type 106, a detection confidence measure (e.g., numeric value) 108, and an object frame 110. Acne may be closely adjacent on a face human and thus markings may overlap when viewed at a particular scale. In one embodiment (although not shown in the BW examples), a type of acne can be distinguished using different colors for the markings or by other means. Other types of marking can be used (e.g. color-coded object frames without text), etc. In this example, the type of acne is represented as "infl" (inflammatory), "rete" (retention) and "pigm" (pigmentation). In terms of a detection confidence measure, the value is on a scale of 0 to 1 so that a measure of 0.58 represents 58% confidence that the detected occurrence is acne. In the embodiment, additional filtering is used to select from detected acne occurrences for viewing. For example, a threshold of 50% is used to view occurrences at or above the threshold. The threshold may vary.

Typical structure of first model

In one embodiment, the main architecture of the model is based on a YOLOv3 object detection algorithm.

Object frame prediction (location)

In short, _YOLOv3 predicts object frames using dimension groups in the form of anchor boxes, predicting four coordinates ( t _x , t _y , th , t _w ) for each object frame ( sometimes referred to herein as "bbox"). A frame can be offset from an upper left corner of the image by ( c _x , c _y ). The frame width and height are predicted as offsets from cluster centroids (σ( t _x ), σ( t _y )). Frame center coordinates are predicted with respect to the location of the filter application using a sigmoid function. A sum-of-squares-error loss is used during training.

YOLOv3 describes a prediction of an objectness score for each object frame using logistic regression. A score must be 1 if the object frame prerequisite overlaps a ground reality object more than any other object frame prerequisite. If the object frame prerequisite is not the best, but overlaps a ground-truth object by more than a certain threshold, the prediction is ignored, using a threshold of 0.5. An object framework prerequisite is assigned in YOLOv3 for each ground truth object. If an object frame prerequisite is not assigned to a ground reality object, there is no loss of coordinate or class predictions, only object character.

Class prediction (eg, object prediction or acne by type)

Multiple marking classification is used for class prediction for each frame via independent logistic classifiers. Class predictions are trained using binary cross-entropy loss.

YOLOv3 describes frame predictions on three scales. From a basic feature extractor, several convolutional layers are added. The last of these predicts a 3-d tensor encoding object frame, object character, and class predictions. Accordingly, for 3 frames at each scale, the tensor is N x N x 3* (4 + 1 + 80) for the 4 object frame offsets, 1 object character prediction, and 80 class predictions.

Then the feature map of previous 2 layers is oversampled by 2x and a previous feature map is merged with the oversampled features using concatenation. A few more convolutional layers are added to process the combined feature map. A similar tensor is predicted at twice the size. A similar approach is used again to predict a third tensor for the third scale, combining computer calculation of priors as well as refined features from prior processing. Grouping by k - means is used to determine object frame prerequisites where 9 groups and 3 scales are chosen arbitrarily and the groups are evenly divided across all scales.

The backbone network for feature extraction described in YOLOv3 is a 53-layer convolutional network named DARKNET-53.

Amendment

In accordance with an embodiment given here, the model disclosed in YOLOv3 is modified to focus on smaller objects by:

• reduction in the size of the backbone network;
• retention of a single YOLO detection layer and recalculation of the size (shape factor) of the anchor box and
• Fine-tuning the model to the best performance on the training dataset for the present acne localization task.

When retaining a single YOLO detection layer, prediction is performed at a scale level in the embodiment. Thus, in the embodiment, the YOLO detection layer is the layer that handles a prediction of object frames. It receives a feature (e.g. an output from backbone layers) and outputs an object character, class prediction, and bbox location. The layer itself has no limit on the number of predictions. In one embodiment, during an inference, the layer will output a greater number of predicted frames. These are filtered using non-maximal suppression (NMS) with a manually established confidence threshold.

In the embodiment, filtering using NMS is focused, at least in part, on filtering out redundant detections on the same occurrence of acne. When the model predicts two or more frames on the same acne, the NMS filters out occurrences with lower confidence and retains the higher confidence frame on the specific occurrence.

In one embodiment, the model is trained to predict three object classes for the respective 3 types of acne. There is no additional or background class as YOLO does not do background prediction (e.g. YOLOv3 does not draw an object frame on a background or indicate that it this is the background. YOLOv3 only predicts target classes). The number and types of classes can be easily adjusted to fit the data format. For example, if more object classes (e.g. acne or other objects) were identified in the data, the model could be adjusted and trained to predict more classes.

With respect to tuning, in one embodiment, various operations have been performed including: 1) intensive data augmentation including: converting images to HSV color space and adding random color dither ( random value change within a range) on saturation and value channels; and the random affine transformation on the image; 2) the use of a multi-step learning rate scheduler; and 3) the use of an evolution algorithm to evolve the hyper-parameters from the above augmentations by multiple-run training and iterative selection of the best setup.

In one embodiment, as described in YOLOv3, the anchor box aspect ratio is calculated using k -means clustering of acne occurrences identified in the image data set (training) (eg as annotated by a human reviewer (eg and expert)). That is, the anchor boxes are grouped in terms of shape factor using k -means grouping operations. Anchor boxes are used to guide object frame prediction in YOLOv3. In other approaches, operations can find a parametric surface of the image that potentially contains the target object and then predict an object frame around the object. But this is quite a computationally demanding task.

It is understood that the technique was developed such that it was determined that there was no need to predict a raw box as the data set must have certain intrinsic characteristics. For example, a car dataset should always have a long rectangular bbox and a face dataset should have a near-square bbox. Using these principles, the image dataset is evaluated to cluster the size of the bbox and find some common sizes that are shared. These sizes are used as anchor boxes. It is then enough for the model to predict the offset with respect to the anchor box, which makes it possible to avoid a lot of computer calculation work. In one embodiment, the algorithm used follows the same approach. The algorithm first gets the set of object frames from the training annotation, extracts and formats the boxes as data pairs (width, height), then uses k -means ( a clustering technique for dividing data into k individual clusters) to find the cluster centroids. The number of groups ( k ) is established manually, and it must be coherent with the YOLO layer which admits a certain number of anchor boxes. In one embodiment, k = 5 for the model.

Table 1 shows a model structure per layer, in accordance with one embodiment, following the modifications thus described.

Layer Kind Filters Size/Extent Entrance Exit 0 Convolutional 16 3x3/1 1024x1024x3 1024x1024x16 1 Maxpool 2x2/2 1024x1024x16 512x512x16 2 Convolutional 32 3x3/1 512x512x16 512x512x32 3 Maxpool 2x2/2 512x512x32 256x256x32 4 Convolutional 64 3x3/1 256x256x32 256x256x64 5 Maxpool 2x2/2 256x256x64 128x128x64 6 Convolutional 128 3x3/1 128x128x64 128x128x128 7 Maxpool 2x2/2 128x128x128 64x64x128 8 Convolutional 256 3x3/1 64x64x128 64x64x256 9 Maxpool 2x2/2 64x64x256 32x32x256 10 Convolutional 512 3x3/1 32x32x256 32x32x512 11 Maxpool 2x2/1 32x32x512 32x32x512 12 Convolutional 1024 3x3/1 32x32x512 32x32x1024 13 Convolutional 128 1x1/1 32x32x1024 32x32x128 14 Oversample 2x2/1 32x32x128 64x64x128 15 Shortcut
(14, 8) Concatenated layer 14 and 8 outputs 16 Convolutional 256 3x3/1 64x64x384 64x64x256 17 Convolutional 24 1x1/1 64x64x256 64x64x24 18 YOLO

A model by itself does not predict a total count of object occurrences (e.g. acne) per class or in the aggregate directly. Thus, a count per class or aggregate may be determined by respective counting of acne occurrences per class from predictions (outputs) of the model, for example, after filtering the predictions, such as by the use of non maximum.

A comparison of the count determined from predictions to a count made from ground truths can be used to evaluate the model. Common object detection metrics depend on how one defines what counts as a "relevant response" and an "item not found". Often an overlap ratio threshold (intersection/union) is used, but this threshold can be set arbitrarily high or low, which significantly influences the accuracy of the figures.

In one embodiment, counts are generated and used as a means of evaluating model performance.

In one embodiment, counts are generated from model inference time usage for use in downstream tasks. Instead of (or in addition to) predicting precise acne locations, an API for the model can be configured to determine counts. Downstream applications may use counts or even scores that are based on predicted counts for tasks such as skin condition severity rating, product and/or service recommendation, etc.

Dataset

A facial image dataset that contains 348 facial images of 116 different subjects (3 views per subject) was collected. A panel of dermatologists tagged the images and identified any visible acne that could belong to any of three types: inflammatory acne, retentional acne, and acne pigmentosa. The identified acne was then saved in center point coordinate format to set ground realities for training and testing etc.

In one embodiment, a group of three (expert) dermatologists reviewed and annotated a set of images. Each expert annotated all of the images and identified the three types of acne. In one embodiment, the annotated data (the three annotation versions) were merged according to the following steps:

merging of the three versions. The total number of frames after this step is the simple addition of those from the original three annotations.

Filtering of redundant frames based on logic similar to NMS. That is, if the I/U (intersection/union) ratio of two frames is larger than a certain threshold, use the larger frame as the ground truth (GT). If multiple frames overlap and all have a high I/U ratio, use the larger frame as the GT.

Assessment

Acne localization performance was measured based on common object detection metrics. It includes a Precision (P), Recall (R), mAP and F1 score. Note that the mAP and F1 scores are direct formulations based on Precision and Recall. The performance evaluation for the localization task is shown in Table 2.

Class Pictures Targets P R mAP F1 together 69 1860 0.225 0.568 0.313 0.322 pigment 69 796 0.233 0.531 0.267 0.324 influence 69 275 0.246 0.636 0.443 0.355 stay 69 784 0.197 0.536 0.23 0.288

The following abbreviations for the three types of acne assessed in the current model are used in Table 2: pigm = pigmentation, infl = inflammation, rete = retentional, and where P = Precision, R = Recall.

The same model is evaluated against the counting task and the results are shown as follows in Table 3:

Pigmentation Inflammatory Retentional Percentage < 5 0.50725 0.82609 0.56522 Error by class 6,058 2.4638 5.2319 Average error 4.584541063

The calculated error is the absolute error (|#Predicted - #Ground reality|). The metrics against which the assessment is performed are percentage < 5 (i.e. the ratio of test cases showing absolute error < 5?), error per class, and mean error of the set of classes.

Although the above embodiments are described in relation to processing an entire facial image, in one embodiment a skin mask may be defined, for example, using facial landmarks, to indicate portions of the image that represent the skin. First model embodiments above may process the skin portions of a facial image as indicated by the skin mask.

In one embodiment, operations may reject an image that does not include a face and not process the image for location and acne count. In one embodiment, for example, operations may process the image to locate facial landmarks that are denied or further process the image accordingly.

Second embodiments model - Acne localization based on parametric surfaces

A possible drawback of the Yolo-based approach of the first embodiments above is that noisy annotations can lead to poor results. Therefore, second model embodiments are provided in accordance with a parametric surface-based approach that localizes acne pimples. In one embodiment, a parametric surface-based approach is applied to a full face image. In one embodiment, a parametric surface-based approach is applied using a skin mask to process one or more portions of the face image.

For example, in a mask-driven embodiment, at inference time, a source image of a face (e.g., a self-portrait image) is preprocessed by a facial landmark detector to define a mask of facial skin. An example of a skin mask 200 based on the benchmarks is shown in Figure 2 where portions of the mask are white showing areas to be treated for acne and portions without a mask are black showing areas which are not. processed. In this example, mask 200 shows an upper face component 202 for the forehead, a T-zone component 204 above the bridge of the nose, and a lower face component 206 for the cheeks, jaw line and the chin, where portions are omitted for an eye region (eg, 208), a lower nose region 210, and a lip region 212. The background around the face is also omitted from the mask.

At inference time, in the mask-bound embodiment, resolution parametric surfaces are created within the generated skin mask by scanning from the upper left corner to the lower right corner, with an extent of one-third ( ⅓) of parametric surface width. The parametric surface width is normalized by defining it as the width of the face divided by 15. Each parametric surface is passed through a trained convolutional neural network (i.e. a second model embodiment ) described below. In one embodiment, the model outputs a list of probabilities of the following classes: inflammatory, retentional, pigmented (three types or classes of acne), and healthy skin. In one embodiment, non-maximal suppression (NMS) is applied to select the best frames from the returned detected acne candidates.

In a non-mask embodiment, similar operations are performed using parametric surfaces to process the entire facial image, for example, including skinless portions such as background, hair, eyes and lips. For example, at inference time, in the non-mask-bound embodiment, resolution parametric surfaces are created within the entire image by scanning from upper left to lower right, with an extent of 'one-third (⅓) parametric surface width, as described. Each parametric surface is passed through a trained convolutional neural network (i.e. a second model embodiment) described below. In one embodiment, the model outputs a list of probabilities of the following classes: inflammatory, retentional, pigmented (three types or classes of acne), and healthy skin. In one embodiment, non-maximal suppression (NMS) is applied to select the best frames from the returned detected acne candidates. The healthy skin class gives the classifier an option other than one of the three acne classes.

In one embodiment, the selected frames are useful for performing an acne count, and using the source image, an acne visualization. In one embodiment, a filter is applied to select between these occurrences of the best frames of acne detected using a threshold confidence measure. For example, occurrences detected at the threshold or above the threshold (or only those above the threshold) are counted and/or visualized.

In accordance with one embodiment, Fig. 3 is an illustration of a snapshot 300 showing a source facial image 302 that is annotated to visualize acne (eg, 304, 306, and 308). Snapshot 300 also shows the source image annotated to show the applicable skin mask 310 (here, bounded by a broken line) determined for the face. In Figure 3, acne is visualized using circles around a center point of the frame of each detected occurrence of acne that is to be visualized. In one embodiment, in practice, the circles are of different colors or of different gray scale values to distinguish between the 3 classes of acne. In this illustration, circles are distinguished using a non-broken line style (eg 304) or one of two broken line styles (eg 306, and 308). In one embodiment, the skin mask is not visualized (not shown). The skin mask may not be visible because it has not been used or because it has been used, but we do not want to show it.

Second model type structure and training

In one embodiment, a model based on parametric surfaces includes a residual network architecture (e.g., RestNet-50, having 50 neural network layers) and three fully connected layers where a rectified linear activation function (e.g. LeakyReLU (leaky rectified linear unit) is interleaved between adjacent fully connected (FC) layers. The final layers appear as FC1->LeakyRelu1 -> FC2 -> LeakyRelu2 -> FC3. (See, He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015-10-12). “Deep Residual Learning for Image Recognition”. arXiv:1512.03385.)

To create a model dataset based on parametric surfaces, in accordance with one embodiment, 2450 healthy parametric surfaces and 3577 acne parametric surfaces (parametric surfaces including occurrences of any of the three classes previously described here) were respectively sampled from the full face images of the data set described above with reference to the first model embodiments. In a training phase, a network is trained with a standard cross-entropy loss function, data augmentation including random affine transformation, random horizontal flip, and channel permutation.

In one embodiment, frontal images were used for masking and annotation.

To process a source image to detect occurrences of acne, by type, in accordance with an embodiment where a mask is used, the source image is processed using marker detection to define a mask applicable, defining regions of the face having skin for analysis. In accordance with the applicable mask, a plurality of overlapping parametric surfaces (responsive to a parametric surface size and extent) are extracted from the skin image and processed by the model. The model produces a list of probabilities of the following classes: inflammatory, retentional, pigmented (three types or classes of acne) and healthy skin. Mask axis operations to ignore eyes, lips, hair, background in source image. Multiple detected occurrences of the same acne are filtered to select one/the best occurrence (e.g. using an NMS). A second filter is applied to select detected occurrences having at least a minimum confidence level.

Similar to what has been described with reference to embodiments of the first model, in one embodiment using a type of second model, detected occurrences of acne per type are counted. In one embodiment, a score is computed. In one embodiment, the detected acne is visualized in association with a source image. In one embodiment, the mask is visualized in association with the source image, showing the regions of the skin that have been processed.

Downstream applications

The following description relates to any of the first model embodiments and second model embodiments. Thus, in accordance with the embodiments, analysis operations use a model that can be configured to output locations of detected acne occurrences in a source image (e.g., of skin such as on a face in frontal or profile mode) where different types of acne (object classes) can be distinguished by the model. Analysis or other determination operations may count occurrences, for example, to determine an aggregate count and/or count for each type.

Location, type, and/or account define acne data that is useful in downstream applications. In one embodiment, the occurrences of acne are visualized on the source image or displayed using visualization operations. In one embodiment, the source image is annotated (eg, using overlays or other techniques) to indicate a location. In one example, the location consists of respective locations for each of the instances on the source image. Visualization operations, in one embodiment, indicate a respective type of acne for each of the occurrences on the source image. View operations, in one embodiment, provide a view of the count.

Acne data (e.g. count and type (grading)) are useful for predicting skin condition scores in operations. For example, more retentional acne and less inflammatory acne indicate more moderate severity and vice versa.

In one embodiment, based on acne data and/or information derived therefrom (eg, scores) operations recommend products and/or services to target acne. Transactions (eg, e-commerce transactions) facilitate a purchase, in accordance with one embodiment.

In one embodiment, visualization operations guide application of a product, for example, using location to indicate where to apply. In one example, the source image is used in the visualization to provide a user-specific tutorial through a graphical user interface (GUI).

Various paradigms and computational systems can be considered for practical applications of the model and its output. In one example, a computer network includes at least one client computing device (eg, for a dermatologist or consumer) and a web-based electronic commerce computing system. In the embodiment, the client computing device is enabled to determine occurrences of acne in a source image and obtain a recommendation for a product and/or a service recommendation and purchase the same.

A model for analyzing source images and providing acne data as output may be configured for run-time execution on a user device or on a server or other remote device. An application can be configured to obtain a source image and use the model to generate acne data. Acne data may be used to provide one or more of a diagnosis such as an acne score, a visualization of acne on the source image, and a recommendation for a product or service to specifically treat acne .

Figure 4 is a block diagram of an exemplary computer network 400 in which a user operated personal computing device 402 404 is in communication, via a communications network 406, with server computing devices located remotely, namely a server 408 and a server 410. In one embodiment, the user 404 is a consumer. The figure also shows a second user 412 and a second computing device 414 configured for communication via a communications network 406. In one embodiment, the second user 410 is a dermatologist. In one embodiment, server 408 is configured to provide an instance of the model (418) for processing images to generate acne data and a recommendation for a product and/or service to treat acne. Server 408, in one embodiment, generates an acne score and recommendation from received acne data (e.g., without having processed the image itself). In one embodiment, server 408 generates acne data including location data for each detected acne occurrence such as to be used to visualize acne. In one embodiment, server 408 generates the visualization to provide it to another device for display. In one embodiment, server 408 generates acne data including one count per type (eg, non-viewing), to generate a score.

In one embodiment, server 410 provides an e-commerce interface for purchasing products and/or services, for example, as recommended by server 408.

In one embodiment, computing device 402 is intended for personal use by user 404 and is not available to the public. However, services from the server are available to the public. Here, the audience includes registered users and/or customers, etc. A publicly available computing device 416, such as can be found in a traditional store, is also coupled to the 406 network.

Computing device 402 is configured to perform acne location, etc., as described herein, i.e., assess acne location and determine count, etc. In the embodiment, a model (CNN) 418 is stored and used on an edge computing device 402. In the embodiment, a second instance of the model 418 is stored at server 408 and provided for use by other users. other computing devices such as via cloud service, web service, etc. for analysis of image(s) received from a computing device (e.g. 416, etc.).

Computing device 402 is configured to communicate with server 408, for example, to provide acne data (which may include score data) and to receive product/service recommendations in response to the acne data and/or to other information about the user, e.g. age, sex, etc. Computing device 402 (or server 408 on its behalf) is configured to communicate with server 410 to obtain e-commerce services to purchase the recommended product(s) and/or service(s). ).

Computing device 402 is shown as a hand-held mobile device (e.g., smart phone or tablet). However, the physical device can be another computing device such as a laptop, desktop computer, workstation, etc. Locating and counting acne, etc., as described herein can be implemented on other types of computing device. Computing devices 402, 414, and 416 can be configured using one or more native applications or browser-based applications, for example.

The computing device 402, in the embodiment, includes a user device, for example, for acquiring one or more images such as a photo of the skin, in particular a face, and processing the one or more images in order to generate respective acne data, etc. These activities are indicated as a realization of skin diagnosis. A skin diagnosis can be performed in conjunction with a skin treatment plan where images are acquired periodically and analyzed to determine skin scores such as for acne, as described. Scores can be stored (locally, remotely or both) and compared between sessions, for example to show trends, improvement, etc. Skin scores and/or skin images may be accessed by user 404 of computing device 402 and made available (e.g., via server 408 or otherwise (electronically) communicated via the communication network 406) of another user (eg, a second user 412) of computer system 400 such as a dermatologist. The second computing device 414 can also perform skin diagnosis, as described. It may receive images from a remote source (e.g., computing device 402, or server 408, etc.) and/or may capture images via an optical sensor (e.g., camera) coupled thereto or any other way. Template 418 can be stored and used from second computing device 414 or server 408, as described.

An application can be provided to perform skin diagnosis, suggest one or more products and monitor skin changes following one or more product applications (which can define treatment sessions in a treatment plan) on a time period. The computer application may provide an operational flow such as a series of graphical user interfaces (GUIs) of instructions and/or other user interfaces, which are typically interactive and receive user input, to perform any of the following activities:

• a skin diagnosis such as for acne;
• a product recommendation such as for a treatment plan;
• a product purchase or other acquisition;
• a product application reminder, instruction and/or record (e.g. logging) for respective treatment sessions;
• subsequent skin diagnosis (e.g. one or more follow-ups); And
• a presentation of results (eg comparative results);

such as in accordance with a treatment plan schedule to monitor the progress of a skin treatment plan. Any of these activities can generate data that can be stored remotely, e.g., for review by user 412, for review by another individual, for aggregation with other user data ( e.g. to measure in aggregate the effectiveness of treatment plans), etc.

Comparative results (e.g., before and after results) may be presented via the computing device 402, during and/or at the time of completion, etc. of a treatment plan. As noted, aspects of skin diagnosis can be performed on computing device 402 or by a remotely coupled device (e.g., cloud server or other arrangement).

Figure 5 is a block diagram of computing device 402, in accordance with one or more aspects of the present disclosure. The computing device 402 includes one or more processors 502, one or more input devices 504, a gesture-based I/O device 506, one or more communication units 508, and one or more output devices 510. The computing device 402 also includes one or more storage devices 512 storing one or more modules and/or data. In one embodiment, the modules include a model 418, an application 516 having components for a graphical user interface (GUI 518) and/or an operational flow for process monitoring (e.g., process monitor 520), an image acquisition 522 (eg an interface) and a process/product selector 530 (eg an interface). The data may include one or more images for treatment (e.g., image 524), diagnostic data 526 (e.g., acne data, respective scores, ethnicity, gender, or other diagnostic data). (user), treatment data 528 such as log data related to specific treatments, treatment plans with schedules such as for reminders, etc.)

The 516 application provides the functionality to acquire one or more images such as video and process the images to determine a skin diagnosis with a deep neural network as provided by the 418 model.

The storage device(s) 512 can store additional modules such as an operating system 532 and other modules (not shown) including communication modules; graphics processing modules (eg for a GPU of 502 processors); a card module; a contact module; a diary module; a photo/gallery module; a photo editor (image/multimedia); a media player; and/or a streaming module; social networking applications; a browser module; etc Storage devices can be listed here as storage units.

Communication channels 538 may couple each of components 502, 504, 506, 508, 510, 512, and any modules (e.g., 418 and 516) for communications between components, whether in communication, physically and/or operationally. In some examples, the communication channels 338 may include a system bus, a network connection, an inter-process communication data structure, or any other data communication method.

The one or more processors 502 may implement functionality and/or execute instructions within the computing device 402. For example, the processors 502 may be configured to receive instructions and/or data from the storage devices 512 to to execute the functionality of the modules shown in Figure 5, among others (eg the operating system, applications, etc.). Computing device 402 can store data/information on storage devices 512. Some of the functionality is described further below. It is understood that operations may not fit exactly within the modules 418 and 516 of FIG. 5 so that one module can provide support for the functionality of another.

Computer program code for performing operations may be written in any combination of one or more programming languages, e.g. an object-oriented programming language such as Java, Smalltalk, C++ or the like, or a conventional procedural programming language, such as the "C" programming language or similar programming languages.

Computing device 402 may generate output for display on a screen of gesture-based I/O device 506 or in some instances, for display by a projector, monitor, or other display device. It is understood that the gesture-based I/O device 506 may be configured using a variety of technologies (e.g., related to input capabilities: resistive touch screen, acoustic wave touch screen surface, capacitive touchscreen, projectively capacitive touchscreen, pressure-sensitive screen, acoustic pulse recognition touchscreen, or other presence-sensitive screen technology; and relating to capabilities output: liquid crystal display (LCD), light-emitting diode (LED) display, organic light-emitting diode (OLED) display, dot-matrix display, e-ink, or similar monochrome or color display).

In the examples described herein, the gesture-based I/O device 506 includes a touch screen device capable of receiving as input a touch interaction or a gesture from a user interacting with the touch screen. This gesture may include a tapping gesture, a swiping or swiping gesture, a typing gesture, a pause gesture (e.g. where a user touches the same location on the screen for at least a threshold time period) or the user touches or points to one or more locations of the gesture-based I/O device 506. The gesture-based I/O device 506 may also include a non-tapping gesture. Gesture-based I/O device 506 may output or display information, such as a graphical user interface, to a user. Gesture-based I/O device 506 may present various applications, functions, and capabilities of computing device 402 including, for example, application 516 for acquiring images, viewing images, processing images, and displaying new images, messaging applications, telephone communications, contact and calendar applications, web browsing applications, gaming applications, e-book applications and financial, payment and other applications or functions among others.

Although this disclosure illustrates and discusses a gesture-based I/O device 506 primarily as a display screen device with I/O capabilities (e.g., a touch screen), others Examples of gesture-based I/O devices can be used which can detect motion and which may not include a screen per se. In this case, computing device 402 includes a display screen or is coupled to a display device to present new images and application GUIs 516. Computing device 402 may receive gesture-based input from a pointing pad/touchpad, one or more cameras, or other input device responsive to presence or gesture, where presence means aspects of a user's presence including for example movement of all or part of the user.

One or more communication units 508 may communicate with external devices (e.g., server 408, server 410, second computing device 412) as for the purposes as described and/or for other purposes (e.g., a printing) such as via a communications network 404 by transmitting and/or receiving network signals on the one or more networks. Communication units may include various antennas and/or network interface cards, chips (e.g. Global Positioning Satellite (GPS)), etc. for wireless and/or wired communications.

Input devices 504 and output devices 510 may include one or more any buttons, switches, pointing devices, cameras, a keyboard, a microphone, one or more sensors (e.g., biometrics, etc.), a monitor -speaker, a bell, one or more lights, a haptic (vibrating) device, etc. One or more of these may be coupled via a universal serial bus (USB) or other communication channel (eg 338). A camera (an input device 504) may be facing forward (i.e., same side) to allow a user to capture image(s) using the camera while looking at the gesture-based I/O device 506 to take a "selfie" (self-portrait).

The one or more storage devices 512 may take different shapes and/or configurations, for example, such as short-term memory or long-term memory. 512 storage devices can be configured for short-term storage of information such as volatile memory, which does not retain stored contents when power is removed. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), etc. Storage devices 512, in some examples, also include one or more computer-readable storage media, for example, to store larger amounts of information than volatile memory and/or to store that information long-term, retain information when the power supply is removed. Examples of nonvolatile memory include magnetic hard disks, optical disks, floppy disks, flash memories, or forms of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM).

In one embodiment, as noted, other user-facing computing devices (e.g., 414 and 416) are configured similarly to computing device 402. In one embodiment, for a dermatologist user, the he 516 application is configured differently, for example, with functionality to work with multiple patients. In one embodiment, for a device located in a store, the application 516 is configured differently, for example, with functionality for working with multiple consumers, for store branding, etc.

In another example, a network model 418 is located remotely and a computing device (e.g., any of 402, 414, and 416) is activated via application 516 as appropriately configured, to communicate a image for processing and returning diagnostic data (eg acne data). In such an example, application 516 is configured to perform these activities.

Although not shown, a computing device may be configured as a training environment to train the neural network model 514 for example using the network as shown in Fig. 5 together with data appropriate training and/or test.

Model 418 can be adapted to a lightweight architecture for a computing device which is a mobile device (e.g. smart phone or tablet) having less processing resources than a "larger" device such as a laptop , desktop computer, workstation, server, or other computing device of comparable generation.

Although not shown in detail, the server devices shown are computing devices similar in basic construction (processors, storage devices, communication devices, input and output devices) to computing device 402, but in the form of a server-side device. That is, often server devices are not configured with consumer-oriented hardware, and have fewer input and output devices, fewer user applications, server-oriented I/O, etc.

Figure 6 is a flowchart of operations 600 in accordance with one embodiment.

At 602, a source (face) image is received. For example, the source image is a self-portrait image captured by a camera of a computing device.

Operations in 604 are optional and shown in a dashed line style.

A face skin mask is determined to be used in order to guide an image processing.

At 606, the operations process the face image using a model to generate localized occurrences of acne by type (e.g., one of three classes). In one embodiment, the model is configured to: detect at least one type of acne in images; and detection by focusing on small objects in images. In one embodiment the model is a first model embodiment. In one embodiment the model is a second model embodiment. In one embodiment, the treatment is guided by a skin mask as produced by the operations at 604.

At 608, the detected occurrences are filtered as described above to select better locations and occurrences with higher confidence levels. Thus, the final detected acne is determined, by type.

At 610, an account is determined, by type. At 610, a score is generated, for example, representing a measure of severity. At 612, the final detected acne is viewed in association with the source image. For example, the image is annotated to show the final detected acne by type. In one embodiment, the image has annotations superimposed on it. At 612, the score is presented.

In 614, a recommendation is obtained. In 614, the recommendation is presented. In one embodiment, the score is provided to an e-commerce service to generate the recommendation for a product and/or service.

In one embodiment, operations 600 may be performed by a single computing device (eg, device 402), in communication with another device (eg, 408 and/or 410), as appropriate.

In one embodiment, operations are performed by more than one computing device (eg, 402, 408, and 410). For example, operations at 606 include communicating the source image to a computing device 408 providing the model as a service. This computing device 408 can also filter (step 608). This device 408 can also count acne (step 610). This device 408 can also visualize the acne (step 612) and return the source image on which the visualization and the count and/or a score are superimposed (e.g. for a presentation by the device 402). A third computing device 410 may provide the recommendation, for example, in response to a query including the score. In 600 operations, the steps shown here can be divided into more than one step or combined into fewer steps.

Practical implementation may include any or all of the features described here. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, program products, and other ways, combining the features described herein . A number of embodiments have been described. Nevertheless, it is understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. Additionally, other steps may be included in, or steps may be eliminated from, the described process, and other components may be added to, or removed from, the described systems. Accordingly, other embodiments are included within the scope of the following claims.

Throughout the specification and claims of this specification, the words "include", "contain" and variations thereof mean "including, but not limited to" and they are not intended to exclude (and do not exclude) other components, integers or steps. Throughout this thesis, the singular includes the plural unless the context indicates otherwise. In particular, when the indefinite article is used, the memoir should be understood as considering a plurality as well as a singularity, unless the context indicates otherwise.

The features, integers, characteristics, or groups described in conjunction with a particular aspect, embodiment, or example of the invention are intended to apply to any other aspect, embodiment, or example unless there is an inconsistency between them. . All of the features disclosed herein (including the claims, abstract and accompanying drawings), and/or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of these features and/or steps are mutually exclusive. The invention is not limited to the details of any of the aforementioned examples or embodiments. The invention extends to any new feature or new combination of features of the features disclosed in this specification (including the claims, the abstract and the accompanying drawings) or to any new step, or any new combination, of the steps of any method or process disclosed.

Claims (10)

Process comprising:
analyzing a source image to determine respective locations of acne occurrences; And
visualizing acne occurrences on the source image for display;
where the source image is analyzed using a model based on parametric surfaces configured to:
the partitioning of the source image into parametric skin surfaces,
detection by focusing on small objects in the source image, and
determining occurrences of acne and at least one type of acne in images based on detection by focusing on small objects in the source image; And
wherein the parametric surfaces are determined in accordance with a skin mask. A method according to claim 1, wherein the model is a deep learning neural network model configured for object classification and localization and configured to process the image on a pixel level, working end-to-end to directly detect an acne location without cropping the source image. A method according to claim 1, further configured to:
generating respective anchor boxes providing location information for each of the detected acne occurrences;
wherein an anchor box shape factor, to be used to define one of the anchor boxes, is computed using clustering by k -means from identified acne occurrences in a image data set. A method according to claim 1, wherein the visualization of the occurrences of acne indicates a respective location for each of the occurrences on the source image. A method according to claim 1, wherein the visualization of acne indicates a respective type of acne for each of the occurrences in the source image. A method according to claim 1, further comprising determining a count of acne occurrences. A method according to claim 1, further comprising obtaining a recommendation for a specific product and/or service to treat the occurrences of acne. A method according to claim 1, further comprising communicating with an electronic commerce system to purchase the product and/or service. Process comprising:
analyzing a source image to determine respective locations of acne occurrences by focused detection on small objects in the source image; and detecting at least one type of acne in the source image;
generating and providing an acne score in response to a count of occurrences;
generating a recommendation for a specific product and/or service to treat acne occurrences; And
generating one or more occurrences of a virtual visualization including the acne occurrences on the source image. A computer system comprising a circuit configured to provide:
an interface for receiving a source image and returning an annotated source image that visualizes occurrences of acne determined by a model configured to process the source image;
a recommender component configured to recommend a product and/or service to specifically treat at least some of the acne occurrences; And
an e-commerce transaction component to facilitate a purchase of the product and/or service;
where the model is configured for:
detection by focusing on small objects in images; And
detecting at least one type of acne in images.