WO2023217608A1

WO2023217608A1 - Method for predicting the development of cutaneous signs over time

Info

Publication number: WO2023217608A1
Application number: PCT/EP2023/061690
Authority: WO
Inventors: Benjamin ASKENAZI; Frédéric FLAMENT; Matthieu PERROT; Julien Despois; Panagiotis-alexandros BOKARIS; Hussein JOUNI
Original assignee: LOreal SA
Current assignee: LOreal SA
Priority date: 2022-05-10
Filing date: 2023-05-03
Publication date: 2023-11-16
Anticipated expiration: 2024-11-10
Also published as: CN119677453A; FR3135557A1; FR3135557B1

Abstract

The method relates to a cosmetic and non-therapeutic method for predicting the appearance of an area of interest of a user, the method comprising (i) acquiring an input image of the area of interest, processing (31) the input image, on the one hand, with (ii) an image segmentation and analysis algorithm for determining the optical properties of the area of interest and extracting features of the cutaneous signs and, on the other hand, with (iii) an algorithm for predicting and/ or displaying the development of the cutaneous signs.

Description

Method for predicting the development of cutaneous signs over time

The subject of the invention is a cosmetic and non-therapeutic method for predicting the development of cutaneous signs in an area of interest of a user over time and in motion. It also targets a system for implementing this method.

More generally, a cosmetic product is a product as defined in Regulation (EC) No 1223/2009 of the European Parliament and of the Council dated 30 November 2009 relating to cosmetic products.

Prior art

Computer-assisted analysis of the skin has become widespread over the course of the last decade with the availability of controlled lighting systems and of digital image capture and processing capabilities.

Because of the importance attributed to the appearance of the face, computer-assisted analysis studies have concentrated on the skin of the face. There are commercially available systems for imaging the skin of the face which can capture digital images in a controlled manner. These systems are often coupled to computer analysis systems for displaying and quantifying the visible features of the skin in standard white light images such as hyperpigmented spots, lines and texture, as well as the non-visible features in fluorescence or hyperspectral absorption images such as UV spots.

The document US8218862 discloses a method for generating a cutaneous mask delimiting a region of interest (ROI) in an image of skin comprising:

detecting the skin in the image of skin in order to generate a skin map, skin detection comprising:

performing a raw skin segmentation operation on the image of skin;

converting the raw skin segmented image of skin into an image with a colour space having at least three channels; and

filtering two of the at least three channels;

generating a melanin index image on the basis of the filtered channels; and

performing a thresholding operation on the melanin index image in order to separate the cutaneous and non-cutaneous areas in the cutaneous image;

providing an initial contour based on the map of the skin; and

optimizing the initial contour in order to generate a contour of the cutaneous mask, the skin map and the cutaneous mask each comprising at least one natural limit of the skin.

The six "Atlas des signes du viellissement cutané [Atlases of the signs of cutaneous ageing]" published by MED’COM since 2007 are also known. These works systematically study and establish a characterization and a classification of skin linked to age and geography.

By identifying several features of the face, they make it possible for professionals to qualify and to quantify the signs of age as a function of their development. They define these criteria - openness of the pores, depth of the lines on the forehead and crow's feet - and assign them a severity score, which starts from zero and can go up to 9.

These atlases represent a portion of the knowledge on which the tools for evaluating cutaneous signs are based. The performance thereof for evaluating cosmetic products or for modelling the kinetics of the development of ageing based on hundreds of thousands of photos is known.

In order to automate the diagnostics, the data from the atlases were used to feed the algorithms, by associating a photo with a score. Then, they were trained to score these various observation areas.

By virtue of thousands of selfies studied across the globe, algorithms based on artificial intelligence have learnt to recognise the various signs of the ageing of the face in any type of photograph.

By combining the data originating from selfies with information on the pace of life, the sleep or the surroundings of users, these diagnostics now make it possible to offer a user an estimated "ageing curve", which can be compared to an average of people of the same age.

The patent application WO2020169214 filed by the applicant discloses predicting cutaneous ageing by taking into account the atlases and also the surroundings or the habits of a user, notably sleep time, pollution and/or living place. By combining the data originating from selfies with information such as the pace of life, the sleep or the surroundings of users, this prediction offers an "ageing curve" based on these data.

There is a need to improve the performance of the above methods further in order to better:

Provide complete and accurate diagnostics of the face combining diverse features.

Personalize cosmetic care/a product based on the diagnostic, either by creating a new product or by selecting it from a given range.

Personalize the user experience by displaying the results of the diagnostic and/or by simulating their development for knowledge sharing.

Predict the future development of an area of interest of a user.

Definition of the invention

The subject of the invention is a cosmetic and non-therapeutic method for predicting the appearance of an area of interest of a user, the area of interest being divided into regions of interest comprising cutaneous signs, the method comprising (i) acquiring an input image of the area of interest, processing the input image, on the one hand, with (ii) an image segmentation and analysis algorithm for determining optical properties of the area of interest and extracting features of the cutaneous signs and, on the other hand, with (iii) an algorithm for predicting and/or displaying the future development of the cutaneous signs from the input image by comparing the features of the cutaneous signs extracted from the input image with reference data prestored in a reference database, (iv) ideally or acceleratedly simulating the future appearance of the area of interest as a function of time. By virtue of the method according to the invention, the cutaneous signs are analysed one by one, for markedly improved prediction in terms of reliability and in terms of overall performance, notably relative to WO2020169214. The features of each cutaneous sign are detected, analysed and compared one by one with identical features extracted from a reference database. In fact, the analysis of the cutaneous signs is much finer and more complete than that of the prior art, which was an overall analysis of the area of interest.

The invention makes use of the best of clinical and instrumental evaluation in a single method.

The invention brings together the best of two objective means for measuring the features of the face in the cosmetic field, that is to say:

A clinical means with deep learning algorithms based on artificial intelligence which are trained with dermatologists and experts calibrating with the reference of published photographic clinical scales. Automatic clinical visual evaluation is within reach.

An instrumental means with specific image segmentation and analysis software for measuring optical properties such as the light/skin interactions.

The "appearance" can be understood as being the overall appearance of the area of interest, either in motion or fixed.

In addition, it can be imagined that each datum newly collected by the method according to the invention can feed the reference data prestored in the reference database:

In order to better simulate the ageing and the appearance,

In order to further improve the adjustment of the detection capabilities and of the accuracy of the software.

Accurate evaluations of cutaneous signs surpassing those of the prior art in terms of accuracy and of inclusivity are obtained in accordance with the method of the invention, by virtue of processing the images. The system is based on the knowledge from decades of research and of data covering all the various skin types stored in the reference database.

Examples of "areas of interest" can include the hair, the nails, the face, the scalp, the neck, the regions of the eyes, the ears, the torso, the arms, the legs and/or other body parts.

A "region of interest" is a portion of the area of interest. For example, if the area of interest is the face, the contour of the eyes, the forehead, the cheekbones and the contour of the lips are associated areas of the interest.

Preferably, the external body part is the face.

The "reference database" can comprise:

Data stored with one or more sequences of images comprising at least two images,

One or more atlases displayed on screen or printed on a medium or in a form stored on a computer storage medium, such as the atlas described in the patent application WO2011/141769, each atlas representing various gradations of at least one feature of bodily typology,

Synthetic images,

Internet publications.

The data can be processed by artificial intelligence (AI) algorithms which can include fuzzy logic, neural networks, genetic programming and decision tree programming.

All engines can be trained on the basis of inputs such as information on the product, advice from experts, a user profile or data based on sensory perception. Using an input, an AI engine can implement an iterative learning process.

Training can be based on a wide variety of learning rules or training algorithms from scientific publications, such as the publications:

Frederic Flament et al., "Effect of the sun on visible clinical signs of aging in Caucasian skin", Clinical, Cosmetic and Investigational Dermatology, 2013, 6, 221-232.

Frederic Flament et al., "Effets de la saisonnalité et d'une photo-protection quotidienne sur certains signes du visage des femmes chinoises [Effects of seasonality and of daily photoprotection on certain signs of the face of Chinese women]", International Journal of Cosmetic Science, 2017, 39, 256-268.

Frédéric Flament et al., "Comment une fatigue induite par une journée de travail peut modifier certains signes faciaux chez des femmes caucasiennes d'âge différent [How fatigue induced by a day of work can modify certain facial signs in Caucasian women of different ages]", International Journal of Cosmetic Science, 2017, 39, 467-475.

Frédéric Flament et al., "Une pollution urbaine extérieure chronique sévère altère certains signes de vieillissement du visage chez les femmes chinoises [Severe chronic external urban pollution alters certain signs of ageing of the face in Chinese women]", Journal international des sciences cosmétiques [International Journal of Cosmetic Science] 2018, 40, 467-481.

The invention also relates to a cosmetic and non-therapeutic method for predicting the appearance of an area of interest of a user, the area of interest being divided into regions of interest comprising cutaneous signs, the method comprising (i) acquiring an input image of the area of interest, processing the input image, on the one hand, with (ii) an image segmentation and analysis algorithm for determining optical properties of the area of interest and extracting features of the cutaneous signs and, on the other hand, with (iii) an algorithm for predicting and/or displaying the rejuvenation of the cutaneous signs from the input image by comparing the features of the cutaneous signs extracted from the input image with reference data prestored in a reference database, (iv) ideally or acceleratedly simulating the future appearance of the area of interest as a function of time.

The invention also relates to a cosmetic and non-therapeutic method for predicting the appearance of an area of interest of a user, the area of interest being divided into regions of interest comprising cutaneous signs, the method comprising (i) acquiring an input image of the area of interest, processing the input image, on the one hand, with (ii) an image segmentation and analysis algorithm for determining optical properties of the area of interest and extracting features of the cutaneous signs and, on the other hand, with (iii) an algorithm for predicting and/or displaying, in the event that the performance of a product is known, an effect produced on cutaneous signs from the input image by comparing the features of the cutaneous signs extracted from the input image with reference data prestored in a reference database, (iv) ideally or acceleratedly simulating the future appearance of the area of interest as a function of time.

The invention also relates to a cosmetic and non-therapeutic system for predicting the appearance of an area of interest of a user, comprising communication means which are suitable for communicating with a user computing device and configured to import, from an image library of the user computing device, at least one input image comprising a representation of the area of interest of the user; processing means configured to subject the input image to (ii) an image segmentation and analysis algorithm for determining optical properties of the area of interest and extracting features of the cutaneous signs, and to (iii) an algorithm for predicting and/or displaying the future development of the cutaneous signs from the input image by comparing the features of the cutaneous signs extracted from the reference image with reference data prestored in a reference database, (iv) means for ideally or acceleratedly simulating the future appearance of the area of interest as a function of time.

The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method described above.

The invention relates lastly to a computer-readable medium comprising instructions which, when they are executed by a computer, cause the latter to implement the method described above.

Preferred embodiments

Preferably, the cosmetic and non-therapeutic method for detecting and for quantifying cutaneous signs in an area of interest of a user according to the invention exhibits one or more of the following features, taken alone or in combination:

Processing the input image includes an overall analysis of the area of interest as well as an analysis of regions of interest.

The algorithm for predicting and/or displaying is a deep learning algorithm based on artificial intelligence.

The algorithm for predicting and/or displaying the future development, the rejuvenation or the produced effect of the cutaneous signs from the input image is based on a photorealistic or realistic physical 3D/2D rendering algorithm.

The algorithm for predicting and/or displaying the future development, the rejuvenation or the produced effect of the cutaneous signs from the input image is based on a neural rendering algorithm.

Processing the input image comprises evaluating at least two cutaneous signs.

It comprises (v) processing one or more pieces of information describing the lifestyle of the user.

It comprises (vi) estimating a rate of acceleration of ageing from each cutaneous sign as a function of the information describing the lifestyle of the user, of the input image and of the prestored reference data.

It comprises (vii) evaluating the appearance of each cutaneous sign as a function of time, based on the rate of acceleration of ageing and on the input image.

It comprises (viii) simulating the future appearance of the area of interest in an exposome of the user and possibly (viii) simulating the future appearance of the area of interest in an ideal exposome.

The regions of interest are chosen from the forehead, the contour of the eyes, the nose, the cheekbones, the contour of the lips, the chin, the lips and the cheeks.

The machine learning model is a pretrained convolutional neural network.

The cutaneous signs are chosen from pores, skin texture, a line, a fine line, crow's feet, sagging, shadows under the eyes, skin grain, a facial expression, morphology, redness, sheen.

The features of the cutaneous signs are chosen from openness, surface area, volume, depth, width, length, colour, degree of moisturization, degree of sheen, distribution, intensity, reflectivity, frequency, homogeneity, suppleness, firmness, translucency, luminosity, pigmentation, degree of mattness, elasticity or density.

The simulation is generated by unsupervised learning algorithms, notably generative adversarial networks (GANs).

A cosmetic product is selected and the future appearance of the area of interest to which the product was applied is simulated.

The invention can be better understood from reading the following detailed description of a non-limiting exemplary implementation thereof, and from examining the schematic and partial appended drawing, in which:

Brief description of the drawings

is a block diagram of a prediction method in accordance with the present invention.

As shown in , the method according to the invention comprises a step 31 of evaluating the various features of the cutaneous signs on the basis of an input image.

According to the invention, a holistic analysis with multi-area overall mapping making overall evaluation as well as targeted evaluation possible is performed.

For example, the weight of the area of the eyes with respect to the overall face in terms of apparent age is calculated.

More than 20 features of the face can be automatically captured: more signs analysed means a more specific and more accurate evaluation for each cutaneous sign.

Each portion of the face will be analysed:

Top with lines on the forehead,

Eyes with drooping eyelid, crow's feet, bags, periorbital lines, shadows under the eyes,

Nose/cheeks with texture, pores, pigmentary spots, nasolabial fold,

Mouth with corners of the lips,

Lower portion of the face with ptosis.

In a step 11, these features evaluated on the basis of the input image are compared with reference features prestored in a database.

In a step 41, the responses of the user to a questionnaire relating to their life habits can be stored and be processed in a database.

These responses can, for example, belong to categories such as physical features of the user, their living place, the demography of their place of residence, nutritional information, information on the use of cosmetics, environmental information, information on the use of beauty products, way of life, dietary habits, the geographical location of the place of residence, the location of the workplace, work habits, sleep habits, exercise habits, relaxation habits, beauty care habits, smoking and alcohol consumption habits, sun exposure habits, the use of sunscreen, the propensity to tan, the number of sunburns and of serious sunburns, dietary requirements, dietary supplements or vitamins, travel habits, information on physical condition, undesirable reactions to products, the use of previous beauty care products and their efficacy, leisure, civil status, country and region of birth, whether the subject is a city, suburb or country dweller, the size of the urban area in which the subject lives, whether the subject is retired.

In a step 42, the life context of the user can be stored and processed additionally in the same database.

In a step 43, the database is fed with scientific data such as publications, reviews, articles, encyclopaedias, atlases, patents, data on the exposome and ageing, testimony.

In a step 32, the information gathered at the end of the

steps

11, 41, 42 and 43 is processed in order to estimate the rate of acceleration of ageing from each cutaneous sign.

An "accelerator" is a parameter which will accelerate ageing, such as the sun, pollution or certain medicines.

The accelerator modifies the predicted information on the ageing of the external body part of the user with respect to predicted information generated without any accelerator.

Generally, a user not having an associated accelerator would be a user living a very healthy or ideal life.

The "rate of acceleration" is a numerical value which quantifies the influence of an accelerator on ageing.

The rate of acceleration is high when the influence of the accelerator on the predicted information is high, such as the sun, and weak if it has only a weak effect.

In a step 33, the future development of the cutaneous signs is evaluated on the basis of the information originating from the

steps

31 and 32, namely the features of the various cutaneous signs and the rate of acceleration.

In a step 34, the appearance of the face of the user in their predictable actual exposome is simulated, notably according to the information which they communicated in the

steps

41 and 42.

In a step 35, the appearance of the face of the user in an imaginary ideal exposome can be simulated.

Of course, the invention is not limited to the exemplary embodiments which have just been described.

Claims

Cosmetic and non-therapeutic method for predicting the appearance of an area of interest of a user, the area of interest being divided into regions of interest comprising cutaneous signs, the method comprising (i) acquiring an input image of the area of interest, processing (31) the input image, on the one hand, with (ii) an image segmentation and analysis algorithm for determining optical properties of the area of interest and extracting features of the cutaneous signs and, on the other hand, with (iii) a deep learning algorithm based on artificial intelligence, for predicting and/or displaying the future development (33) of the cutaneous signs from the input image by comparing (11) the features of the cutaneous signs extracted from the input image with reference data prestored in a reference database (43), (iv) ideally or acceleratedly simulating (34, 35) the future appearance of the area of interest as a function of time.
Cosmetic and non-therapeutic method for predicting the appearance of an area of interest of a user, the area of interest being divided into regions of interest comprising cutaneous signs, the method comprising (i) acquiring an input image of the area of interest, processing (31) the input image, on the one hand, with (ii) an image segmentation and analysis algorithm for determining optical properties of the area of interest and extracting features of the cutaneous signs and, on the other hand, with (iii) a deep learning algorithm based on artificial intelligence, for predicting and/or displaying the rejuvenation of the cutaneous signs from the input image by comparing (11) the features of the cutaneous signs extracted from the input image with reference data prestored in a reference database (43), (iv) ideally or acceleratedly simulating (34, 35) the future appearance of the area of interest as a function of time.
Cosmetic and non-therapeutic method for predicting the appearance of an area of interest of a user, the area of interest being divided into regions of interest comprising cutaneous signs, the method comprising (i) acquiring an input image of the area of interest, processing (31) the input image, on the one hand, with (ii) an image segmentation and analysis algorithm for determining optical properties of the area of interest and extracting features of the cutaneous signs and, on the other hand, with (iii) a deep learning algorithm based on artificial intelligence, for predicting and/or displaying, in the event that the performance of a product is known, an effect produced on cutaneous signs from the input image by comparing (11) the features of the cutaneous signs extracted from the input image with reference data prestored in a reference database (43), (iv) ideally or acceleratedly simulating (34, 35) the future appearance of the area of interest as a function of time.
Method according to any one of the preceding claims, characterized in that the algorithm for predicting and/or displaying is a deep learning algorithm based on artificial intelligence.
Method according to any one of the preceding claims, characterized in that processing the input image includes an overall analysis of the area of interest as well as an analysis of regions of interest.
Method according to any one of the preceding claims, characterized in that processing the input image comprises evaluating at least two cutaneous signs.
Method according to any one of the preceding claims, characterized in that it comprises (v) processing one or more pieces of information describing the lifestyle of the user (41, 42).
Method according to any one of the preceding claims, characterized in that it comprises (vi) estimating (32) a rate of acceleration of ageing from each cutaneous sign as a function of the information describing the lifestyle of the user, of the input image and of the prestored reference data.
Method according to any one of the preceding claims, characterized in that it comprises (vii) evaluating (33) the appearance of each cutaneous sign as a function of time, based on the rate of acceleration of ageing and on the input image.
Method according to any one of the preceding claims, characterized in that it comprises (viii) simulating the future appearance of the area of interest in an exposome of the user (34) and possibly (viii) simulating the future appearance of the area of interest in an ideal exposome (35).
Method according to any one of the preceding claims, characterized in that the areas of interest are chosen from the forehead, the contour of the eyes, the nose, the cheekbones, the contour of the lips, the chin, the lips and the cheeks.
Method according to any one of the preceding claims, in which the algorithm for predicting and/or displaying the future development, the rejuvenation or the produced effect of the cutaneous signs from the input image is based on a deep learning algorithm based on artificial intelligence.
Method according to any one of the preceding claims, in which the algorithm for predicting and/or displaying the future development, the rejuvenation or the produced effect of the cutaneous signs from the input image is based on a photorealistic or realistic physical 3D/2D rendering algorithm.
Method according to any one of the preceding claims, in which the algorithm for predicting and/or displaying the future development, the rejuvenation or the produced effect of the cutaneous signs from the input image is based on a neural rendering algorithm.
Method according to any one of the preceding claims, in which the machine learning model is a pretrained convolutional neural network.
Method according to any one of the preceding claims, in which the cutaneous signs are chosen from pores, skin texture, a line, a fine line, crow's feet, sagging, shadows under the eyes, skin grain, a facial expression, morphology, redness, sheen.
Method according to any one of the preceding claims, in which the features of the cutaneous signs are chosen from openness, surface area, volume, depth, width, length, colour, degree of moisturization, degree of sheen, distribution, intensity, reflectivity, frequency, homogeneity, suppleness, firmness, translucency, luminosity, pigmentation, degree of mattness, elasticity or density.
Method according to any one of the preceding claims, in which the simulation is generated by unsupervised learning algorithms, notably generative adversarial networks (GANs).
Method according to any one of the preceding claims, in which a cosmetic product is selected and the future appearance of the area of interest to which the product was applied is simulated.
Cosmetic and non-therapeutic system for predicting the appearance of an area of interest of a user, comprising communication means which are suitable for communicating with a user computing device and configured to import, from an image library of the user computing device, at least one input image comprising a representation of the area of interest of the user; processing means configured to subject the input image to (ii) an image segmentation and analysis algorithm for determining optical properties of the area of interest and extracting features of the cutaneous signs, and to (iii) a deep learning algorithm based on artificial intelligence, for predicting and/or displaying the future development of the cutaneous signs from the input image by comparing the features of the cutaneous signs extracted from the reference image with reference data prestored in a reference database, (iv) means for ideally or acceleratedly simulating the future appearance of the area of interest as a function of time.
Computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to one of Claims 1 to 19.
Computer-readable medium comprising instructions which, when they are executed by a computer, cause the latter to implement the method according to one of Claims 1 to 19.

Method for predicting the development of cutaneous signs over time