WO2023217644A1

WO2023217644A1 - Method for diagnosing the skin by means of 3d facial analysis

Info

Publication number: WO2023217644A1
Application number: PCT/EP2023/061901
Authority: WO
Inventors: Panagiotis-alexandros BOKARIS; Véronika MAREK; Frédéric FLAMENT; Benjamin ASKENAZI
Original assignee: LOreal SA
Current assignee: LOreal SA
Priority date: 2022-05-10
Filing date: 2023-05-05
Publication date: 2023-11-16
Anticipated expiration: 2024-11-10
Also published as: CN119173905A; FR3135552A1; FR3135552B1

Abstract

The subject of the invention is a method for modelling an element of interest in an area of the skin of a user comprising (i) acquiring at least one RGB or greyscale input image of the area of interest, (ii) acquiring at least one high- or medium-resolution 2D input image of the area of interest, (iii) reconstructing the area of interest of the user in 3D form.

Description

Method for diagnosing the skin by means of 3D facial analysis

Technical field

The present invention relates to a cosmetic method without a therapeutic aim for predicting the ageing of an element of interest of the skin of a user. The invention further relates to a data processing system comprising a processor configured to execute the method.

Prior art

Individuals are always curious to learn what will happen in the future.

An area of particular interest relates to the curiosity of an individual as to their future appearance.

Given that a "crystal ball" or a "time machine" exists only in fairy tales and science fiction, it is impossible to obtain an exact prediction of the future appearance of an individual.

With the advance of modern research techniques and data processing devices, however, somewhat accurate evaluations of the prognosis for an individual are becoming more and more possible.

One field where the use of information relating to the prospects for an individual arouses particular interest is that of beauty.

In the process of ageing, a person may develop afflictions such as lines, fine lines, hair loss and skin discolourations, such as those caused by overexposure to the sun or more generally by their way of life.

The document US2017270593 discloses a method for providing a personalized product recommendation to a consumer, comprising: a) capturing an image of a user with an image capture device; b) analysing the image with a computing device coupled to the image capture device, in which the computing device analyses the image using a convolutional neural network to predict the age of the user, and in which predicting the age of the user comprises identifying at least one pixel which indicates the age of the user; c) using the at least one pixel to generate a two-dimensional heat map which identifies a region of the skin in the image which contributes to the predicted age of the user; d) recommending a product to the user to be applied to the region of the skin in order to reach a target cutaneous age; and e) offering the user the possibility of buying the product.

The article by Natalia Múnera "Evaluation of fringe projection and laser scanning for 3d reconstruction of dental pieces", Dyna, Medellin, Colombia, February 2012 discloses 3D scanning of human teeth based on fringe projection or laser scanning. Human teeth can be reconstructed in order to display the topography of the teeth.

A method which is widely applied for 3D reconstruction of objects is the moiré effect. This phenomenon is produced when two regular patterns are superposed on a surface. It began to be used in optical metrology in the 1920s. The moiré effect has been used in fields as diverse as the life sciences and engineering. In the life sciences, the moiré effect has provided valuable information in facial plastic surgery, in the study of malformations of the vertebral column, and in the morphometry of the face.

The article by Kawano, Y., "Three dimensional analysis of the face in respect of zygomatic fractures and evaluation of the surgery with the aid of Moiré topography", J. Cranio Maxill. Surg. 15, pp. 68-74, 1987 discloses an analysis of the facial deformation of patients having zygomatic fractures, using Moiré topography, which is a three-dimensional morphometry method. Facial moiré photographs of the patients were taken at angles of 45 degrees before and after an operation and were analysed quantitatively to determine the degree of the deformation of the cheek by means of computer processing of the data. Each datum was compared in the interval between the injury and the operation and in relation to the surgical procedures. The radiographic results and normal individuals were also studied for the purposes of comparison.

There is a need to improve methods for predicting the ageing of the skin.

There is a need to improve methods for diagnosing the skin, notably in terms of reliability, efficacy and accuracy.

There is a need to evaluate the performance of cosmetic products and other factors having an impact on ageing.

Definition of the invention

The subject of the invention is a method for modelling an element of interest of the skin of a user comprising (i) acquiring at least one RGB or greyscale input image of the area of interest, (ii) acquiring at least one high- or medium-resolution 2D input image of the area of interest, (iii) reconstructing the area of interest of the user in 3D form.

By virtue of the particular way in which the images are acquired, an improvement in the cutaneous diagnosis for all the elements of interest is observed. An opportunity to monitor the development of the elements of interest of the skin of a person with accuracy over time results therefrom.

According to the invention, the element of interest is, for example, a line, a fine line, a spot, a scar, pores, a texture, shadows under the eyes, bags under the eyes.

The area of interest is, for example, a cheek, a lip, a forehead, crow's feet.

Red, green and blue, abbreviated to RGB, is a system for computer coding of colours, the closest to hardware. Computer screens recreate a colour by means of additive synthesis on the basis of three primary colours, a red, a green and a blue, forming on the screen a mosaic which is too small to be perceived. RGB coding indicates a value for each of these primary colours.

Low-resolution images have around 72 pixels, or squares of colour, per inch.

High-resolution images have at least 300 pixels per inch (ppi or dpi).

Medium-resolution images have between 72 pixels per inch and 300 pixels per inch.

3D information is notably used to compensate for the effect of the viewing angle and of the orientation of the light, which modifies the appearance of a person in a 2D input image, like shadows or specularities which have an impact on the evaluation of clinical signs.

The algorithm which makes it possible to have the in-depth high-resolution reconstruction of a face in the 3D form is not explicitly described here as there are many different technologies which make such a scan possible.

In general, many data sources may be combined in order to make it possible to monitor raised details on the skin.

The method according to the invention is cosmetic and non-therapeutic.

Advantageously, the method for diagnosing an element of interest in an area of the skin of a user comprises (i) acquiring at least one RGB or greyscale input image of the area of interest, (ii) acquiring at least one high- or medium-resolution 2D input image of the area of interest, (iii) reconstructing the area of interest in 3D form, (iv) extracting data on the basis of the reconstruction in 3D form.

The modelling method may use 3D face measurements to accurately extract the 3D topology of the face via face reconstruction.

The invention also relates to a cosmetic and non-therapeutic method for monitoring an area of interest of a user over time comprising (i) acquiring at least one RGB or greyscale input image of the area of interest, (ii) acquiring at least one high- or medium-resolution 2D input image of the area of interest, (iii) reconstructing the area of interest in 3D form, (v) monitoring, over time, the reconstruction in 3D form generated by comparing reconstructions in 3D form which are generated at different instants in time.

The modelling method may use 3D face measurements to accurately extract the 3D topology of the face via face reconstruction. Recording the 3D facial reconstructions of the same person over the course of time makes it possible to superpose the parts which do not vary significantly, like bone structure, and to evaluate the difference in the parts which might move or alter over the course of time, like the elements of interest such as lines and fine lines or the firmness of the skin. In this way, the development of the skin of the person may be monitored over time and, furthermore, the performance of the various cosmetic products applied and all the other factors which have an impact on the ageing of the person, like an exposome or nutrition, may be evaluated. A recommendation of a cosmetic product may result therefrom.

The invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to execute the method as described above.

The invention also relates to a data processing system comprising:

A memory in which the computer program product as described above is stored,

A processing unit configured to implement the computer program product,

A photographic device configured to acquire the RGB or greyscale input images of the area of interest and the high- or medium-resolution 2D input image of the area of interest,

A screen configured to display the reconstruction of the area of interest of the user in 3D form generated by the processing unit.

The invention also relates to a computer-readable medium comprising instructions which, when they are executed by a computer, cause the computer to execute the method described above.

Preferred embodiments

Modelling method

Preferably, the modelling method according to the invention has one or more of the following features, taken alone or in combination:

The reconstruction in 3D form is a high-resolution reconstruction in 3D form.

The reconstruction in 3D form is carried out by combining global or local high-resolution 3D acquisition with high-resolution local/global depth analysis.

It comprises 3D scanning based on fringe projection or laser scanning.

It comprises triangulation based on images acquired from different viewpoints.

It comprises high-resolution estimation based on images of a lower resolution with a machine learning model. The method comprises acquiring at least one additional datum describing personal information of the user, notably on the basis of a questionnaire filled out by the user, and processing this additional datum.

Method for monitoring over time

Preferably, the method for monitoring an area of interest of the user over time has one or more of the following features, taken alone or in combination:

It comprises (vi) processing the reconstructions in 3D form which are generated at different instants in time by superposing a predetermined time-invariant part, notably bone structure, and detecting the development of the elements of interest between the different instants in time on the basis of the superposition.

Monitoring over time comprises calculating a gradient of the development of the area of interest on the basis of comparing parameters recorded at the two instants in time.

It comprises recommending a cosmetic product based on the reconstruction of the area of interest of the user in 3D form.

Computer program product

It comprises instructions which, when the program is executed by a computer, cause the latter to: Acquire at least one RGB or greyscale input image of an area of interest of a user, Acquire at least one high- or medium-resolution 2D input image of the area of interest, Reconstruct a 3D form of the area of interest of the user.

It comprises instructions which, when the program is executed by a computer, cause the latter to: Compare reconstructions in 3D form which are generated at different instants in time.

It comprises instructions which, when the program is executed by a computer, cause the latter to: Recommend a cosmetic product depending on the reconstruction of the area of interest of the user in 3D form, Present the product which is recommended to the user on a screen.

Description of the figures

Further features and advantages of the invention will become apparent from reading the following detailed description of non-limiting exemplary implementations thereof and from examining the appended drawing, in which:

schematically illustrates one embodiment and implementation of the invention.

In the example shown, the method comprises, on the one hand, a step of importing three RGB input photographic images 1, 2, 3 of the area of interest, the three images being taken at various camera angles, and, on the other hand, a step of importing a high- or low-resolution local scan 4.

For example, importing the photographic image may result from a capture performed by the user, in self-portrait mode or possibly by photographing their reflection in a mirror.

Next, the method comprises a step of segmentation in the scan 4, so as to identify at least one area of interest 5 in the face of the user.

The segmentation step may be implemented automatically by an image segmentation algorithm which is, for example, suitable for recognizing objects, defining contours, recognizing patterns, removing background, or performing other suitable and known image processing operations.

The segmentation step is, for example, capable of detecting an element of interest, such as a line.

Of course, the elements of interest identified in the scan depend on the exposures.

As an alternative, the segmentation step may be performed “manually” by the user, for example by means of a request addressed to the user to identify the position of said at least one area of interest in the image.

A machine learning model may advantageously be provided to correct a deformation of the colour, for example by taking into account the geographic position of the exposure (inside or outside), the timestamp of the exposure (day or night), the weather at the time of the exposure (sun, clouds, rain, snow). The parameters making it possible to estimate the correction to be performed may be more complex than the basic elements presented above.

The machine learning model may, for example, be of the convolutional neural network type. Using a machine learning model makes it possible to benefit from extremely fine accuracy in determining the colour of the areas of interest.

After importing the images, the method comprises processing the images in order to perform a reconstruction 6 of the face of the user in 3D form, on the basis of a processing unit, like a processor or a microprocessor, for example.

The processing unit may be configured to monitor, over time, the reconstructions in 3D form 6 and 7 acquired at two different instants in time, on the basis of images of the user taken at different instants in time.

In particular, the images may be acquired regularly, for example every year. Each new 3D modelling 8 may be recorded in order to facilitate monitoring.

The data extracted from each 3D modelling 6, 7, 8 may be associated with a time indication relating to a time of acquisition of the images. For example, the time indication may be a date of acquisition of the image.

Claims

Method for modelling an area of interest of a user comprising (i) acquiring at least one RGB or greyscale input image of the area of interest, (ii) acquiring at least one high- or medium-resolution 2D input image of the area of interest, (iii) reconstructing the area of interest of the user in 3D form.
Method according to the preceding claim, the reconstruction in 3D form being a high-resolution reconstruction in 3D form.
Method according to one of the preceding claims, the reconstruction in 3D form being carried out by combining global or local high-resolution 3D acquisition with high-resolution local/global depth analysis.
Method according to any one of the preceding claims, comprising 3D scanning based on fringe projection or laser scanning.
Method according to any one of the preceding claims, comprising triangulation based on images acquired from different viewpoints.
Method according to any one of the preceding claims, comprising high-resolution estimation based on images of a lower resolution with a machine learning model.
Method according to any one of the preceding claims, comprising acquiring at least one additional datum describing personal information of the user, notably on the basis of a questionnaire filled out by the user, and processing this additional datum.
Cosmetic and non-therapeutic method for monitoring an area of interest of a user over time comprising (i) acquiring at least one RGB or greyscale input image of the area of interest, (ii) acquiring at least one high- or medium-resolution 2D input image of the area of interest, (iii) reconstructing the area of interest in 3D form, (v) monitoring, over time, the reconstruction in 3D form generated by comparing reconstructions in 3D form which are generated at different instants in time.
Method according to the preceding claim, comprising (vi) processing the reconstructions in 3D form which are generated at different instants in time by superposing a predetermined time-invariant part, notably a bone structure, and detecting the development of the elements of interest between the different instants in time on the basis of the superposition.
Method according to either one of Claims 8 and 9, in which monitoring over time comprises calculating a gradient of the development of the area of interest on the basis of comparing parameters recorded at the two instants in time.
Method according to any one of the preceding claims, comprising recommending a cosmetic product based on the reconstruction of the area of interest of the user in 3D form.
Computer-readable medium comprising instructions which, when they are executed by a computer, cause the computer to execute the method according to any one of Claims 1 to 11.
Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to:
Acquire at least one RGB or greyscale input image of an area of interest of a user, Acquire at least one high- or medium-resolution 2D input image of the area of interest,
Reconstruct a 3D form of the area of interest of the user.
Computer program product according to Claim 13 comprising instructions which, when the program is executed by a computer, cause the latter to:
Compare reconstructions in 3D form which are generated at different instants in time.
Computer program product according to Claim 13 or 14 comprising instructions which, when the program is executed by a computer, cause the latter to:
Recommend a cosmetic product depending on the reconstruction of the area of interest of the user in 3D form,
Present the product which is recommended to the user on a screen.
Data processing system comprising:
- a memory in which the computer program product according to any one of Claims 13 to 15 is stored,
- a processing unit configured to implement the computer program product,
- a photographic device configured to acquire the RGB or greyscale input images of the area of interest and the high- or medium-resolution 2D input image of the area of interest,
- a screen configured to display the reconstruction of the area of interest of the user in 3D form generated by the processing unit, and possibly to present a product which is recommended to the user.