CN115997236A - Video analysis for obtaining optical properties of faces - Google Patents

Abstract

Systems and methods for obtaining optical properties of skin on a person's face through facial video analysis are disclosed. Capturing a video of the face, tracking landmarks on the face, defining and tracking regions of interest using the landmarks, obtaining some measured values/optical properties, transforming the time-based video into the angular domain, and obtaining additional measured values/optical properties. Such optical properties may be measured using real-time video or pre-recorded video.

Description

Video analysis for obtaining optical properties of faces

Cross Reference to Related Applications

The present application claims priority from U.S. patent application Ser. No.17/007,860, filed 8/31/2020, the contents of which are incorporated herein by reference in their entirety for all purposes.

The present application claims priority from french patent application No. fr2011194 filed on month 11 and 2 of 2020, the contents of which are incorporated herein by reference in their entirety for all purposes.

Technical Field

Embodiments described herein relate generally to systems and methods for analyzing optical properties of skin.

Background

Digital photographs have become the most popular medium for capturing and assessing the color, luster, and morphological effects of cosmetics. However, even the highest resolution image is insufficient: they are static, even if one sees the world in motion, they capture only one specific lighting condition, and image processing requires very strict, standardized geometries to ensure reliable comparative analysis.

To address this problem, scientists have developed machine learning methods to improve photo analysis, but they have limitations: they are "black boxes," require very large image sets, require very standardized data, train fastest with low resolution images, and require very specific training targets.

Disclosure of Invention

In view of the above-mentioned problems, the present disclosure describes systems and methods for analyzing optical properties of skin, comprising: capturing a video of the skin; tracking one or more landmarks on the skin from the captured video of the skin; identifying one or more regions of interest from the one or more landmarks on the tracked skin; transforming the captured video into an angle domain; and calculating the optical properties of the skin.

In one embodiment, the optical property comprises at least one of color, brightness, texture, luster, radiance, uniformity, skin tone, iridescence, and glow.

In one embodiment, video is captured from a handheld device.

In one embodiment, the environment is changed based on the calculated optical properties of the skin.

In one embodiment, the skin is on the face.

In one embodiment, the optical properties are calculated for pre-existing video.

In one embodiment, the calculated optical properties of the skin are collected for data analysis.

Drawings

Fig. 1A is a system view of a potential scene in which a camera is used to capture video of a user's face, and the optical parameters of the face in the captured video are calculated locally, over a network, or a combination of both.

Fig. 1B is a system view of a second potential scenario in which a phone is used to capture video from and process the video with the cloud to obtain optical parameters.

FIG. 2 is a flow chart summarizing the steps of calculating a measured value/optical property in one embodiment.

Fig. 3 illustrates frames of video from captured participants moving their heads.

Fig. 4 is an example showing landmarks tracked for each frame in one example.

Fig. 5A is a first example showing regions of interest on the left and right sides of a face with one head oriented, where the regions of interest are created using tracked landmarks.

Fig. 5B is a second example showing regions of interest on the left and right sides of the face with three different head orientations, where the regions of interest are created using tracked landmarks.

Fig. 6 is an example showing the optical properties of skin on the face for one frame of video, where the optical properties are average colors per region of interest.

Fig. 7 is an example showing optical properties of skin on a face for a plurality of frames of video, where the optical properties are average gray values per region of interest.

Fig. 8 illustrates an angular transformation showing the average RBG values for the left and right forehead regions of interest at different head orientations/angles.

Fig. 9A shows a graph of peak brightness versus time point obtained for all participants in the study using a Panasonic GH5 camera, where each participant used a first product on one half of their face known to have a higher peak brightness (NARS) and a second product on the other half of their face known to have a lower peak brightness (ELDW).

Fig. 9B shows a peak brightness product difference plot obtained for all participants in the study using a Panasonic GH5 camera, where each participant used a first product on one half of their face known to have a higher peak brightness (NARS) and a second product on the other half of their face known to have a lower peak brightness (ELDW).

Fig. 10A shows a graph of peak brightness versus time obtained for all participants in the study using an iPhone 8 camera, where each participant used a first product on one half of their face known to have a higher peak brightness (NARS) and a second product on the other half of their face known to have a lower peak brightness (ELDW).

Fig. 10B shows a peak brightness product difference plot obtained for all participants in the study using an iPhone 8 camera, where each participant used a first product known to have a higher peak brightness (NARS) on one half of their face and a second product known to have a lower peak brightness (ELDW) on the other half of their face.

Fig. 11A shows a graph of gloss level versus time point obtained for all participants in the study using Samba, where each participant used a first product known to have a higher peak brightness (NARS) on one half of their face and a second product known to have a lower peak brightness (ELDW) on the other half of their face.

Fig. 11B shows a graph of gloss level product difference obtained for all participants in the study using Samba, where each participant used a first product known to have a higher peak brightness (NARS) on one half of their face and a second product known to have a lower peak brightness (ELDW) on the other half of their face.

Fig. 12 shows a side-by-side comparison of gloss/peak brightness differences for three different acquisition types, where (a) is from a Samba polarized camera, (B) is a facial video system with a self-acquired iPhone video, and (C) is a facial video system with a Panasonic GH5 camera on a tripod.

Fig. 13 shows the statistics of a paired T-test that uses results from three different acquisition types to test whether ELDW and NARS were found to be statistically different.

Fig. 14 shows the actual products placed on the left and right side faces of each participant randomly distributed in the experiment.

Detailed Description

In one embodiment, the present disclosure presents systems and methods for assessing optical properties (i.e., effects) such as, but not limited to, color, brightness, texture, luster, radiance, uniformity, skin tone, iridescence, glow, etc., of skin on a person's face. One approach is to (1) evaluate and (2) visualize the optical effects by capturing video in a relaxed real life capture environment and analyze it with software having facial landmark detection capabilities. The disclosed technology can use the video and dynamic motion of a person to quantify the optical properties of a person's face; the result is better sensitivity, robustness and flexibility (without the need to use a high-end camera) when these optical properties are quantized.

Fig. 1A and 1B illustrate system views of two exemplary embodiments. In fig. 1A, a camera 1002, such as a Digital Single Lens Reflex (DSLR) camera, may be used to record video of a face 1001. In one embodiment, the distance between the face 1001 and the camera 1002 may be the normal distance between the person and the mirror when the person is applying the cosmetic. Captured video data may be sent to processor 1003, network controller 1004, memory 1005, and/or display 1006. The processor 1003 may be used to process the captured video from the camera 1002 and perform calculations to determine optical properties. The network controller 1004 may be used to send and/or receive data over a network. The memory 1005 may be used to read and write data. The display 1006 may be used to visualize data such as the processed (i.e., calculated) optical properties of the face 1001. In another embodiment, light may impinge on face 1001 to obtain better illumination. In another embodiment, a green screen may be placed behind the user's face 1001 to scratch out (key-out) the background to better visualize the face 1001.

In fig. 1B, a telephone 1007 connected to the internet can be used to self-capture video of the face 1001. The self-captured video may be sent to the cloud 1008 (i.e., over a network). The self-captured video may be sent from cloud 1008 to server 1009 for processing. The server 1009 may calculate the optical properties of the face 1001 from the self-captured video and send the results back to the phone 1007 via the cloud 1008. In another embodiment, the cloud 1008 and server 1009 may be omitted and the phone 1007 may calculate all optical properties of the face 1001 locally (i.e., without an internet connection) from the self-captured video and display the results to the user.

FIG. 2 is a flowchart of an exemplary embodiment of traversing a method 100. The method may be configured as a processing circuit.

In fig. 2, S102 will capture video. The captured video may be a human face. In another embodiment, the captured video may be skin elsewhere, such as the legs, arms, back, etc. In this example skin on a person's face will be used. The video may be pre-recorded or recorded in real time. In recorded video, a person may naturally move their head (e.g., look left, center, right, and then repeat). A person may remove the glasses and pull the hair back with a headband to optimize facial tracking and minimize occlusion of facial areas. Fig. 3 shows an example of some frames from a captured video. The video may be recorded by a device capable of recording video, such as a handheld device (e.g., a smart phone, a tablet, etc.), a webcam from a laptop, a DSLR camera, etc.

In one embodiment, the video capture system may have one LED light, DSLR on a tripod in line with the central axis of illumination, and a chair on which the subject sits. Video can be recorded at 720p resolution (up to 4K) and 60fps (up to 120 fps). In one embodiment, during recording, the subject may be instructed to naturally move their head for up to 60 seconds and a minimum of 15 seconds. In another embodiment, the subject may move their head for less than 15 seconds or more than 60 seconds. In another embodiment, the DSLR and tripod may be replaced by a smartphone (e.g., iPhone), and the smartphone user may self-capture video of their face. For in-store customer experience visualization, real-time capture may be performed using a computer Vision (e.g., USB3Vision type) camera connected to a workstation.

In fig. 2, S104 is landmark tracking. Video files (evaluation systems) or video streams (real-time visualizations) are read by a facial landmark tracking software library. Fig. 4 shows an example of landmark tracking, where 70 points on the face (e.g., center of pupil, corner of eye, nostril, etc.) are detected for each video frame. In one embodiment, the number of landmarks may be greater than or less than 70. The landmark data may include time of presence, intensity of presence, x-position and y-position. The data may be output into a Comma Separated Value (CSV) file.

Landmark tracking software can be found in the open source Dlib (https:// gitsub. Com/davisking/Dlib). Examples of such landmark tracking can be seen in the OpenPose software developed by Carnegie Mellon (https:// gitsub. Com/CMU-periodic-Computing-Lab/openpost).

In fig. 2, S106 will define and track a region of interest (ROI). The tracked landmarks may be used to construct one or more regions of interest (i.e., patches or regions) on the face. Further, the captured video may be filtered to use only the best video frames (e.g., frames with clear ROIs, frames with a minimum number of landmarks, etc.). These ROIs/patches/regions may be defined as the locations most relevant to human perception. Their size and location can be known from specialized perception studies, such as pupil tracking studies, for example, to learn the exact point of interest of a person on the face of a subject. An example of S106 is shown in fig. 5A and 5B, where landmarks are used to create ROIs for multiple regions, such as left and right forehead, eye bags, cheeks, and chin.

In fig. 2, S108 is used to calculate a measured value/optical property. For all (or some) ROIs, calculations for optical properties such as color, brightness (i.e. light) and texture may be performed. The average, median, upper and lower quartiles are calculated from the set of RGB triples contained within the ROI to estimate the color (e.g., at frame 12, the average color of forehead-ROI #2 is rgb= [180, 122, 80 ]). After converting the RGB triplet into a single 8-bit gray value (ranging from 0 to 255), a similar function can be used to calculate luminance and brightness. The texture within each ROI may be calculated using block processing functions (or filters) such as 2D standard deviation, 2D entropy, and first and second derivatives. Fig. 6 shows an example in which the average color per ROI for each ROI is calculated for one frame of captured video. In addition, fig. 7 shows calculated color and brightness over time for each ROI of each frame of captured video.

In fig. 2, S110 will perform a transformation to the angle domain. The time-based color/light/texture data may be converted into the angle domain. This is achieved by calculating the ratio of the distance between the left and right flank landmarks with respect to the central axis and solving the trigonometric equation; for example, when the head turns to the right, it can be observed that the right eye-to-nose distance is much smaller than the left eye-to-nose distance, and therefore the angle can be estimated based on these distances. This allows the following results to be produced: when the head is rotated 22 degrees from the center to the left, the color of the right cheek is rgb= = [200, 112,98]. In other words, the calculated measurements may be remapped as a function of head orientation (i.e., angle of 22 °) rather than time (i.e., frame number 12). An example in which the left and right forehead ROIs are mapped according to color/brightness (y-axis) and head orientation (x-axis) is shown in fig. 8. Note that in this example, a more negative x-axis value corresponds to the participant's face turning more left, a more positive x-axis value corresponds to the face turning more right, and an x-axis value at zero represents the participant's face centered.

In fig. 2, S112 will calculate additional higher levels of measured value/optical properties such as gloss, radiance, uniformity, skin tone, iridescence, glow. After the color and light measurements have been mapped to the angle domain, a higher level of analysis may be performed on the data, e.g., facial "gloss" may be associated with "peak brightness within the angle range" in the video. In other words, as a person turns their head left and right, each ROI experiences a maximum brightness at an angle that maximizes the reflection of the light source, and the "peak brightness" gray value can distinguish between a matt product and a glossy product. For example, iridescence is an optical effect in which color appears to change due to angle, so variability of average color within each ROI can be tracked and quantified to estimate color change. Other metrics may be determined by similar nonlinear functions (i.e., functions of maximum brightness and angle) acting on the angular and spatial domains (i.e., maximum brightness over all ROIs); in the case of radiance and glow, a regression model may be used to relate a set of angle-based metrics to consumer perception data and cosmetic expert evaluation data.

In another embodiment, S110 may be performed before S108, and S108 and S112 may be performed together as one step after S110.

The above method 100 was tested with peak brightness/gloss metrics using a quasi-professional grade camera (Panasonic GH 5), a handheld device camera (iPhone 8 front camera), and a differential polarized camera (Samba of Bossa Nova). Samba may be considered the most advanced. For example, samba is a standard living gloss instrument at L' Oreal (in vivo shine instrument).

Six white female participants were each coated with two products on each half of their face. These products are (1) Estee Lauder Double Wear,3w1 Tawny (known as matte), abbreviated ELDW, and (2) NARS Natural Radiant Longwear Foundation, light 4.5vienna 6606 (known as brighter than ELDW), abbreviated NARS. Thus, in peak brightness/gloss measurements, NARS Foundation should have a higher peak brightness/gloss value than ELDW. The faces of the participants were captured at baseline (T0), post-application (T1), 2 hours post-application (T2), and 5 hours post-application (T3). Facial analysis software incorporating the method 100 described above is used to translate landmarks per frame into regions per frame, into metrics per frame.

The calculated peak brightness results across all zones for each participant, each time point using video acquired by Panasonic GH5 on a tripod are shown in fig. 9A, and the corresponding peak brightness product differences are shown in fig. 9B. The calculated peak brightness results across all regions for each participant, each time point, using video acquired by iPhone (manually acquired, i.e., acquired by the participant by himself) are shown in fig. 10A (each curve represents one side of the face), and the corresponding peak brightness product differences are shown in fig. 10B (each curve represents one participant). For video captured by Panasonic GH5 and iPhone, the results show that NARS has a higher peak brightness than ELDW for each participant. There is also a consistent evolution in peak brightness difference, where NARS is initially much brighter than ELDW at T1, and then evolves closer to ELDW at T2 and T3. This consistency is not found in Samba results.

The gloss results for each participant, each time point using Samba are shown in fig. 11A, and the corresponding gloss level differences for each participant, each time point are shown in fig. 11B. Note that at T1 and T2 (both circled in fig. 11B), respectively, the Samba gloss levels for participants #3 and #5 are higher for ELDW than NARS. Overall, samba gloss levels were not as regular as the peak brightness data shown in fig. 9A, 9B, 10A, and 10B.

The product difference graphs are summarized in fig. 12 for side-by-side comparison, where (a) is from Samba, (B) is a facial video system with self-captured iPhone video, and (C) is a facial video system with Panasonic GH5 camera on tripod. The gloss evolution seen from these two facial video systems is much clearer than Samba. The results of the facial video system show that the coated NARS side of the face is more consistently brighter (i.e., negative product difference) than Samba results, whether using an iPhone with self-capture capability or a quasi-professional-grade camera on a tripod. These gloss difference curves can be translated into superior discrimination for any statistical test.

As shown in fig. 13, paired T-test (n=6) was used to compare data from each instrument of NARS and ELDW. Statistics show that facial video systems using peak brightness from self-acquired iPhone video can distinguish between both NARS and ELDW products at each time point and with very low p-values. This is essentially the same as the statistical intensity of facial video systems using robustly supported and more advanced Panasonic GH5 cameras. However, samba fails to distinguish between NARS and ELDW at T1 and T3.

Thus, a facial video system with a GH5 camera shows that while cheaper, smaller, non-contact, more consumer-related and easier to use, it can distinguish between gloss performance with statistical confidence that the most advanced technology has never seen. In addition, the same high level of discrimination is also produced when using normal smartphone videos acquired by the test subjects themselves.

During the experiment, apple iPhone 8 runs Pro Camera by Moment and has the following camera settings: 720p@60fps, ISO22 (lowest), 1/60s shutter (60 fps slowest). The Panasonic GH5 was placed on a tripod approximately 50 cm from the subject and white balance corrected to XRite Graycard (reverse of XRite Video Colorchecker), ISO200, all-I Frame Compression mbps. The subject has a pneumatic stool that can be rotated, raised and lowered. There is a green screen backdrop approximately 100 cm behind the subject's head, illuminated with a Compact Fluorescent Lamp (CFL) of approximately 6000K. There was a Bescor LED lamp (about 6000K) and the light measured in front of the face with a Sekonic color note (Light and Color Meter) was 6200K. During the experiment, the participant IDs and randomly assigned products applied to the left and right sides of their faces are shown in fig. 14.

In one exemplary embodiment, the techniques described above may be applied to change the environment based on the color of a facial feature of a person (e.g., a customer). For example, a customer may enter a store to find lipstick cosmetics, and the shelves may change color to match the customer's skin tone. As another example, a customer may enter a flagship and the walls may change color to match their face. Such benefits may include the ability to easily pick make-up products, lighten skin colors, and generate marketing hot flashes, such as, for example, how advanced or unique a display store is. As another example, customers may enter a flagship store, and the robotic sculpture may change its shape to reflect their color makeup; this allows customers to view their faces from multiple angles as if they would be seen by others.

In one embodiment, the above-described techniques may be performed at home (e.g., using a smart phone) or at a vendor (e.g., using a DSLR camera); this may be done in real time (e.g., optical properties are calculated at about the same time as the video is captured) or for pre-existing video (i.e., optical properties are calculated for video recorded in the past).

In another embodiment, the techniques described above may be used to create a database that includes data collected from different users regarding calculated optical properties of the skin. The database may be used for data analysis. The data may be further organized by other characteristics such as age, region, gender, etc. For example, for trend analysis, collecting data about the optical properties of social media red people (e.g., from their social media posts) may lead to insight into trends, such as popularity of a particular cosmetic style for a certain age group, a particular lipstick color for a certain region of the world, and so forth. The results of the data analysis can be used to create business ideas and make better business decisions, such as being able to identify trends.

The methods and systems described herein may be implemented in a variety of technologies, but are typically directed to processing circuitry. In one embodiment, the processing circuitry is implemented as one or a combination of the following: an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a generic logic array (GAL), a programmable logic array (PAL), a circuit for allowing one-time programmability of logic gates (e.g., using fuses), or reprogrammable logic gates. Furthermore, the processing circuitry may comprise a computer processor and have embedded and/or external non-volatile computer readable memory (e.g., RAM, SRAM, FRAM, PROM, EPROM and/or EEPROM) that stores computer instructions (binary executable instructions and/or interpreted computer instructions) for controlling the computer processor to perform the processes described herein. The computer processor circuit may implement a single processor or multiple processors, each supporting a single thread or multiple threads, and each having a single core or multiple cores. The processing circuitry used to train the artificial neural network need not be the same as the processing circuitry used to implement the trained artificial neural network to perform image denoising described herein. For example, processor circuitry and memory may be used to generate (e.g., as defined by the interconnections and weights of) a trained artificial neural network, and an FPGA may be used to implement the trained artificial neural network. Further, training and use of the trained artificial neural network may use a serial implementation or a parallel implementation to improve performance (e.g., by implementing the trained neural network on a parallel processor architecture such as a graphics processor architecture).

While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the invention. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes in the form of the embodiments described herein may be made without departing from the spirit of the invention. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the invention.

Claims (16)

1. A system for analyzing optical properties of skin, comprising:

processing circuitry configured to:

capturing a video of the skin;

tracking one or more landmarks on the skin from the captured video of the skin;

identifying one or more regions of interest from the tracked one or more landmarks on the skin;

transforming the captured video into an angle domain; and

the optical properties of the skin were calculated.

2. The system of claim 1, wherein the optical property comprises at least one of color, brightness, texture, luster, radiance, uniformity, skin tone, iridescence, and glow.

3. The system of claim 1, wherein the video is captured by a handheld device.

4. The system of claim 1, further comprising:

changing an environment based on the calculated optical properties of the skin.

5. The system of claim 1, wherein the skin is on the face.

6. The system of claim 1, wherein the optical property is calculated for a pre-existing video.

7. The system of claim 1, further comprising:

the calculated optical properties of the skin are collected for data analysis.

8. The system of claim 5, wherein transforming the captured video into an angular domain is achieved by comparing a distance between a first landmark on a left side of the face and a distance between a second landmark on a right side of the face relative to a central axis.

9. A method for analyzing optical properties of skin, comprising:

capturing a video of the skin;

tracking one or more landmarks on the skin from the captured video of the skin;

identifying one or more regions of interest from the tracked one or more landmarks on the skin;

transforming the captured video into an angle domain; and

the optical properties of the skin were calculated.

10. The method of claim 9, wherein the optical property comprises at least one of color, brightness, texture, luster, radiance, uniformity, skin tone, iridescence, and glow.

11. The method of claim 9, wherein the video is captured by a handheld device.

12. The method of claim 9, further comprising:

changing an environment based on the calculated optical properties of the skin.

13. The method of claim 9, wherein the skin is on the face.

14. The method of claim 9, wherein the optical property is calculated for a pre-existing video.

15. The method of claim 9, further comprising:

the calculated optical properties of the skin are collected for data analysis.

16. The method of claim 13, wherein transforming the captured video into an angular domain is achieved by comparing a distance between a first landmark on a left side of the face and a distance between a second landmark on a right side of the face relative to a central axis.

Images