FR3142589A1 - Instrumentation of makeup applicators to improve and share application techniques - Google Patents

Abstract

Instrumentation of makeup applicators to improve and share application techniques. In some embodiments, a system for generating makeup application presentations is provided. The system includes a camera, at least one motion sensor associated with a first applicator, and a computing device including at least one processor and a non-transitory computer-readable medium, with computer-executable instructions stored thereon. The instructions, in response to execution by the at least one processor, cause the computing device to perform actions including: receiving a first video of a makeup application process from the camera, the first video representing at least part of a face and at least part of an applicator; receiving signals from the at least one motion sensor indicating movement of the applicator; measuring the movement of the applicator relative to one or more facial landmarks to determine a relative trajectory of the applicator and storing the relative trajectory of the applicator. Figure for the abstract: none

Description

Instrumentation of makeup applicators to improve and share application techniques Summary

The purpose of this summary is to present a selection of concepts in a simplified form which are described more fully below in the detailed description. This summary is not intended to identify key features of the claimed subject matter or to be used as an aid in determining the scope of the claimed subject matter.

In some embodiments, a method of applying makeup with an instrument is provided. A computing device receives a first video of a makeup application process, the first video depicting at least a portion of a first face and at least a portion of a first applicator. The computing device measures the movement of the first applicator relative to one or more marks of the first face to determine a relative trajectory of the applicator. The computing device records the relative trajectory of the applicator.

In some embodiments, a system for generating makeup application presentations is provided. The system includes a camera, at least one motion sensor associated with a first applicator, and a computing device including at least one processor and a non-transitory computer-readable medium, with computer-executable instructions stored thereon. The instructions, in response to execution by the at least one processor, cause the computing device to perform actions including: receiving a first video of a makeup application process from the camera, the first video representing at least part of a face and at least part of an applicator; receiving signals from the at least one motion sensor indicating movement of the applicator; measuring the movement of the applicator relative to one or more facial landmarks to determine a relative trajectory of the applicator and storing the relative trajectory of the applicator.

In some embodiments, a computing device is provided. The computing device includes circuitry for receiving a first video of a makeup application process, the first video illustrating at least a portion of a first face and at least a portion of a first applicator; circuits for measuring the movement of the first applicator relative to one or more marks of the first face to determine a relative trajectory of the applicator and circuits for storing the relative trajectory of the applicator.

• la réception, par un dispositif informatique, d’une première vidéo d’un procédé d’application de maquillage, la première vidéo représentant au moins une portion d’un premier visage et au moins une portion d’un premier applicateur ;
• la mesure, par le dispositif informatique, du mouvement du premier applicateur par rapport à un ou plusieurs repères du premier visage pour déterminer une trajectoire relative de l'applicateur et
• le stockage, par le dispositif informatique, de la trajectoire relative de l’applicateur.

The subject of the invention is a process for applying instrumented makeup, the process comprising:

• receiving, by a computing device, a first video of a makeup application process, the first video representing at least a portion of a first face and at least a portion of a first applicator;
• measuring, by the computing device, the movement of the first applicator relative to one or more marks of the first face to determine a relative trajectory of the applicator and
• the storage, by the computing device, of the relative trajectory of the applicator.

• la réception, par le dispositif informatique, d’une deuxième vidéo, la deuxième vidéo étant une vidéo en direct qui représente au moins une portion d’un second visage et au moins une portion d’un second applicateur ;
• l’alignement, par le dispositif informatique, de la trajectoire relative de l’applicateur sur un ou plusieurs repères du deuxième visage et
• la génération, par le dispositif informatique, d’une présentation de la trajectoire relative de l’applicateur à fournir en association avec la capture de la deuxième vidéo.
• fournir la présentation de la trajectoire relative de l’applicateur sous forme d’images superposées sur la deuxième vidéo.
• la présentation de la trajectoire relative de l’applicateur en :

According to other advantageous aspects of the invention, the method comprises one or more of the following characteristics, taken individually or in all technically possible combinations:

• receiving, by the computing device, a second video, the second video being a live video that represents at least a portion of a second face and at least a portion of a second applicator;
• the alignment, by the computing device, of the relative trajectory of the applicator on one or more marks of the second face and
• generating, by the computing device, a presentation of the relative trajectory of the applicator to be provided in association with capturing the second video.
• provide the presentation of the relative trajectory of the applicator in the form of images superimposed on the second video.
• the presentation of the relative trajectory of the applicator in:

comparing a detected location of the second applicator with the relative trajectory of the applicator and

• la génération de la présentation de la trajectoire relative de l’applicateur à fournir en association avec la capture de la deuxième vidéo comprenant la présentation de la trajectoire relative de l’applicateur sur un affichage du dispositif informatique avec la deuxième vidéo pendant la capture de la deuxième vidéo.

providing haptic feedback to determine that the detected location of the second applicator is outside the relative trajectory of the applicator.

• generating the presentation of the relative trajectory of the applicator to be provided in association with capturing the second video including presenting the relative trajectory of the applicator on a display of the computing device with the second video while capturing the second video.

• des circuits de réception d’une première vidéo d’un procédé d’application de maquillage, dans lequel la première vidéo représente au moins une portion d’un premier visage et au moins une portion d’un premier applicateur ;
• des circuits de mesure du mouvement du premier applicateur par rapport à un ou plusieurs repères du premier visage pour déterminer une trajectoire relative de l’applicateur et
• des circuits pour le stockage de la trajectoire relative de l’applicateur.

The invention also relates to a computing device, comprising:

• circuits for receiving a first video of a makeup application method, in which the first video represents at least one portion of a first face and at least one portion of a first applicator;
• circuits for measuring the movement of the first applicator relative to one or more marks of the first face to determine a relative trajectory of the applicator and
• circuits for storing the relative trajectory of the applicator.

• des circuits de réception d’une deuxième vidéo, la deuxième vidéo étant une vidéo en direct qui représente au moins une portion d’une deuxième visage et au moins une portion d’un deuxième applicateur ;
• des circuits permettant d’aligner la trajectoire relative de l’applicateur sur un ou plusieurs repères du deuxième visage et
• des circuits de génération d’une présentation de la trajectoire relative de l’applicateur à fournir en association avec la capture de la deuxième vidéo.
• des circuits pour fournir la présentation de la trajectoire relative de l’applicateur sous forme d’images superposées sur la deuxième vidéo.
• des circuits pour présenter la trajectoire relative de l’applicateur en :

According to other advantageous aspects of the invention, the computing device comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations:

• circuits for receiving a second video, the second video being a live video which represents at least one portion of a second face and at least one portion of a second applicator;
• circuits making it possible to align the relative trajectory of the applicator on one or more marks of the second face and
• circuitry for generating a presentation of the relative trajectory of the applicator to be provided in association with the capture of the second video.
• circuitry to provide presentation of the relative trajectory of the applicator in the form of images superimposed on the second video.
• circuits to present the relative trajectory of the applicator by:

comparing a detected location of the second applicator with the relative trajectory of the applicator and

• la génération de la présentation de la trajectoire relative de l’applicateur à fournir en association avec la capture de la deuxième vidéo comprenant la présentation de la trajectoire relative de l’applicateur sur un affichage du dispositif informatique avec la deuxième vidéo pendant la capture de la deuxième vidéo.

providing haptic feedback to determine that the detected location of the second applicator is outside the relative trajectory of the applicator.

• generating the presentation of the relative trajectory of the applicator to be provided in association with capturing the second video including presenting the relative trajectory of the applicator on a display of the computing device with the second video while capturing the second video.

The foregoing aspects and many related advantages of this invention will be more readily appreciated as they become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, in which:

There represents a block diagram that illustrates aspects of a non-limiting exemplary embodiment of a mobile computing device according to various aspects of the present disclosure.

There represents a diagram which illustrates a non-limiting example of an embodiment of a system which allows relative trajectories of the applicator to be shared between subjects according to various aspects of the present disclosure.

There is an illustration depicting a non-limiting example embodiment of a captured video of a subject's face according to various aspects of the present disclosure.

[Fig 4A-4B] Figures 4A and 4B are a flowchart which illustrates an example of a non-limiting embodiment of a method of instrumenting makeup applications and generating presentations according to various aspects of the present disclosure.

There illustrates a non-limiting example of an embodiment of an image of a first video of a makeup application method according to various aspects of the present disclosure.

There illustrates a non-limiting example of an embodiment of an image of a second video with superimposed images according to various aspects of the present disclosure.

detailed description

It is becoming more and more common for makeup application techniques to be shared between people. However, makeup tutorials are usually a video of a person applying makeup on themselves, and a viewer must figure out how to tailor the makeup application to their own facial structure. Additionally, no live guidance is provided as the viewer attempts to recreate makeup application. What is desired are systems that provide presentations that help improve makeup application and make it suitable for demonstration application despite differences in facial structure between the demonstrator and applicator.

In embodiments of the present disclosure, a first subject uses an instrumentation computing device to record a makeup application. From this recording, a relative applicator trajectory is determined that specifies the locations of a makeup applicator during application. The relative trajectory of the applicator is defined with respect to the distances from different facial landmarks, so that the relative trajectory of the applicator can be presented relative to other faces by calculating the distances from different facial landmarks of others faces. Presentations including, among other things, phantom outlines of applicators, audio feedback, haptic feedback or visual prompts may be generated to assist a second subject in guiding an applicator along the relative trajectory of the applicator and thus improving the application of makeup by the second subject.

There represents a block diagram that illustrates aspects of a non-limiting exemplary embodiment of a mobile computing device according to various aspects of the present disclosure. Although the illustrated instrumentation computing device 110 may typically be a mobile computing device such as a smartphone or tablet, in some embodiments the instrumentation computing device 110 may be implemented by any computing device or set computing devices, including, but not limited to, a desktop computing device, a portable computing device, a mobile computing device, a server computing device, a computing device of a cloud computing device and/or combinations thereof this. As discussed in more detail below, the instrumentation computing device 110 is configured to monitor the application of makeup to determine the trajectories of the applicators during application and present information based on the determined trajectories in order to apply later the makeup.

As illustrated, the instrumentation computing device 110 includes one or more processors 102, one or more communication interfaces 104, a trajectory database 108; a camera 11116 and a computer readable medium 106.

As used herein, the expression "data bank" includes, in particular, any suitable device configured to store data for access by a computing device. An example of a database is a highly reliable and fast relational database management system (DBMS) that runs on one or more computing devices and is accessible over a high-speed network. Another example of a data store is a key value store. However, any other suitable storage technique and/or device capable of quickly and reliably providing the stored data in response to queries may be used, and the computing device may be accessible locally instead. to be connected to a network, or can be provided as a cloud service. A database may also include data stored in an organized manner on a computer-readable storage medium, such as a hard drive, flash memory, RAM, ROM, or any other type of computer-readable storage medium. computer. One skilled in the art will recognize that separate data banks described herein may be combined into a single data bank, and/or that a single data bank described herein may be separated into several data banks, without departing from the scope application of this disclosure.

As used here, the expression "computer readable medium" includes, in particular, a removable or non-removable device which implements any technology capable of storing information in a volatile or non-volatile manner so that it can be read by a processor of a computing device, including, in particular: a hard disk; flash memory; a solid-state static disk; random access memory (RAM); a read only memory (ROM); a CD-ROM, DVD or other disk storage; a magnetic cassette; a magnetic tape and a magnetic disk memory.

In some embodiments, the processors 102 include any suitable type of general purpose computer processor. In some embodiments, the processors 102 include one or more special purpose computer processors or AI accelerators optimized for specific computing tasks, including, but not limited to, graphics processing units (GPUs), vision processing units (VPT) and Tensor Processing Units (TPU).

In certain embodiments, the communication interfaces 104 include one or more hardware and/or software interfaces adapted to provide communication links between constituents. The communications interfaces 104 may support one or more cable communications technologies (including, but not limited to, Ethernet, FireWire, and USB), one or more wireless communications technologies (including, but not limited to, Wi-Fi, WiMAX, Bluetooth , 2G, 3G, 4G, 5G and LTE), and/or combinations of these technologies.

In some embodiments, camera 116 includes any type of digital camera or combination of two or more digital cameras configured to capture at least a portion of a subject's face. In some embodiments, camera 116 captures images using visible light, images using infrared light, depth information, and/or any other type of image information. Some non-limiting example embodiments of devices that can be used as a camera 116, alone or in combination with other devices, include a front-facing camera of a mobile computing device, a stereoscopic camera, and a time-of-flight camera. In some embodiments, camera 116 includes one or more optical sensors, image sensors, optical transducers, or the like.

As shown in the illustration, the computer readable medium 106 stores logic which, in response to execution by one or more processors 102, causes the mobile computing device 110 to provide a face detection engine 112, a determination engine trajectory 114 and a trajectory presentation engine 118.

As used herein, the term "engine" includes, in particular, logic embodied in hardware or software instructions, which may be written in one or more programming languages, including, but not limited to, C, C++, C#, COBOL, JAVA™, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Go and Python. An engine can be compiled into executable programs or written in interpreted programming languages. Software engines can be called from other engines or from themselves. In general, the engines described herein refer to logic modules that may be merged with other engines or may be divided into sub-engines. The engines may be implemented by logic stored in any type of computer-readable computer storage media or device and be stored and executed by one or more general-purpose computers, thereby creating a special-purpose computer configured to provide the engine or its functionality. The motors may be implemented by programmed logic in application-specific integrated circuits (ASICs), a field-programmable gate array (FPGA), or other hardware device.

In some embodiments, facial detection engine 112 is configured to use information collected by camera 116 to construct a model of a subject's face, and to detect facial landmarks in the model. In some embodiments, the trajectory determination engine 114 is configured to determine a trajectory traced by an applicator during application of makeup to the face based on one or more information collected by the camera 116 and data from motion sensor associated with the applicator and received via a communication interface 104. In certain embodiments, the trajectory determination engine 114 determines the trajectory with respect to the facial landmarks detected by the face detection engine 112, and stores the relative trajectory of the applicator in the trajectory database 108. In some embodiments, the trajectory presentation engine 118 is configured to generate a presentation of a relative trajectory of the applicator from the bank of trajectory data 108 to help improve subsequent makeup application.

In some embodiments, the trajectory determination engine 114 includes computational circuitry configured to generate a per-pixel prediction score for a presence or absence of an applicator in an image using one or more image classifiers. convolutional neural network images. In some embodiments, the trajectory determination engine 114 includes computational circuitry configured to predict a trajectory traversed by the applicator and to generate a virtual display of the predicted trajectory traversed by the applicator on at least a portion of the user's face based on per-pixel prediction scores for the presence or absence of the applicator in the image.

A more detailed description of the configuration of each of these constituents is provided below.

While generating a presentation of a relative applicator trajectory to a subject who has recorded the relative applicator trajectory may be useful in supporting consistent and repeatable application of makeup by the subject, a benefit provided by embodiments of the present disclosure that use relative applicator trajectories defined with respect to facial landmarks is the ability to share relative applicator trajectories between subjects. There represents a diagram which illustrates a non-limiting example of an embodiment of a system which allows relative trajectories of the applicator to be shared between subjects according to various aspects of the present disclosure. As illustrated, the system 200 includes a first instrumentation computing device 202 used by a first subject 208, a second instrumentation computing device 204 used by a second subject 210, and a third instrumentation computing device 206 used by a third subject 212. Although three computer instrumentation devices and subjects are illustrated on the , in some embodiments, a greater or lesser number of instrumentation computing devices and subjects may be present in the system 200.

The first subject 208 may use the first instrumentation computing device 202 to determine and record a relative trajectory of the applicator, which the first instrumentation computing device 202 then transmits to a trajectory management computing system 214. The computing system trajectory management system 214 then transmits the relative trajectory of the applicator to the second computer instrumentation device 204 and to the third computer instrumentation device 206.

Since the relative trajectory of the applicator is recorded by the first subject 208 but is defined relative to the facial landmarks, the same facial landmarks can be determined for other subjects, and the relative trajectory of the applicator can be presented to other subjects as if he had initially recorded it. In other words, the second instrumentation computing device 204 may present the relative trajectory of the applicator to match the facial characteristics of the second subject 210 and the third instrumentation computing device 206 may present the relative trajectory of the applicator to match the facial characteristics of the third subject 212.

In some embodiments, the trajectory management computer system 214 provides a variety of functionality that is made possible by the ability to exchange relative applicator trajectories between subjects. For example, in some embodiments, the trajectory management computer system 214 may provide an interface that allows a second subject 210 to search for tutorials of particular makeup applications (e.g., cat eye eyeliner, eyeliner sooty, etc.). The trajectory management computing system 214 may provide one or more relative applicator trajectories relevant to the second instrumentation computing device 204 to create the appearance generated by, optionally, the first subject 208 using the first instrumentation computing device 202, and the second instrumentation computing device 204 presents the relative trajectories of the applicator to the second subject 210. Such an embodiment may be useful to influencers and others in developing an audience and improving the ability to guide topics through makeup tutorials.

As another example, in some embodiments, the trajectory management computer system 214 may provide a social connection interface that allows subjects to connect with each other to provide preference information relating to appearances, colors and/or appearances. or products; share relative trajectories of the applicator; review relative applicator trajectories published by others and so on. The social connection interface may then use a recommendation engine to determine one or more relative recommended applicator trajectories that a subject may desire based on their preference information and reviews that have been published, and may provide for presentation of the relative trajectories of the recommended applicator to the instrumentation computing device associated with the subject.

There is an illustration depicting a non-limiting example embodiment of a captured video of a subject's face according to various aspects of the present disclosure. In some embodiments, the video may be presented on a display of an instrumentation computing device (such as a mobile computing device) while the video is captured, although the instrumentation computing device itself does not is not illustrated on the (or the rest of the illustrations in this disclosure) for the sake of clarity.

There illustrates some non-limiting examples of facial landmarks that can be detected by the facial detection engine 112. As illustrated, the video includes a wing 302, or “wing” of the nose; a “nostril” 304, an internal palpebral angle of the eye 306; an external palpebral angle of the eye 308; a nasal point 310, a median junction between the eyes and under the eyebrows. In some embodiments, other facial landmarks, including, among others, features of lips, ears, bone structure, and/or other facial landmarks, may be detected. In some embodiments, the surface contour of the face may itself be used as a facial cue. By finding these facial landmarks in a model of the face in the video, it is possible to determine and record a relative trajectory of the applicator (relative to the facial landmarks).

Figure 4A - and Figure 4B are a flowchart which illustrates an example of a non-limiting embodiment of a method for instrumenting makeup applications and generating presentations according to various aspects of the present disclosure. In method 400, a relative trajectory of the applicator is determined when recording a makeup application. The relative trajectory of the applicator can then be used to generate presentations during a second makeup application to enhance the second makeup application, as described above.

From a starting block, method 400 proceeds to optional block 402, where a first motion sensor is associated with a first applicator. In some embodiments, the first motion sensor includes a three-, six-, or nine-axis accelerometer that generates signals representing an orientation, position, and/or movement of the motion sensor (and, by association, any object to which it is directed). motion sensor is attached). In some embodiments, other types of motion sensors, including, among others, a photo sensor, a camera, a magnetic sensor, a tomographic sensor, an infrared sensor, an ultrasonic sensor, a force sensor , and/or combinations thereof.

The motion sensor may be associated with the first applicator in any suitable manner. For example, the motion sensor may be integrated into a housing of the first applicator or permanently connected to the first applicator. Another example: the motion sensor can be removably coupled to the first applicator, for example using a clip, a strap, a ratchet, a pocket or any other means of coupling removable. Another example: the motion sensor may be attached to a glove, a watch or other portable device worn by the subject on the hand holding the first applicator.

At option block 404, the first motion sensor is coupled to a first instrumentation computing device 110. In some embodiments, the first motion sensor communicates via Bluetooth, Wi-Fi or other wireless communication technology, and coupling the first motion sensor with the first instrumentation computing device 110 causes the first motion sensor to transmit information to the first instrumentation computing device 110. In some embodiments, coupling the first motion sensor with the first instrumentation computing device 110 may include using the camera 116 of the first instrumentation computing device 110 to scan an identifier of the first motion sensor. Blocks 402 and 404 are described as optional because, in some embodiments, the method 400 can operate using the video information without the motion sensor information.

In block 406, a camera 116 associated with the first instrumentation computing device 110 captures a first video of a makeup application process, the first video representing at least part of a face of a first subject and at least part of the first applicator. In some embodiments, the first video includes a two-dimensional representation of a scene that includes the facial portion of the first subject and the portion of the first applicator. In some embodiments, the first video includes a three-dimensional representation of the scene. In some embodiments, the first video includes depth information to accompany the two-dimensional representation of the scene.

There illustrates a non-limiting example of an embodiment of an image of a first video of a makeup application method according to various aspects of the present disclosure. The image shown on the is a non-limiting example of an image from a first video captured in block 406. In the image, part of the face of the first subject is visible and the first subject applies makeup using the first applicator 502. In the illustrated embodiment, the first applicator 502 is an eyeliner pencil, and the applied makeup 504 is an eyeliner, but these examples should not be considered limiting. In other examples, the first applicator 502 may be a brush, sponge, tweezers, powder puff, makeup printer, or any other type of tool that can be used to apply makeup. In other examples, the makeup 504 applied may be lipstick, blush, mascara, foundation, concealer, or any other type of makeup.

Returning to Figure 4A, process 400 proceeds to optional block 408, where a trajectory determination engine 114 of the first instrumentation computing device 110 receives signals from the first motion sensor while capturing the first video. The signals indicate the orientation, position and/or movement of the first motion sensor, and therefore of the first applicator, during the video. In some embodiments, the trajectory determination engine 114 correlates these signals with the representation of the first applicator in the first video. The actions of block 408 are illustrated and described as optional because, in some embodiments, the method 400 can operate using the video data without the motion sensor signals.

At block 410, a facial detection engine 112 of the first instrumentation computing device 110 constructs a model of the face of the first subject based on the first video. In some embodiments, the facial model may be a two-dimensional model based on two-dimensional information present in the first video. In some embodiments, the facial model may be a three-dimensional model constructed from a point cloud derived from depth information or other three-dimensional information in the first video. In some embodiments, the facial model may be a three-dimensional model constructed from two-dimensional information present in the first video using, for example, machine learning image processing techniques (e.g., LOLNeRF developed by Google, Inc.). In some embodiments, construction of the model may include predictive portions of the model that are obscured in one or more frames of the first video, such as portions of the face that are obscured by the first applicator or by a hand of the subject. In some embodiments, an initial model of the face may be constructed based on a portion of the first video captured with an unobstructed view of the face (e.g., before makeup application begins).

In block 412, the facial detection engine 112 detects one or more facial landmarks in the face model of the first subject. In some embodiments, the facial landmark(s) are detected on the model itself. In some embodiments, the facial landmark(s) are detected in two-dimensional images of the first video and then correlated to locations in the facial model. Any suitable technique for detecting facial landmarks can be used, including, among others, image processing routines provided by the publicly available open source dlib library.

In block 414, a trajectory determination engine 114 of the first instrumentation computing device 110 constructs a model of locations of the first applicator based on the first video and, optionally, on the motion sensor signals. In some embodiments, the trajectory determination engine 114 uses similar techniques used in block 410 to construct the face model to construct a similar model of the first applicator in the same coordinate space as the face model, and as with block 410 techniques, can generate a two-dimensional model or a three-dimensional model.

Use of the motion sensor is optional because, in some embodiments, the information from the first video may be adequate to construct the model of the first applicator locations. In other embodiments, the motion sensor signals may be used to enhance the detail of the location model, including providing detailed rotation and attitude information that may not be readily visible at first. video. The motion sensor signals may be particularly useful if a large portion of the first applicator is obscured in the first video (for example, if the subject grips the first applicator near a tip of the first applicator, such that most of the first applicator is hidden, etc.) or if the first applicator is smooth, reflective or designed in such a way that detailed features of the first applicator other than the shape are not easily detectable in the first video.

In block 416, the trajectory determination engine 114 measures the locations of the first applicator relative to the one or more facial landmarks to determine a relative trajectory of the applicator. Once the models of the face and the first applicator are generated, the relative distances between the first applicator and the facial landmarks can be easily measured on each frame of the first video. The relative distances for each image can then be combined in series to create the relative trajectory of the applicator. The relative trajectory of the applicator may also include the attitude (e.g., angle of incidence relative to a normal surface, twist over time, etc.) of the first applicator relative to the facial landmarks. In some embodiments, the first applicator may also include a force sensor that transmits signals to the instrumentation computing device 110, such that an amount of force with which the first applicator is applied may be included in the relative trajectory of the applicator. In some embodiments, an amount of force with which the first applicator is applied can be determined by measuring an amount of deflection of at least a portion of the first applicator is visible in the first video.

In block 418, the trajectory determination engine 114 stores the relative trajectory of the applicator in a trajectory data bank 108 of the first computer instrumentation device 110. The method 400 then passes to a tracking terminal ("terminal A ").

From terminal A (Figure 4B), the method 400 passes to block 420, where the trajectory determination engine 114 transmits the relative trajectory of the applicator to a trajectory management computer system 214, and to block 422, the trajectory management computer system 214 transmits the relative trajectory of the applicator to a second instrumentation computer device 110 for storage in a trajectory data bank 108 of the second instrumentation computer device 110. As indicated above, the trajectory management computer system 214 may distribute the relative trajectory of the applicator to the second instrumentation computer device 110 for various reasons. For example, a second subject associated with the second instrumentation computing device 110 may navigate the relative trajectories of the applicator or transmit a request for the relative trajectories of the applicator, and the trajectory management computing system 214 may transmit the trajectory relative of the applicator to the second computer instrumentation device 110 in response to the request. As another example, the trajectory management computer system 214 may push the relative trajectory of the applicator toward the second instrumentation computing device 110 without receiving a request if the trajectory management computer system 214 determines that the second subject associated with the second computer instrumentation device 110 may be interested in the relative trajectory of the applicator.

At the optional block 424, a second motion sensor is associated with a second applicator, and at the optional block 426, the second motion sensor is coupled to the second computer instrumentation device 110. The association of the second motion sensor with the second applicator and the coupling with the second instrumentation computing device 110 is similar to that mentioned above with regard to the first motion sensor, the first applicator and the first instrumentation computing device 110, and is therefore not discussed in more detail here for the sake of brevity.

At block 428, a camera 116 associated with the second instrumentation computing device 110 captures a second video, wherein the second video is a live video that represents at least a portion of a second face of a second subject and at least a portion of the second applicator. As used herein, "live video" is video captured and processed in real time, and which continues to be captured during subsequent actions of method 400. A non-limiting example of capturing live video is the capture of a selfie video by a front camera of a mobile computing device, where the selfie video is presented on a screen of the mobile computing device while the video is captured, such that the display of the mobile computing device appears to a mirror. Another non-limiting example of capturing a live video is the capture of a video by another type of camera which is broadcast to the second instrumentation computing device 110 during its capture.

At block 430, a facial detection engine 112 of the second instrumentation computing device 110 constructs a model of the second face of the second subject based on the second video, and at block 432, the facial detection engine 112 detects one or more facial landmarks in the model of the second face of the second subject. The techniques for constructing the model of the second face and detecting facial landmarks are similar to those mentioned above in relation to the first face, and are therefore not described again here for the sake of brevity.

In block 434, a trajectory presentation engine 118 of the second instrumentation computing device 110 generates a presentation of the relative trajectory of the applicator, based on the facial landmark(s) in the model of the second face, while capturing the second video. As part of generating the relative applicator trajectory presentation, the trajectory presentation engine 118 may align the relative applicator trajectory to the live video by finding the facial landmarks in the live video and determining the location of the presentation based on distances from facial landmarks indicated by the relative trajectory of the applicator.

In various embodiments, different types of presentations may be made, each of which may contribute to improving the application of makeup by the second subject. As a non-limiting example, the presentation may use the relative trajectory of the applicator and the facial landmarks of the second face to determine where the second applicator would be located if the same makeup pattern were applied to the second face instead of the first. , and can then generate images to overlay on the live video to guide the second applicator along the relative applicator trajectory. In some embodiments, presentations may not be made if the second applicator is within a margin of error of the relative trajectory of the applicator.

There illustrates a non-limiting example of an embodiment of an image of a second video with superimposed images according to various aspects of the present disclosure. On the , a portion of the face of a second subject, different from the first subject used to initially determine the relative trajectory of the applicator, is shown. The second subject's face will be shaped differently (e.g., different eye shapes and locations, different nose shapes and locations, etc.), but the relative applicator trajectory represents the applicator locations relative to the landmarks. faces and not an absolute location, and the trajectory presentation engine will therefore be able to compensate for differences in face shapes.

In the image, the facial part and the second applicator 602 are visible. Two different types of presentations are illustrated. A first type of presentation is a phantom presentation 604. The phantom presentation 604, shows a translucent shape, outline or other form of presentation intended to represent the ideal location of the second applicator 602 to correspond to the location of the first applicator in the relative trajectory of the applicator. In each frame, the phantom presentation 604 may be moved to coincide with the continuation of the relative trajectory of the applicator such that the phantom presentation 604 appears to be animated along the relative trajectory of the applicator, and may also be moved based on the movement of facial landmarks to remain in a fixed relative position relative to the face.

A second type of presentation illustrated on the is an indicator presentation 606. In the indicator presentation 606, instead of fully representing the desired location of the second applicator 602, a plurality of indicators are superimposed on the second applicator 602 to request a desired movement of the second applicator 602 (in this case, a row of chevrons indicating the desired direction of movement of the second applicator 602).

Although both the phantom presentation 604 and the indicator presentation 606 are shown on the , in some embodiments, only one type of presentation can be performed. In some embodiments, other types of presentations may be made. For example, if the second applicator 602 is determined to be outside a predetermined range around the relative trajectory of the applicator, an auditory or haptic indication may be provided to prompt the second applicator 602 to return to the relative trajectory of the applicator. the applicator. Likewise, in embodiments in which the relative trajectory of the applicator includes an amount of pressure applied by the applicator, the presentation may include a visual, auditory, or haptic indication when it is determined that an amount of pressure applied with the second applicator 602 is different from the amount of pressure indicated by the relative trajectory of the applicator.

Returning to Figure 4B, after generating the presentation, process 400 then proceeds to an end block and terminates.

The description of the method 400 presents embodiments in which the relative trajectory of the applicator is determined during an application by a first subject of makeup on his or her own face, and the relative trajectory of the applicator is then used during an application of makeup by a second subject on his or her own face for the purpose of clearly depicting various aspects of the disclosed subject matter. However, in other embodiments, different situations may arise. For example, in some embodiments, the first subject may record the relative trajectory of the applicator, and then may also replay the relative trajectory of the applicator to ensure consistency across multiple makeup applications. As another example, in some embodiments, the first subject may record the relative trajectory of the applicator on a first side of the face, provide a command to reflect the relative trajectory of the applicator, and then use the relative trajectory of the applicator on the second side of the face to provide symmetrical application.

As another example, the subject and the person applying the makeup may be different people. In such an embodiment, the person applying the makeup may use a head-mounted display or other augmented reality device to overlay presentations on the subject while applying the makeup. Such embodiments may have additional use cases in which a first person performs tasks relating to interacting with different faces. For example, a first dental hygienist may record a relative trajectory of the applicator to follow a trajectory of a dental tool in a first mouth, and a second dental hygienist may view presentations related to the relative trajectory of the applicator when using the dental tool in a second mouth. In such an embodiment, landmarks in the mouth may be used (e.g., specific tooth locations) instead of the facial landmarks discussed above.

Although illustrative embodiments have been illustrated and described, it will be appreciated that various changes may be made therein without departing from the spirit and scope of the invention.

Claims (10)

Process for applying instrumented makeup, the process comprising:
receiving, by a computing device, a first video of a makeup application process, the first video representing at least a portion of a first face and at least a portion of a first applicator;
measuring, by the computing device, the movement of the first applicator relative to one or more marks of the first face to determine a relative trajectory of the applicator and
the storage, by the computing device, of the relative trajectory of the applicator. A method according to claim 1, further comprising:
receiving, by the computing device, a second video, the second video being a live video that represents at least a portion of a second face and at least a portion of a second applicator;
the alignment, by the computing device, of the relative trajectory of the applicator on one or more marks of the second face and
generating, by the computing device, a presentation of the relative trajectory of the applicator to be provided in association with capturing the second video. A method according to claim 2, further comprising:
Provide the presentation of the relative trajectory of the applicator in the form of images superimposed on the second video. A method according to claim 2, further comprising:
the presentation of the relative trajectory of the applicator in:
comparing a detected location of the second applicator with the relative trajectory of the applicator and
providing haptic feedback to determine that the detected location of the second applicator is outside the relative trajectory of the applicator. The method of claim 2, generating the presentation of the relative trajectory of the applicator to be provided in association with capturing the second video comprising presenting the relative trajectory of the applicator on a display of the computing device with the second video while capturing the second video. Computer device comprising:
circuits for receiving a first video of a makeup application method, in which the first video represents at least one portion of a first face and at least one portion of a first applicator;
circuits for measuring the movement of the first applicator relative to one or more marks of the first face to determine a relative trajectory of the applicator and
circuits for storing the relative trajectory of the applicator. Computer device according to claim 6, further comprising:
circuits for receiving a second video, the second video being a live video which represents at least one portion of a second face and at least one portion of a second applicator;
circuits making it possible to align the relative trajectory of the applicator on one or more marks of the second face and
circuitry for generating a presentation of the relative trajectory of the applicator to be provided in association with the capture of the second video. Computer device according to claim 7, further comprising:
circuitry to provide presentation of the relative trajectory of the applicator in the form of images superimposed on the second video. Computer device according to claim 7, further comprising:
circuits to present the relative trajectory of the applicator by:
comparing a detected location of the second applicator with the relative trajectory of the applicator and
providing haptic feedback to determine that the detected location of the second applicator is outside the relative trajectory of the applicator. A computing device according to claim 7, generating the presentation of the relative trajectory of the applicator to be provided in association with capturing the second video comprising presenting the relative trajectory of the applicator on a display of the computing device with the second video while capturing the second video.