EP4260172A1 - Maquilleur numérique - Google Patents

Maquilleur numérique

Info

Publication number
EP4260172A1
EP4260172A1 EP21843850.5A EP21843850A EP4260172A1 EP 4260172 A1 EP4260172 A1 EP 4260172A1 EP 21843850 A EP21843850 A EP 21843850A EP 4260172 A1 EP4260172 A1 EP 4260172A1
Authority
EP
European Patent Office
Prior art keywords
user
makeup
cosmetic
digital
skin
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP21843850.5A
Other languages
German (de)
English (en)
Inventor
Mindy Christine Troutman
Francesca D. Cruz
Sandrine Gadol
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
LOreal SA
Original Assignee
LOreal SA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/138,143 external-priority patent/US11461946B2/en
Priority claimed from US17/138,078 external-priority patent/US11657553B2/en
Priority claimed from FR2107906A external-priority patent/FR3125613A1/fr
Priority claimed from FR2107909A external-priority patent/FR3125610A1/fr
Application filed by LOreal SA filed Critical LOreal SA
Publication of EP4260172A1 publication Critical patent/EP4260172A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
    • A—HUMAN NECESSITIES
    • A45—HAND OR TRAVELLING ARTICLES
    • A45D—HAIRDRESSING OR SHAVING EQUIPMENT; EQUIPMENT FOR COSMETICS OR COSMETIC TREATMENTS, e.g. FOR MANICURING OR PEDICURING
    • A45D44/00—Other cosmetic or toiletry articles, e.g. for hairdressers' rooms
    • A45D44/005—Other cosmetic or toiletry articles, e.g. for hairdressers' rooms for selecting or displaying personal cosmetic colours or hairstyle
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0282—Rating or review of business operators or products
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0631—Item recommendations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0641—Shopping interfaces
    • G06Q30/0643—Graphical representation of items or shoppers
    • A—HUMAN NECESSITIES
    • A45—HAND OR TRAVELLING ARTICLES
    • A45D—HAIRDRESSING OR SHAVING EQUIPMENT; EQUIPMENT FOR COSMETICS OR COSMETIC TREATMENTS, e.g. FOR MANICURING OR PEDICURING
    • A45D44/00—Other cosmetic or toiletry articles, e.g. for hairdressers' rooms
    • A45D2044/007—Devices for determining the condition of hair or skin or for selecting the appropriate cosmetic or hair treatment

Definitions

  • the present disclosure is directed to a digital make-up artist and a method for interactive makeup advice and makeup tutorial.
  • Hie Apps may provide tools for searching for particular types of make-up, or searching for a product that may be a user’s favorite or just purchasing a previously used product. Some Apps provide assistance in
  • Some Apps provide color matching features to assist in searching for a color that matches clothing, an accessory, or a color from a picture. Also, videos are available on how to apply certain types of makeup.
  • Some of these applications are implemented as Web applications, or an App. Some of these applications involve use of the camera for taking a self portrait photo with the smartphone camera, uploading the photo to the Web application, then applying virtual makeup products to the uploaded image. These applications may offer a variety of options, such as smoothing skin, lifting cheeckbones, adjusting eye color.
  • 5 applications may provide the user with the ability to add any type and color of makeup, as well as change the color intensity.
  • try-on applications offered thus far create a look by way of photo editing tools.
  • Some of the prior try-on applications start with an uploaded photograph and provide one step functions to overlay makeup types and colors, then allow editing of the made-up
  • VSCO photo editing tools for mobile devices with cameras
  • photo editing tools for mobile devices with cameras have become popular for use with social media.
  • One photo-sharing app, VSCO allows users to edit and add filters to photos before sharing them.
  • Many VSCO filters are available for obtaining certain effects.
  • a VSCO filter may consist of values that can be applied to a photo, including exposure, temperature, contrast, fade, saturation, tint, skintone.
  • prior try-on application tools do not provide for creation of custom looks. For example. a user may want a date night look. The prior try-on applications may offer a date night look, but do not provide advice on what makeup products to use that may be best for the user and how the various types of makeup products may be applied to create the look. Instead, some
  • 20 editing may be performed by the user on a face image in an attempt to obtain a custom look.
  • a user may want a date night look that is based on the user’s mood, or a mood that the user may want to portray.
  • 25 application process relies on templates and looks that were created for others. When a user w'ants a certain look that they have in mind, or wants to experiment with a new look, the user may be faced with having to edit a look created for someone else. There is a need to provide a custom try-on experience for a particular user that allow'S interaction in a manner that is comparable to the experience that the user may have with a personal makeup artist. There is a
  • An aspect is a digital makeup artist system that includes a mobile device having a display device, computation circuitry, and a memory; a database system storing cosmetic
  • the mobile device includes a user interface for interacting with a digital makeup artist, in which the digital makeup artist performs an interactive dialog with the user in order to capture needs of the user, including one or more of type of makeup look, indoor or outdoor
  • the computation circuitry is configured to input a face image of the user, analyze, via the machine learning system, the user’s face image to identify face parts, analyze the face image to determine facial characteristics including one or more of skin tone, eye color, hair color, lip color, and skin texture, and generate image frames to be displayed on the display device in synchronization
  • the image frames are generated based on the analyzed face image of the user, the needs of the user obtained through the interactions with the user, one or more of the stored cosmetic routine information, common makeup looks, cosmetic products for skin types and ethnicity, and the user look preferences.
  • An aspect is a digital makeup artist system that includes a mobile device having a
  • the mobile device includes a user interface for interacting with a digital makeup artist, in which the digital makeup artist performs an interactive dialog with the user in order
  • the computation circuitry is configured to input a face image of the user, analyze, via the machine learning system, the user’s face image to identify face parts, analyze the face image to determine facial characteristics including one or more of
  • the 20 skin tone, eye color, lip color, hair color, and skin texture, and generate image frames to be displayed on the display device in synchronization with the interaction with the digital makeup artist to provide the advice.
  • the image frames are generated in synchronization with the interaction based on the analyzed face image of the user, the initial information obtained through tire interactions with the user, one or more of the stored cosmetic routine information, common makeup looks, cosmetic products for skin types and ethnicity, and die user look preferences.
  • FIG. 1 is a diagram of a system in accordance with an exemplary' aspect of the disclosure
  • FIG. 2 is a block diagram of a computer system for a mobile device
  • FIG. 3 illustrates a user interface screen having an avatar in accordance with an exemplary aspect of the disclosure
  • FIG. 4 illustrates a user interface screen for a mobile device in accordance with an exemplary aspect of the disclosure
  • FIG. 5 is a sequence diagram for interaction between a digital makeup artist and a
  • FIG. 6 is a user interface for inputting a type of look in accordance with an exemplary aspect of the disclosure
  • FIG. 7 is a flowchart of face analysis in accordance with an exemplary aspect of the disclosure
  • FIG. 8 is a diagram of an exemplary architecture for a convolution neural network for classification of face shape
  • FIG. 9 is a diagram of an exemplary deep learning neural netw ork for face landmark detection
  • FIG. 10 is a diagram for a recommender system in accordance with an exemplary aspect of the disclosure.
  • FIG. 11 illustrates a non-limiting look-feature matrix in accordance with an exemplary aspect of the disclosure
  • FIG. 12 is a user interface in a mobile application in accordance with an exemplary
  • FIG. 13 is a user interface in a mobile application in accordance with an exemplary aspect of the disclosure.
  • FIG. 14 is an exemplary mobile application in accordance with an exemplary aspect of the disclosure.
  • FIG. 15 is a flowchart for a SLOW DOWN video control command in accordance with an exemplary aspect of the disclosure
  • FIG. 16 is a flowchart for a PAUSE video control command in accordance with an exemplary aspect of the disclosure.
  • FIG. 17 is a flowchart for a RE-DO LAST STEP video control command in
  • FIG. 18 is a flow chart for a SKIP CURRENT STEP video control command in accordance with an exemplary aspect of the disclosure
  • FIG. 19 is a block diagram of a video playback component in accordance with an exemplary aspect of the disclosure
  • FIG. 20 illustrates a blending process that may be used to create a video frame based on a desired feature and an original feature
  • FIGs. 21A to 21F are a sequence diagram for an exemplary interaction using the digital makeup artist for a makeup tutorial in accordance with an exemplary aspect of the
  • FIGs. 22A to 22E are a sequence diagram for an exemplary interaction use case of using the Digital Makeup Artist 520 for a makeup tutorial in accordance with an exemplary aspect of the disclosure
  • FIG. 23 is a sequence diagram for interaction between a digital makeup artist and a
  • FIGs. 24A to 24D are a sequence diagram for an exemplary interaction using the digital makeup artist for a makeup consultation in accordance with an exemplary aspect of the disclosure.
  • aspects of this disclosure are directed to a digital makeup artist that may be consulted for makeup advice and tutorials in a manner that is comparable to the experience that the user may have with a personal makeup artist.
  • the disclosed digital makeup artist provides a
  • FIG. 1 is a diagram of a system in accordance with an exemplary aspect of the disclosure.
  • Embodiments include a software application, or mobile application (App).
  • App mobile application
  • the term mobile application (App) will be used interchangeably with the software application, and makeup application will be used in
  • the software application may be executed on a desktop computer or laptop computer 103.
  • the mobile application may be executed on a tablet computer or other mobile device 101.
  • the software application and mobile application are described in terms of the mobile application 11 1.
  • the mobile application 111 may be downloaded and
  • the desktop computer or laptop computer 103 may be configured with a microphone 103a as an audio input device.
  • the microphone 103a may be a device that connects to a desktop computer or laptop computer 103 via a USB port or audio input port, or wireless via a Bluetooth wireless protocol.
  • the mobile device 101 may be equipped with a built-in microphone.
  • the software application or mobile application may include a communication function to operate in conjunction with a cloud service 105.
  • the cloud service 105 may include a database management service 107 and a machine learning service 109.
  • the database management service 107 may be any of the types of database management systems provided in the cloud service 105, for example, the database management service 107 may
  • the machine learning service 109 may perform machine learning in order to allow for scaling up and high performance computing that may be necessary for tire machine learning. Also, the software application or mobile application may be downloaded from a cloud service 105.
  • FIG. 1 shows a single cloud service, laptop computer and mobile device, it should be understood that any number of mobile devices, laptop computers, as well as desktop computers and tablet computers, may be connected to one or more cloud services.
  • FIG. 2 is a block diagram of a mobile computer device.
  • the functions and processes of the mobile device 101 may be implemented by one or more
  • a processing circuit includes a programmed processor as a processor includes circuitry.
  • a processing circuit may also include devices such as an application specific integrated circuit (ASIC) and conventional circuit components arranged to perform the recited functions. Note that circuitry
  • circuitry 10 refers to a circuit or system of circuits.
  • the circuitry may be in one computer system or may be distributed throughout a network of computer systems.
  • the processing/computation circuit 226 includes a Mobile Processing Unit (MPU) 200 which
  • the processing/computation circuit 226 may have a replaceable Subscriber Identity Module (SIM) 201 that contains information that is unique to the network service of the mobile device 101.
  • SIM Subscriber Identity Module
  • the claimed advancements are not limited by the form of the computer- readable media on which the instructions of the inventive process are stored.
  • the instructions may be stored in FLASH memory, Secure Digital Random Access Memory
  • SDRAM Secure Digital RAM
  • RAM Random Access Memory
  • ROM Read Only Memory
  • PROM Read-Only Memory-
  • EPROM Erasable Programmable Read-Only Memory
  • EEPROM Electrically Erasable Programmable Read Only Memory
  • solid-state hard disk or any other information processing device with which the processing/computation circuit 226 communicates, such as a server or computer.
  • the claimed advancements may be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing
  • the operating system may be for a laptop computer or a desktop computer such as Mac OS, Windows 10, or a Unix operating system.
  • a mobile operating system such as Android, Microsoft® Windows® 10 Mobile, Apple iOS® and other systems known to those skilled in the art may be used.
  • the hardware elements may be realized by various circuitry elements, known to those skilled in the art. For example,
  • MPU 200 may be a Qualcomm mobile processor, a Nvidia mobile processor, a Atom® processor from Intel Corporation of America, a Samsung mobile processor, or a Apple A7 mobile processor, or may be other processor types that would be recognized by one of
  • the MPU 200 may be implemented on an Field-
  • FPGA Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • MPU 200 may be implemented as multiple processors cooperatively working in parallel to perform the instructions of the inventive processes
  • Die processing/computation circuit 226 in FIG. 2 also includes a network controller
  • the network 224 can be a public network, such as the Internet, or a private network such as LAN or WAN network, or
  • the network 25 any combination thereof and can also include PSTN or ISDN sub-networks.
  • the network 224 can also be wired, such as an Ethernet network.
  • the processing circuit may include various types of communications processors for wireless communications including 3G, 4G and 5G wireless modems, WiFi®, Bluetooth®, GPS, or any other wireless form of communication that is known.
  • the processing/computation circuit 226 includes a Universal Serial Bus (USB) controller 225 which may be managed by the MPU 200.
  • USB Universal Serial Bus
  • the processing/computation circuit 226 further includes a display controller 208, such as a NVIDIA® GeForce® 1 GTX or Quadro® graphics adaptor from NVIDIA Corporation of
  • An I/O interface 212 interfaces with buttons 214,
  • the processing/computation circuit 226 may further include a microphone 241 and one or more cameras 231.
  • the microphone 241 may have associated circuitry' 240 for processing the sound into digital signals.
  • the camera 231 may include a camera controller 230 for controlling image capture operation of the camera 231. In an exemplary aspect, the camera
  • the processing/computation circuit 226 may include an audio circuit 242 for generating sound output signals, and may include an optional sound output port.
  • the power management and touch screen controller 220 manages power used by the processing/computation circuit 226 and touch control.
  • the communication bus 222 which
  • ISA Industry Standard Architecture
  • ISA Extended Industry Standard Architecture
  • PCI PCI
  • buttons for interconnecting all of the components of the processing/computation circuit 226.
  • FIG. 3 illustrates a user interface screen having an avatar in accordance with an exemplary aspect of the disclosure.
  • the disclosed avatar is a
  • FIG. 5 graphical representation of a digital makeup artist, but may also be a picture of a makeup artist 305, and may- represent a person which can communicate with a user using speech or text.
  • Speech input can be activated by selecting an icon 311 representing a microphone.
  • a subscreen 301 containing the avatar or picture 305 may provide an area for interacting.
  • the avatar or picture 305 can output speech to the computer’s audio output 242, or an external
  • the avatar or picture 305 can output text 309.
  • the subscreen may include an input box 307 where a user can type in text. In some embodiments. when a user speaks into the microphone 241 , the speech is translated into text, and the text is displayed in the input box 307.
  • the subscreen 301 may be contained in a user interface window 310.
  • the window 310 may be a graphical object that is controlled by the operating
  • the window 310 may display a browser window or a window of a software application or a mobile application.
  • the window 310 may include a menu icon 303 which when selected may display a menu of items that may be selected and perform specified functions.
  • the avatar 305 may be implemented as a software object that performs animations.
  • the avatar 305 may be synchronized with a video played in a video component 320. Control of the video component 320 may be performed by spoken commands that are made using the microphone 241 when the microphone icon 311 is activated. The avatar 305 may respond to speech input by performing natural language processing on the input and outputting a speech response through an audio
  • the avatar 305 may forward messages to the video component 320, and the video component 320 may share information, such as timing information, with the avatar 305.
  • a video played by the video component 320 may be broken up into chapters.
  • a chapter may be identified by a chapter name and time. The time may be in seconds from the beginning of the video, or in terms of a percentage of the
  • the video is a sequence of makeup application steps in which frames of the video are generated based on an image of a user’s face and makeup to be applied.
  • a makeup application step may include application of a certain type of makeup to a certain part of a user’s face image.
  • a certain type of makeup may include a particular makeup product, and particular characteristics of the
  • the avatar 305 is implemented as a conversational agent, which may ask questions and provide solutions with respect to the questions.
  • the conversational agent may also respond to certain types of questions asked by tire user.
  • the conversational agent may respond to statements by the user that are responses to questions that the
  • the conversational agent stores state information in the cloud service 105.
  • the conversational agent may have been previously created using a software tool, such as the Bot Framework SDK available for the
  • FIG. 4 illustrates a user interface screen for a mobile device in accordance with an exemplary aspect of the disclosure.
  • a user interface is provided for smaller screens, such as the screen in a smartphone 101.
  • interaction with the avatar 305 may be through audio and/or text.
  • Speech input can be activated by selecting an icon 311 representing a microphone.
  • the 25 210 may display controls and text associated with conducting a dialog.
  • the avatar 305 can output speech to the computer’s audio output 242, or an external speaker connected to the computer 101.
  • the avatar 305 can output text 309 to the display 210.
  • the section of the display screen 210 may include an input box 307 where a user can type in text. In some embodiments, when a user speaks into the microphone 241, the speech is translated into text,
  • the display screen 210 may- also display an image or video 401 of the user’s face.
  • FIG. 5 is a sequence diagram for interaction between a user and a digital makeup artist for a custom tutorial in accordance with an exemplary aspect of the disclosure.
  • the digital makeup artist 520 may take the form of the avatar 305 or may be conversational agent
  • the digital makeup artist 520 may be performed by a mobile application 111 and may represent a fictional person or an actual makeup artist.
  • a system 510 may be a desktop computer, laptop computer 103, tablet computer, mobile device
  • the system 510 may be a combination of any of a desktop computer, laptop computer, tablet computer, or mobile
  • a user input 530 may in the form of a text input area and/or a microphone input.
  • Play, Pause, Stop and may include a time slider for moving the start time of video playback while the video is paused.
  • the Digital Makeup Artist 520 is used
  • a user may use the mobile application 111 for a purpose such as a use case in which the user has a look in mind, but would like to have the Digital Makeup Artist show how the look may be created.
  • the mobile application 11 I may perform a custom tutorial to create the look.
  • a Digital Makeup Artist 520 of the mobile application 111 may perform an
  • the Digital Makeup Artist 520 may begin a dialog by displaying and/or speaking an initial question, such as: “What type of look do you want to create?”
  • the user may state a desired type of look.
  • the user’s statement may be entered
  • FIG. 6 is a user interface for inputting a user description of a type of look.
  • the user interface may provide a list of various types of looks. Examples of types of
  • 10 looks may include season looks (spring, summer, fall), event looks (Date night, Dinner with
  • Bridal, Prom looks based on time to complete (quick makeup, average makeup, take-your- time makeup), mood looks (cheery, happy, notice-me), styles (natural, evening, glam, gothic, work, beach), aesthetic looks (VSOC, eGirl, soft girl) to name a few.
  • the Digital Makeup Artist 520 may further display and/or speak a question, such as:
  • the user may be provided with a list of experience levels to select from, or may input via voice an experience level.
  • User 530 including level of experience may be found in a stored User Profile.
  • a level of experience may include novice/beginner look, experienced level, expert level, professional.
  • the novice/beginner level may be a user that has little or no experience in applying makeup.
  • the experienced level may be a user that has previously applied makeup, and thus has some experience.
  • the expert level may be a user that has been applying makeup for a while, such as a year or more, as well as has taken steps to learn how to properly apply
  • the professional level may be a user that applies to others.
  • the Digital Makeup Artist 520 may further display and/or speak a question, such as:
  • the Digital Makeup Artist 520 may- further display and/or speak a question, such as:
  • the Digital Makeup Artist 520 can perform a custom tutorial to teach steps to create
  • the Digital Makeup Artist 520 may teach the steps using the user’s own face and the steps may apply makeup in a manner that creates a look that is at it would appear when applied to the user’s face.
  • the camera 231 can be used to capture an image or video of the user’s face. The user may view the captured image or video on the display screen 210.
  • the image or video may be analyzed by tire system 510 in order to obtain information of the user’s face.
  • the image may be analyzed to identify the face contained in the image, determine the face shape, determine the parts of the face, and other characteristics that may be used to provide guidance during the tutorial.
  • Conventional image processing algorithms have been used to identify features in an image, such as image
  • one or both of the artificial neural networks may be performed by tiie mobile devicelOl or laptop computer 103 provided there are sufficient computer resources to perform processing of the artificial neural networks.
  • one or both of the artificial neural networks may be performed by a machine learning service 109
  • the artificial neural networks may be trained in the machine learning service 109, and the trained artificial neural networks may be performed in the mobile device 101 or laptop computer 103.
  • the mobile application 111 may perform image processing operations in order to improve image features, such as to improve lighting. For
  • a user may inadvertently take a self-picture when bright light or sunshine is from a direction behind the user.
  • the mobile application 111 may brighten the face image of the user.
  • Other image processing operations may be performed to improve the image quality.
  • FIG. 7 is a flowchart of the face analysis step in more detail.
  • an analysis may be performed on the captured image to determine a face
  • a machine learning model may be used to detect the face shape of the captured face of the user.
  • the machine learning model may be trained to classify face shape using face images with known face shapes.
  • Recently image classification has been performed using an artificial neural network that is inspired by how the visual cortex of human brain works when recognizing objects. This image classification artificial neural network is a family of models
  • CNN convolution neural networks
  • an architecture of a machine learning model that may be used to classify face shape is a CNN.
  • FIG. 8 is a block diagram of a CNN for classifying face shape. Dimensions and activation functions of the CNN may be varied depending on available processing power and desired accuracy. The dimensions include the number of
  • Activation functions include logistic, rectified linear unit, among others.
  • the convolution neural network may be made up of several types of layers.
  • a convolution component 803 may be made up of a convolution layer 803a, a pooling layer
  • the convolution layer 803a is for developing a 2-
  • the pooling layer 803c acts as a form of downsampling.
  • the rectified linear unit layer 803b applies an activation function to increase the nonlinear properties of the decision function and of the overall network without affecting the receptive fields of the convolution layer itself.
  • a fully connected layer 805 includes neurons that have connections
  • a loss layer specifies how the network training penalizes the deviation between the predicted and true layers.
  • the loss layer 807 detects a class in a set of mutually exclusive classes.
  • the loss layer 807 may be a softmax function.
  • the softmax function provides a probability for each class.
  • the classes 809 are exemplary embodiments.
  • 20 may include square, rectangular, round, oval, oblong, diamond, triangular, and heart face shapes.
  • tire mobile application may analyze facial features and landmarks. Similar to face shape, the facial features and landmarks of the captured face of the user may be detected using a machine learning model.
  • the machine learning model may be trained to detect facial landmarks. As with face shape classification, other approaches to classification may' be used.
  • a CNN architecture similar to FIG. 8 may be used as well for face landmark detection.
  • FIG. 9 is a diagram of a deep learning neural network for face landmark detection.
  • the deep learning neural network is a convolution neural
  • residual connections may be included.
  • inverted residual structures may be included in which residual connections are made to earlier layers in the network.
  • the network is provided as two stages, 903 and 905.
  • Hie first stage 903 is a convolution stage for performing feature extraction.
  • the second stage 905 is a convolution stage for performing feature extraction.
  • the architecture of the first stage 903 includes a convolution section 903a that, provided an input face image 901, performs convolution and max pooling operations.
  • the convolution section 903a is connected to an inverted residual structure 903b.
  • a mask layer
  • the mask layer 15 based on the number of landmarks (e.g., 2 x L, the number of landmarks).
  • 903c encodes the spatial layout of the input object.
  • the architecture of the second stage 905 includes an inverted residual structure 905b that is connected to the inverted residual structure 903b of the first stage 903. Also, the mask layer 903c of the first stage 903is applied to the results of the inverted residual structure 905b
  • Hie ROI and Concatenate Block 911 is based on the number of channels in the inverted residual structure 905b and the number of landmarks.
  • a predict block 913 predicts landmarks and approximate locations in the mask layer 905c. the predictions for the regions of interest of the second stage 903 are combined with the landmarks estimated by mask 903c
  • the landmarks for a face include eyes, nose, lips, cheekbones, areas around the eyes including eye brow's, eye lids, as well as hair.
  • landmarks may include possible facial anomalies.
  • each layer and the number of layers may depend on
  • the machine learning model may be trained using the machine learning service 109 of the cloud service 105.
  • Analysis of facial features, 507 may further include detection of lip shape 705, ey elid shape 707, and hair style 709.
  • the detected landmarks can be used to calculate contours of
  • skin color/tone skin color/tone
  • eye color eye color
  • lip color lip color
  • skin color/tone and skin texture may be determined using image processing techniques. Skin color/tone can be analyzed to determine RGB coordinates and assigned a name of the skin color/tone. The analysis of skin color may reveal differences in skin tone.
  • Types of skin tone may' include fair, light, medium or deep.
  • eye color, lip color and hair color may be determined using image processing techniques.
  • Skin texture may be analyzed using an analysis of variations in brightness.
  • An additional feature of a facial image may be lighting (image brightness).
  • image lighting (brightness) may also be determined using image processing techniques.
  • Brightness may be defined as a measure of the total amount of perceived light in an image.
  • brightness of an image may be increased or decreased from its initial as captured brightness level.
  • past look preferences may be retrieved from a database 107.
  • Past look preferences can include characteristics of a makeup product, including coverage, shade, finish
  • Past user preferences may include makeup product characteristics for a particular part of the face, and can also include a choice of makeup product that was applied for a particular look, in particular, a particular part of the face.
  • one or more makeup filters may be selected/rettieved from the database 107 based on the facial features (parts of a face) and past look preferences determined by the face
  • Some stored makeup face filters may be filters that have been previously created through the generation of a custom look during a custom tutorial or makeup consultation with a digital makeup artist 520.
  • a makeup filter is a feature mask that may be blended with a base frame, i.e., an image having a front view of a user’s face.
  • the feature mask is blended with the base frame
  • the App may make adjustments to the feature mask in order to line up with features of the face in the base frame using location of boundaries of facial features determined using face analysis 507.
  • the feature mask consists of RGB pixel values.
  • the one or more makeup filters may be retrieved from the database 107 using a
  • FIG. 10 is a diagram for a recommender system.
  • the recommender system 1000 may be used for retrieving makeup filters to be used in showing how to apply a virtual makeup (515 in FIG. 5).
  • the recommender system 1000 works off of an indexed database 1005 of image data and makeup filters which may be stored in database 107.
  • the indexed database 1005 may also include answers to common questions about
  • the indexed database 1005 may be populated with makeup information extracted from external databases or Web sites, such as product review information including ratings and public comments.
  • the indexed database 1005 may further include categories of cosmetic products,
  • Makeup products in this category may include a foundation that has anti-aging qualities or sun protection, and makeup products blended with medications, or other skin treatment products.
  • Another category may be skin care products for skin care routines to be used before applying makeup.
  • the recommender system 1000 includes a recommendation engine 1007 that retrieves and ranks recommended makeup filters.
  • a recommended makeup filter may be for the look that the user has input in step 501 and tiie virtual makeup.
  • the recommended makeup filter may be retrieved based on user preferences or favorites. Personal user preferences may be makeup
  • Personal user preferences may be one or more makeup filters that had previously been built during a custom tutorial or consultation with the digital makeup artist 520.
  • Favorites may be makeup characteristics that a user has flagged as being a favorite.
  • Personal user preferences and favorites may be for particular parts of a face or for the entire face.
  • the recommendation engine 1007 may use a look- feature matrix.
  • FIG. 11 illustrates a non-limiting look-feature matrix in accordance with an exemplary aspect of the disclosure.
  • the look-feature matrix illustrated in FIG. 11 is a partial matrix showing two types of virtual makeup for the sake of brevity. Other types of virtual makeup may be included in the look-feature matrix, including, but not limited to, foundation,
  • the look-feature matrix may be stored in the App in the mobile device to be compared to a vector of desired features.
  • the desired features may be a vector of current user preferences and may take into account the user’s current experience level and a desired look (501).
  • the recommendation engine
  • 1007 may use one or more similarity metrics and a scoring algorithm to rank
  • the recommendation engine 1007 may generate a set of features that elevate recommendations in order to encourage creativity' by changing certain characteristics for a virtual makeup from those that are recommended. For example, if the recommendation engine 1007 ranks a recommendation high among retrieved recommendations, it may then change one or more characteristics in order to increase a
  • the recommendation engine 1007 may change one or more characteristics in a retrieved recommendation, such as shade or finish, to one up or one down
  • the recommendation engine 1007 may adjust the application gesture to be more or less precise based on the experience level of the user.
  • the recommendation engine 1007 may be implemented by or supplemented with a machine learning model.
  • a machine learning model may be trained for choosing a shade of foundation that is appropriate for a particular skin undertone, skin type and/or ethnicity.
  • a machine learning model may be trained for choosing a shade of lipstick based on tire lip color of the User.
  • a machine learning model may be trained for
  • a machine learning model for the recommender engine 1007 may be trained using data in the indexed database 1005, as well as using data from external databases, especially those external databases having images and videos that a User may publish their custom looks.
  • Example external databases include databases for social media platforms such as
  • the machine learning model for the recommender engine 1007 may be based on algorithms for relatively small data sets, including decision trees, random forests, or single
  • a deep learning neural network may be used as the machine learning model.
  • Hie architecture for the deep learning neural network may be a variation of a convolution neural netw ork for recognizing features in color images.
  • the recommendation engine 1007 may output one or more recommendations to a
  • the recommendation user interface may also display a sequence of video frames that demonstrate application of a selected recommendation.
  • one or more look options may be displayed on the display screen 210.
  • FIG. 12 is an exemplary user interface in a mobile application.
  • Artist 520 may display custom recommendations 1201 of various looks.
  • the user may
  • the look may be selected by using a pointing device to move a pointer on the display screen 210, a pointing device on the touch screen 221, or by entering a voice command, such as: “Select LOOK1.”
  • a voice command such as: “Select LOOK1.”
  • the user may view other recommended looks, for example, by selecting a function to display a next screen of looks. In one
  • the user interface may provide a scroll bar 1203 that may allow scrolling to view the additional looks.
  • FIG. 13 illustrates a user interface that may be displayed to begin a tutorial in a mobile application in accordance with an exemplary aspect of the
  • tutorial in 517, information may be fed back to the system 510 in order to improve future interactions. Also, the tutorial is interactive. The tutorial may be controlled using commands, including SLOW DOWN, PAUSE, RE-DO A STEP, SKIP A
  • a message 1301 may be displayed to inform the user that commands can be used to control the tutorial.
  • the commands may be selected on the user interface using a pointing
  • the tutorial being played may be corrected to obtain a custom tutorial experience.
  • the command RE-DO A STEP may include an option to make a change to the tutorial.
  • the system 510 may save changes that are made to the tutorial to the database 107 as look preference data that may be used to improve future interactions.
  • FIG. 14 is an exemplary- mobile application in accordance with an exemplary aspect of the disclosure.
  • interaction with the digital makeup artist 520 may be through audio and/or text.
  • the commands may be input verbally.
  • the tutorial may involve applying makeup 1403a of a color
  • FIGs. 15, 16, 17, 18 are flowcharts for video control commands.
  • the tutorial may be controlled using commands, including SLOW DOWN, PAUSE, RE-DO
  • FIG. 15 is a flowchart of the SLOW DOWN video control
  • the SLOW DOWN command may perform a function, S1501 , of reducing the playback rate by a certain amount (X%).
  • the call for the SLOW DOWN command may indicate that the user believes that the portion of the video being played is complicated, or requires careful viewing.
  • the performance of a function of reducing the playback rate may include, in SI 503, storing information indicating that the step is
  • FIG. 16 is a flowchart of the PAUSE video control command.
  • the PAUSE command may perform a function, S1601, of stopping playback of the video at a certain time point (a position in seconds, or a time code).
  • FIG. 17 is a flowchart of the RE-DO A STEP video control command.
  • 25 STEP command may perform a function that makes use of chapters.
  • the beginning of each chapter may include a start image frame and time point.
  • the beginning of each chapter could be marked by a chapter frame.
  • the RE-DO A STEP command may perform a function that may cause the video to go back to the beginning of the current chapter and start playback over beginning from a starting image of the chapter.
  • the function may include reading the name of the current chapter.
  • the function may' store information indicating that the step associated with the current chapter is complicated.
  • the function may read an identifier that indicates the position of the beginning of the chapter, which could be a time point, or a chapter frame.
  • the video frames may be any type of video frames. As will be described later, in some embodiments, the video frames may be any type of video frames.
  • the beginning of a chapter includes a starting image, which may be a state of a face image combined with a mask filter. Any makeup application performed during the playback of the
  • the video may be played back starting from the beginning of the chapter.
  • FIG. 18 is a flowchart of the SKIP A STEP video control command.
  • STEP command may perform a function that may cause the video to skip to the beginning of the next chapter.
  • the function may begin by, in SI 801, reading the name of the current
  • Skipping a step may be an indication that the step is easy and the user does not need to be shown how to perform the step.
  • the function may store information indicating that the step is easy.
  • the function may read the name of the next chapter, and in
  • FIG. 19 is a block diagram of a video playback component in accordance with an exemplary aspect of the disclosure.
  • the video playback component in accordance with an exemplary aspect of the disclosure.
  • the video playback component may perform operations with one or more frames in a tutorial video
  • a video of a tutorial may be divided into chapters.
  • the chapters may be used to divide a video for a makeup tutorial into individual steps, where each step may be replayed, as well as paused or stopped. Each chapter may be designed by a start frame.
  • a chapter of the video may be for a step of applying makeup to a certain part of the user’s face, and may be for applying a certain type of makeup.
  • the video may begin with
  • the video may use information obtained through the analysis of the face image, such as location and labels of parts of the user’s face image, coloring, texture, and lighting.
  • the video may also make use of the selected look chosen by the user. By default, the video may be for a tutorial on how to create the selected look using the user’s face image and the information obtained from the face
  • a chapter of the video may be for a step of applying makeup to a certain part of the user’s face, and may be for applying a certain type of makeup.
  • a chapter may be for a step of applying a concealer to the eyelid of the user’s face image.
  • the concealer may be a makeup product that is provided for the selected look.
  • a makeup look may have a set of chapters, where each chapter may include one or more makeup products, where a makeup product has a set of characteristics, such as color, coverage, shade, and finish.
  • the video playback component 320 may generate the selected look for the particular user’s face and makeup products.
  • a chapter may begin with an image of the user’s face that was made up in a previous chapter, such that chapters represent cumulative results of makeup application.
  • the cumulative results are one or more mask filters that are separate from the original face image.
  • a face image, or a mask filter for a face image 1903 may be provided at the beginning of a chapter as a start image upon which further digital makeup will
  • the beginning of a chapter may include location information 1907 of a part of a face (facial feature) where the makeup product will be applied, as well as information on the makeup product, its characteristics, and type of strokes that might be applied by a specific type of makeup applicator.
  • the chapter may include an accompanying audio component, 1901.
  • a chapter may include a sequence of mask filters 1905 of desired
  • Each video frame may be generated by blending a prior frame 1911 and a desired feature 1913 to obtain a resulting feature 1915.
  • One or more feature masks may be used that represent applying a particular makeup using a makeup applicator for a particular facial feature.
  • FIG. 20 illustrates a blending process that may be used to create a video frame based
  • the desired feature 2001 is recolored, 2003, to match the color(s) of the original feature and obtain a recolored feature 2005.
  • the recolored feature 2005 is multiplied by a feature mask filter 2007.
  • the original feature 2009 is multiplied by the inverse 2011 (i.e., one minus each of the mask values, which range from 0 to 1) of the feature mask filter.
  • the user interface may provide a choice to save the completed look. If the user chooses to save the completed look, YES in 519, in 521, the look may be saved as a finished look, or the steps taken to apply makeup during the tutorial may be saved as a custom filter, or a sequence of custom filters. In addition, in 523, the user interface may provide an option to transfer the custom look or apply /transport the custom
  • FIGs. 21 A to 21F An example operation of the system is provided in FIGs. 21 A to 21F.
  • FIGs. 21 A to 21F An example operation of the system is provided in FIGs. 21 A to 21F.
  • 21 F are a sequence diagram for an exemplary interaction use case of using the Digital
  • Makeup Artist 520 for a makeup tutorial in accordance with an exemplary aspect of the disclosure.
  • the sequence diagram of FIGs. 21A to 21F includes operations and
  • the System 510 may be a computer device, such as a desktop computer. laptop computer 103, tablet computer, or mobile device 101, to name general types.
  • the System 510 may be a computer device, such as a desktop computer. laptop computer 103, tablet computer, or mobile device 101, to name general types.
  • System 510 may also take the form of a combination of a computer device and a cloud service 105 (see FIG. 1). In either case, a Digital Makeup Artist 520 may be performed as
  • Artist 520 either through speech, text, or a combination of speech and text.
  • the User 530 may first select the mobile application to be executed in a user interface of a computer device. Once the mobile application is started, the User 530 may be given a choice of using the Digital Makeup Artist 520 for teaching in a tutorial or providing advice in
  • the Digital Makeup Artist 520 may ask a question, “What kind of look do you want to create.”
  • the User 530 may state a name of a look.
  • the look may be a pre-existing type of look, or may the name of a new look that the User 530 has in mind.
  • the System 510 may provide a list of pre- existing looks for the User 530 to choose from. The pre-existing looks may be stored in the database 107 or may be provided locally in the App 111.
  • the Digital Makeup Artist 520 may ask the User 530 to provide their level of experience in applying makeup.
  • the level of experience may be a
  • the System 510 may present a list of experience levels for the User 530 to choose from In 2107, the User 530 indicates a level of experience.
  • the Digital Makeup Artist 520 may ask the User 530 how much of their face do they' want to apply makeup in a tutorial.
  • a tutorial may be provided for a part of a face or
  • the User may choose to have a tutorial presented for a part of the face, such as their eyes.
  • the Digital Makeup Artist 520 may ask the User 530 how much time they' have for a tutorial.
  • the tutorial may be a shortened version if a user does not have much time, or may be a full version if the user has sufficient time for the full
  • the User 530 may provide a response that is qualitative, or may provide a response that is an amount of time in minutes. In 2115, the User 530 may provide a qualitative response, such as, “I don’t have much time.” The System 510 may interpret this response as that a shortened version of a tutorial should be performed. However, the User 530 may further express that they are willing to save the tutorial at an intermediate state to be
  • the System 510 may save the tutorial at a point so that it may be played back starting from an intermediate point.
  • the Digital Makeup Artist 520 may ask the User 530 to determine that die User 530 wishes to use any necessary makeup and makeup applicators that are already available for use, for example, makeup products that die User 530 has
  • the System 510 may provide makeup and makeup applicators that may be needed for the tutorial.
  • the User 530 may provide a qualitative response to indicate that they have some makeup products, but would be open to the System 510 choosing makeup and makeup applicators.
  • the Digital Makeup Artist 520 may ask the User 530 to take a photo of their
  • the photo may be taken with a camera 231 that is built in to the mobile device 101 , or with a camera that is external.
  • the System 510 may perform analysis of the face image.
  • the face analysis may be performed to obtain locations of the parts of the face, and facial features including skin color, skin texture, lighting, as well as past look preferences,
  • the System 2125 may select a makeup for application in the tutorial.
  • the Digital Makeup Artist 520 may ask the User 530 if they like the choice of makeup.
  • the User 530 may respond that they do not agree with the choice, and would rather have the tutorial performed using a makeup provided by the User.
  • the Digital Makeup Artist 520 may indicate that the tutorial will begin. In some embodiments, in 2135, the Digital Makeup Artist 520 may provide a list of products that may be used during the tutorial, in case that the User 530 would like to obtain the products to apply makeup during the tutorial.
  • the System 510 may begin generating and playing the video of the tutorial.
  • the Digital Makeup Artist 520 may start the tutorial by stating initial makeup that may be applied.
  • the Digital Makeup Artist 520 may speak or provide a text instruction indicating that a concealer will be applied first, followed by a primer.
  • the video of the tutorial may be played by the System 510 in conjunction with instruction by the Digital Makeup Artist 520, either through speech output, text output, or
  • the Digital Makeup Artist 520 may provide instruction for selecting an eye shadow.
  • the Digital Makeup Artist 520 may provide instruction for choosing a blending brush.
  • the Digital makeup Artists 520 may provide instruction for applying eye shadow.
  • the User 530 may input a video control command.
  • the User 530 may input a command to RE-DO A STEP.
  • the command may- include a request to make a change to a step, such as Re-do with a one-up color.
  • the Digital Makeup Artist 520 may output a request for clarification as to what the user means by “one-up color.”
  • the System 510 may generate new frames for a video beginning with selecting eye shadow of 2145 that uses a new color.
  • the Digital Makeup Artist 520 may provide instruction for choosing a blending brush. In 2157, the Digital Makeup Artist 520 may provide instruction for applying eye shadow.
  • the User 530 may input a command, such as PAUSE.
  • the System 510 may perform a pause function, until the User 530 inputs a PLAY command.
  • the Digital Makeup Artist 520 may provide instruction for performing blending with another shade of eye shadow.
  • the Digital Makeup Artist 520 may provide instruction for selecting an eye shadow of the next shade.
  • the Digital Makeup Artist 520 may provide instruction for selecting an eye shadow of the next shade.
  • Makeup Artist 520 may provide instruction for choosing a blending brush.
  • the Makeup Artist 520 may provide instruction for choosing a blending brush.
  • Digital Makeup Artist 520 may provide instruction for performing blending.
  • the User 530 may input a command, such as “RE-DO A STEP,” to re do the blending.
  • the System 510 may restart the video from the beginning of the blending step.
  • the Digital Makeup Artist 520 may provide instruction for performing sharpening of an eye brow.
  • the Digital Makeup Artist 520 may provide instruction for performing a step of cutting of crease.
  • the Digital Makeup Artist 520 may provide instruction for performing a step of selecting a concealer with a pearl finish.
  • the Digital Makeup Artist 520 may input a statement that the concealer is too bright, and ask if the concealer can be toned down.
  • the Digital Makeup Artist 520 may input a statement that the concealer is too bright, and ask if the concealer can be toned down.
  • 5 may respond by stating that it will try another concealer, which may be performed by issuing
  • the System 510 may restart the video from the beginning of the step of cutting of a crease, depending on where the beginning of the chapter occurs.
  • the System 510 may, as an example, select a concealer with mate finish and play the video
  • the User 530 may input a PAUSE command to take a moment to observe the result of the new concealer.
  • the User 530 may input a PLAY command to start playing the video.
  • the Digital Makeup Artist 520 may provide instruction for performing a step of shaping around an eye. In 2193, the Digital Makeup Artist 520 may provide instruction for
  • the System 510 may choose a pencil that is appropriate for the user’s skin tone. In 2197, the System 510 may draw on the face image using the chosen pencil.
  • the Digital Makeup Artist 520 may provide instruction for performing a step of applying eye liner.
  • the User 530 may choose to stop the video to be
  • Artist 520 may output a question as to whether the user would like to save the finished look.
  • the User may choose to save the look and instruct the System 510 to save the look.
  • the System 510 saves the final look to the database 107.
  • the Database 107 In some embodiments, the
  • 25 final look may be saved in the database as a preferred look for a certain type of look. For example, a look created using a tutorial may be saved as a preferred look for the type of look that was initially described (see 503 in FIG. 5). The final look may also be transported to other platforms, such as social media platforms or video conferencing platforms. Some current social media platforms where a user may post a photograph or video include
  • Some current video conferencing platforms include Microsoft Teams, FaceTime, Google Hangouts or Google
  • the System 510 may save to the database 107 one or more mask filters associated with the creation of the final look.
  • the one or more mask filters may be transported to other platforms, and may be used to create a custom look for the other platform
  • FIGs. 22A to 22E are a sequence
  • the Digital Makeup Artist 520 may ask a question, “What kind of look do you want to create.”
  • the User 530 may
  • the eGirl look may be a look that a user would like to try for posting in social media or when video conferencing with friends or colleagues.
  • a pre-existing eGirl look may be stored in the database 107 or may be provided locally in the App 111.
  • the Digital Makeup Artist 520 may ask the User 530
  • the level of experience may be a category, such as New, Experienced, Expert, and Professional.
  • the System 510 may present a list of experience levels for the User 530 to choose from In 2227, the User 530 indicates a level of experience as Experienced, but is new to applying an eGirl look.
  • the Digital Makeup Artist 520 may ask the User 530 how much of their face
  • a tutorial may be provided for a part of a face or the whole face.
  • the User may choose to have a tutorial presented for the whole face.
  • the Digital Makeup Artist 520 may ask the User 530 how much time they' have for a tutorial.
  • the tutorial may be a shortened version if a user does not have much time, or may be a full version if the user has sufficient time for the full
  • the User 530 may provide a response that is qualitative, or may provide a response that is an amount of time in minutes. In 2235, the User 530 may provide a qualitative response, such as, “I don’t have much time.” The System 510 may interpret this response as that a shortened version of a tutorial should be performed. In some embodiments, in 2237, the
  • Digital Makeup Artist 520 may ask the User 530 to determine that the User 530 wishes to use
  • any necessary digital makeup and makeup applicators that are already available for use for example, digital makeup products that the User 530 has purchased, or that the User 530 would prefer that the System 510 may provide digital makeup and makeup applicators that may be needed for the tutorial.
  • the User 530 may provide a qualitative response to indicate that they' have some digital makeup products.
  • the Digital Makeup Artist 520 may ask the User 530 to take a photo of their face. The photo may be taken with a camera 231 that is built in to the mobile device 101, or with a camera that is external.
  • the System 510 may perform analysis of the face image. As mentioned
  • the face analysis may be performed to obtain locations of the parts of the face, and facial features including skin color, skin texture, lighting, as well as past look preferences, which is information that may be used in generating a video for the tutorial.
  • the System 510 may select a digital makeup for application in the tutorial.
  • the Digital Makeup Artist 520 may ask the User 530 if they like the choice of
  • the User 530 may respond that they do not agree with the choice, and would rather have the tutorial performed using digital makeup provided by the User.
  • the Digital Makeup Artist 520 may indicate that the tutorial will begin. In some embodiments, in 2255, the Digital Makeup Artist 520 may provide a list of digital makeup products that may be used during the tutorial, in case that the User 530 would like to
  • the System 510 may begin generating and playing the video of the tutorial.
  • the Digital Makeup Artist 520 may start the tutorial by stating initial makeup that may be applied.
  • the Digital Makeup Artist 520 may speak or provide a text instruction indicating that a primer will be smoothed on, in order to prep for foundation.
  • the video of the tutorial may be played by the System 510 in conjunction with instruction by the Digital Makeup Artist 520, either through speech output, text output, or both speech and text.
  • the Digital Makeup Artist 520 may provide instruction for selecting foundation having light coverage and natural finish.
  • the Digital Makeup Artist 520 may provide instruction for selecting foundation having light coverage and natural finish.
  • Artist 520 may provide instruction for sweeping Blush across the bridge of the nose, and extending slightly onto the cheeks.
  • the Digital makeup Artists 520 may provide instruction for applying Highlighting Power onto tip of nose using tapered brush.
  • the User 530 may input a video control command.
  • the User 530 may input a command to RE-DO A STEP.
  • the command may include a request to make a change to a step, such as Re-do with a one-up blush color.
  • the Digital Makeup Artist 520 may output a request for clarification as to what the user means by “one-up blush color.”
  • the System 510 may generate new frames for a video beginning with sweeping blush of 2267 that uses a new blush color.
  • the Digital Makeup Artist 520 may provide instruction for applying eyebrow mechanical pencil with light strokes. In 2277, the Digital Makeup Artist 520 may provide instruction for combing through brows.
  • the User 530 may input a command, such as PAUSE.
  • the System 510 may perform a pause function, until the User 530 inputs a PLAY command.
  • the Digital Makeup Artist 520 may provide instruction for selecting eye shadow of pink shade. In 2283, the Digital Makeup Artist 520 may provide instruction for selecting a foot applicator for applying eye shadow. In 2285, the Digital Makeup Artist 520 may provide instruction for applying the eye shadow to eye lids. In 2287, the Digital Makeup
  • Artist 520 may provide instruction for performing blending into the crease with a fluffy
  • the User 530 may input a command, such as “RE-DO A STEP,” to re do the blending.
  • the System 510 may restart the video from the beginning of the blending step.
  • the Digital Makeup Artist 520 may provide instruction for creating a wing
  • the Digital Makeup Artist 520 may provide instruction for performing a step of applying mascara to lashes using a few swipes.
  • the Digital Makeup Artist 520 may provide instruction for applying a barely-there lip gloss.
  • the User 530 may input a statement that the lip gloss is too light, and ask if the lip gloss can be toned up.
  • Artist 520 may output a question as to whether the user would like to save the finished look.
  • the User 530 may choose to save the look and instruct the System 510 to save the look.
  • the System 510 saves the final look to the database 107.
  • the final look may be saved in the database as a preferred look for a certain type of look, such
  • the eGiri look created using a tutorial may be saved as a preferred look for the egirl look that was initially described (see 503 in FIG. 5).
  • the final look may also be transported to other platforms, such as social media platforms or video conferencing platforms.
  • Some current social media platforms where a user may post a photograph or video include Facebook, Linkedln, Instagram, TikToc, and Snapchat, to name
  • Some current video conferencing platforms include Microsoft Teams, FaceTime,
  • changes that were made during the tutorial such as choice of shade of a color, finish of a color, may be stored as look preference data.
  • the System 510 may save to the database 107 one or
  • the one or more mask filters may be transported to other platforms, and may be used to create a custom look for the other platform
  • Makeup Artist 520 can provide personal makeup consultation.
  • the Digital Makeup Artist 520 can provide personal makeup consultation.
  • the Digital Digital Converter 25 may provide advice on how to boost the user’s makeup look, or provide advice on improvements that may be made to the user’s makeup application technique.
  • Makeup Artist 520 may provide advice on a makeup application technique that may better address a problem area, or bring out a special facial feature.
  • the Digital Makeup Artist 520 may provide advice on makeup application that may be appropriate for time of day (morning,
  • the Digital Makeup Artist 520 may provide advice on ways to bring out the user’s personality.
  • FIG. 23 is a sequence diagram for interaction between a digital makeup artist and a
  • the User 530 may be given a choice of using the Digital Makeup
  • the User 530 may request cosmetic advice via the user interface window 310 (see FIG.
  • the System 510 may request that the User 530 take a photograph or video of their
  • the System 510 may analyze an image of the user’s face to identify face parts, sense skin tone, lip color, hair color, and skin texture.
  • the System 510 may conduct an interactive dialog with the User 530 to obtain further information related to a user’s needs, including skin condition, indoor/outdoor looks, favorite facial features, and any facial feature concerns.
  • the User 530 may
  • the System 510 creates one or more custom recommendations for
  • custom recommendations may be obtained using a recommender system
  • the recommender system 1000 may be used for retrieving makeup filters to be used in creating the custom recommendations for makeup routines and may be used for retrieving cosmetic routines, such as for skin care .
  • the System 510 may display the retrieved one or more recommended cosmetic routines. Makeup routines may be
  • the recommender system 1000 includes a recommendation engine 1007 that retrieves and ranks recommended makeup filters.
  • the recommendation engine 1007 may also draw from an external repository for additional information, including one or more of frequently asked questions and answers, common makeup looks, cosmetic products for specific skin
  • the recommender engine 1007 may further draw from an external repository of cosmetic categories, including skin care, and makeup that has skin care qualities, Makeup that has skin care qualities may include foundation with SFP for protection from sunlight, makeup with anti-aging qualities.
  • a recommended makeup filter may be for favorite features and facial feature
  • step 15 concerns that the user has input in step 2357.
  • the recommender engine 1007 may be supplemented with a machine learning model to recommend a shade of makeup based on skin undertone.
  • the user interface window 310 may take input from
  • the further user input can be in the form of refinements or adjustments in particular features shown in the recommended routine. For example, the user may input that the lipstick color is too bold.
  • the further user input can be in the form of refinements or adjustments in the overall makeup routine. For example, the user may input that the makeup routine is too bold, or that the user
  • the System 510 may make adjustments to the recommended makeup routine in accordance with the user input and makeup look data that matches the type of adjustment. For example, the System 510 may make an adjustment by retrieving a mask filter for longwear makeup that has been stored in the database 107. In
  • the System 510 may generate and display the revised makeup routine for the user’s
  • the System 510 may provide recommendations for makeup products that may be used to create the finished face image using the makeup routine.
  • the System 510 may store the finished face image and adjustments to the makeup routine used in creating the finished face image in the database
  • the mask filter for longwear makeup may be stored with a label that it is a makeup look preference (makeup filter) for a makeup routine
  • the User 530 may choose to transport/publish the created finished face image to a platform that provides live video or still images that the
  • Platforms that provide live video include social media platforms and video conferencing platforms, including Facebook, Linked-in, Google
  • FIGs. 24A to 24D are a sequence diagram for an exemplary interaction
  • FIGs. 24 A to 24D includes operations and communication by a System 510, the Digital Makeup Artist 520 and a User 530.
  • the User 530 may select a mobile application for the Digital Makeup Artist.
  • the User 530 may ask to obtain
  • the Digital Makeup Artist 520 may ask whether the user has a preferred look that they wish for advice about.
  • the User 530 may provide an answer that further defines the desired look.
  • the Digital Makeup Artist 520 may further narrow down the type of advice to give by, in 2409, asking the user if they have a makeup look preference.
  • the User 530 may provide an answer that indicates that they do not have a
  • the System 510 may request that the user take a photo or video of their face and will perform analysis on the image of the user’s face.
  • the results of the analysis may include location of parts of the user’s face, and characteristics such as skin color, skin texture, lighting, as well as previous look preferences.
  • the Digital Makeup Artist 520 may ask whether the user has a favorite facial feature.
  • the User 530 may respond with one or more favorite facial features, for example. lips as in 2417.
  • the System 510 may create a custom recommendation for one or more makeup routines.
  • the System 510 may display the makeup routines and makeup product characteristics.
  • the Digital Makeup Artist 520 may ask the user
  • User 530 may provide a response, such as that the makeup look is too bold.
  • the Digital Makeup Artist 520 may respond by asking whether it is a particular part, or parts, of the face that stand out, or is the entire face too bold. In 2431, the
  • the System 510 performs an adjustment to the makeup look which may involve retrieving a mask filter from database 107, and the choice of mask filter may take into account past look preferences.
  • the System 510 may display the adjusted makeup look.
  • the User 530 may review the adjusted makeup look, and provide further
  • the System 510 may make an adjustment to the makeup look, and in 2443, display the further adjusted look.
  • the Digital Makeup Artist 520 may inform the user that the system has increased the shade of the eye shadow, and ask the user if that adjustment sufficiently improves the brightness of the eyes.
  • the User 530 may respond with a
  • the System 510 may adjust the shade further, and in 2451, may display the further adjusted look.
  • tire Digital Makeup Artist 520 may again inform the user that the system has increased the shade, and again ask the User 530 if the adjustment is sufficient.
  • tire User 530 may respond that the adjustment looks good.
  • the System 510 may display the final makeup look.
  • the System 510 may display the makeup products that may be used to create tire final makeup look, and in 2465, save tire makeup routine, final makeup look, and adjustments that were made, as user look preferences in the database 107.
  • the Digital Makeup Artist 520 provides makeup advice that is tailored to the user.
  • the Digital Makeup Artist 520 continues to improve its recommendations through an accumulation of user look preferences and custom looks.
  • the Digital Makeup Artist 520 can try on makeup looks for a user before
  • the user applies makeup to their own face, and teaches the user how to apply the makeup to create a custom look.
  • the Digital Makeup Artist 520 can create custom makeup looks based on stored look preferences and information about the user’s facial features.
  • the words “a,” “an” and the like generally carry a meaning of “one or more,” unless stated otherwise.
  • Numerous modifications and variations of the present invention are possible in light of tiie above teachings. For example, data gathered from various consumers’ skin tones and texture will allow scaling of the artificial neural network to more than a single consumer.
  • the artificial neural network will be able to predict for each product shade the rendering of
  • the digital makeup artist system includes a mobile device having a display device, computation circuitry, and a memory; a database system storing cosmetic routine information, common makeup looks, cosmetic products for skin types and ethnicity, and user look preferences of a user; a machine learning system for analyzing an image of a face; and the mobile device includes a user interface for interacting
  • the computation circuitry is configured to input a face image of the user, analyze, via the machine learning system, the user’s face image to identify face parts, analyze the face image
  • facial characteristics including one or more of skin tone, eye color, hair color, lip color, and skin texture
  • image frames to be displayed on the display device in synchronization with the interaction with the digital makeup artist, in which the image frames are generated based on the analyzed face image of the user, the needs of the user obtained through tiie interactions with the user, one or more of the stored cosmetic routine information, common makeup looks, cosmetic products for skin types and ethnicity, and tire user look preferences.
  • tire video includes chapters, where each chapter is a step in a cosmetic routine, and tire input from the user includes a video control command to control playing of the video.
  • the digital makeup artist system of feature (6) in which the adjustment to generation of image frames includes a change to a characteristic of a color of a makeup for a face part in the face image, and the characteristic of the color is one or more of coverage,
  • interactive dialog with the user that is conducted by the digital makeup artist includes speech input, the mobile device performs natural language processing on the speech input, and the digital makeup artist outputs a speech response.
  • the digital makeup artist receives a speech input indicating that the user’s skin is dry', and die image frames are generated based on cosmetic routines that are stored in relation to the indication that the user’s skin is dry.
  • the image frames are generated based on the cosmetic routine information and cosmetic products for skin care.
  • a digital makeup artist system includes a mobile device having a display device, computation circuitry, and a memory; a database system storing cosmetic routine information, common makeup looks, cosmetic products for skin types and ethnicity, and user look preferences of a user; a machine learning system for analyzing an image of a face; and tire mobile device includes a user interface for interacting
  • the computation circuitry is configured to input a face image of the user, analyze, via the machine learning system, the
  • face image to identify face parts, analyze tire face image to determine facial characteristics including one or more of skin tone, eye color, lip color, hair color, and skin texture, and generate image frames to be displayed on the display device in synchronization with the interaction with the digital makeup artist to provide the advice, in which the image frames are generated in synchronization with the interaction based on the analyzed face
  • the computation circuitry performs the interaction that creates a custom recommendation for a cosmetic routine that is tailored for the favorite facial features.
  • the interaction includes the digital makeup artist prompting the user to input at least one problem facial area that is of concern, and the computation circuitry performs the interaction that creates a custom recommendation for a cosmetic routine that is tailored for the problem facial area that is of concern.
  • circuitry is configured to output recommended makeup and skin care products for the refined cosmetic routine.
  • adjustment of the recommended cosmetic routine includes a request to change a makeup characteristic of a face part in the face image, and the request to change the makeup characteristic includes a change of one or more of coverage, shade, and finish of a makeup color in the face part in the face image.
  • the digital makeup artist receives a speech input indicating that the user’s skin is dry, and the image frames are generated based on cosmetic routines that are stored in relation to the indication that the user’s skin is dry.
  • the image frames are generated based on the cosmetic routine information and cosmetic products for skin care.
  • die cosmetic product is a foundation
  • die skin undertone machine learning model is for choosing the shade of the foundation that is appropriate for the particular skin undertone of the face image.
  • 20 computation circuitry performs the interaction that creates a custom recommendation for a cosmetic routine including obtaining the user’s skin type and ethnicity, and die image frames of the cosmetic routine are generated based on the user’s skin type and ethnicity and user look preferences of a user in order to create the custom recommendation that uses a cosmetic product stored in the database that is designated for the skin type and die ethnicity.
  • the digital makeup artist system of feature (38) in which the cosmetic routine information and cosmetic products stored in the database system are for skin care-type makeup, for skin type and ethnicity, and the image frames are generated based on the cosmetic routine information and cosmetic products for skin care-type makeup, for the skin

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Development Economics (AREA)
  • General Engineering & Computer Science (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Economics (AREA)
  • Human Computer Interaction (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Game Theory and Decision Science (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Un système de maquilleur numérique comprend un dispositif mobile, un système de base de données stockant des informations sur la routine cosmétique, des styles de maquillage courants, des produits cosmétiques pour des types de peau et une ethnicité, et des préférences de style d'un utilisateur. Le dispositif mobile comprend une interface utilisateur destinée à interagir avec un maquilleur numérique. Le maquilleur numérique effectue un dialogue interactif avec l'utilisateur afin de capturer les besoins de l'utilisateur, y compris des types de style de maquillage, de style en intérieur ou en extérieur, de l'état de la peau, de zones à problème du visage, de traits de visage préférés. Les circuits de calcul analysent l'image du visage de l'utilisateur pour identifier des parties de visage, analysent l'image de visage pour déterminer des caractéristiques faciales, et génèrent des trames d'image à afficher en synchronisation avec l'interaction avec le maquilleur numérique sur la base de l'image de visage analysée, des besoins de l'utilisateur, des informations sur la routine cosmétique stockées, des styles de maquillage courants, des produits cosmétiques pour des types de peau et une ethnicité, et les préférences de style de l'utilisateur.
EP21843850.5A 2020-12-30 2021-12-21 Maquilleur numérique Pending EP4260172A1 (fr)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US17/138,143 US11461946B2 (en) 2020-12-30 2020-12-30 Digital makeup artist
US17/138,078 US11657553B2 (en) 2020-12-30 2020-12-30 Digital makeup artist
FR2107906A FR3125613A1 (fr) 2021-07-22 2021-07-22 Maquilleur numérique
FR2107909A FR3125610A1 (fr) 2021-07-22 2021-07-22 Maquilleur numérique
PCT/US2021/064507 WO2022146766A1 (fr) 2020-12-30 2021-12-21 Maquilleur numérique

Publications (1)

Publication Number Publication Date
EP4260172A1 true EP4260172A1 (fr) 2023-10-18

Family

ID=79601754

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21843850.5A Pending EP4260172A1 (fr) 2020-12-30 2021-12-21 Maquilleur numérique

Country Status (4)

Country Link
EP (1) EP4260172A1 (fr)
JP (1) JP2024505359A (fr)
KR (1) KR20230118191A (fr)
WO (1) WO2022146766A1 (fr)

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7437344B2 (en) * 2001-10-01 2008-10-14 L'oreal S.A. Use of artificial intelligence in providing beauty advice
US20160093081A1 (en) * 2014-09-26 2016-03-31 Samsung Electronics Co., Ltd. Image display method performed by device including switchable mirror and the device
US11315173B2 (en) * 2016-09-15 2022-04-26 GlamST LLC Applying virtual makeup products
US10537165B2 (en) * 2018-01-05 2020-01-21 L'oreal System including a makeup compact and client device for guiding emakeup application

Also Published As

Publication number Publication date
JP2024505359A (ja) 2024-02-06
KR20230118191A (ko) 2023-08-10
WO2022146766A1 (fr) 2022-07-07

Similar Documents

Publication Publication Date Title
US11776187B2 (en) Digital makeup artist
US10799010B2 (en) Makeup application assist device and makeup application assist method
US11961169B2 (en) Digital makeup artist
US10783672B2 (en) Makeup part generating apparatus, makeup part utilizing apparatus, makeup part generating method, makeup part utilizing method, non-transitory computer-readable recording medium storing makeup part generating program, and non-transitory computer-readable recording medium storing makeup part utilizing program
US9058765B1 (en) System and method for creating and sharing personalized virtual makeovers
US20190026013A1 (en) Method and system for interactive cosmetic enhancements interface
US20220202168A1 (en) Digital makeup palette
US20130301947A1 (en) Method and system for applying cosmetic and/or accessorial enhancements to digital images
CN109359317A (zh) 一种口红配色选择的模型构建方法和口红配色选择方法
CN103995911A (zh) 基于智能信息终端的美容选配方法与系统
WO2021197186A1 (fr) Procédé de maquillage auxiliaire, dispositif terminal, support de stockage et produit de programme
CN111968248A (zh) 基于虚拟形象的智能化妆方法、装置、电子设备及存储介质
CN106960187A (zh) 化妆导航系统、装置和方法
CN112819718A (zh) 图像处理方法及装置、电子设备及存储介质
WO2022257766A1 (fr) Procédé et appareil de traitement d'image, dispositif et support
JP2022043244A (ja) アバタに用いられるスライダ処理方法及び装置、電子機器、記憶媒体及びコンピュータプログラム
US11321882B1 (en) Digital makeup palette
CN112083863A (zh) 图像处理方法、装置、电子设备及可读存储介质
EP4260172A1 (fr) Maquilleur numérique
CN114021022A (zh) 一种妆容信息获取方法、装置、车辆及存储介质
US20180181110A1 (en) System and method of generating a custom eyebrow stencil
WO2022146615A1 (fr) Palette de maquillage numérique
KR102532561B1 (ko) 개인 스타일에 대한 컨설팅 데이터를 제공하는 방법
US11943564B2 (en) Providing video appearance adjustments within a video communication system
FR3125613A1 (fr) Maquilleur numérique

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230713

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)