WO2022001308A1

WO2022001308A1 - Interaction method, interaction apparatus, and intelligent mirror

Info

Publication number: WO2022001308A1
Application number: PCT/CN2021/088728
Authority: WO
Inventors: 董辰; 卢曰万; 徐涵; 赵学知; 丁弦; 郜文美
Original assignee: 华为技术有限公司
Priority date: 2020-06-28
Filing date: 2021-04-21
Publication date: 2022-01-06
Also published as: CN113850708A

Abstract

An interaction method, an interaction apparatus, and an intelligent mirror. The interaction method comprises: determining a target makeup look, wherein the target makeup look is one or more facial local makeup looks (101); prompting an operation step regarding the target makeup look (102); and before a prompt action of the operation step is completed, periodically executing the following adjustment operations: collecting the current facial features of a user; and calculating the similarity between the current facial features of the user and the makeup look of a corresponding part in the target makeup look, and according to the similarity, adjusting, in real time, the operation step regarding the target makeup look which subsequently needs to be prompted (103); determining whether the prompting of the operation step is completed (104); and if so, ending the process (105). By means of the method, the operation of a target makeup look of one or more facial local features can be guided, the pertinence of interaction between a user and a corresponding auxiliary makeup device during a makeup process is refined, and the high efficiency and the flexibility of the interaction manner are improved.

Description

Interactive method, interactive device, smart mirror

technical field

The present invention relates to the technical field of smart home, in particular to an interaction method, an interaction device and a smart mirror.

Background technique

With the improvement of people's living standards, more and more people put more energy into skin care and makeup. At present, there are many beauty apps that can beautify the photos taken, including the selection and beautification of eye makeup.

In the prior art, the technical solutions for makeup detection are to detect the entire face of a human face, and to provide makeup guidance based on the results of the full-face detection. Lack of detection of local features of faces cannot provide targeted makeup guidance services. In addition, in the prior art, after the makeup instruction, the user cannot know the quantitative result after makeup.

Application content

The embodiments of the present invention provide an interaction method, an interaction device, and a smart mirror, which can periodically compare the user's current facial features with the determined target makeup during the operation guidance process for target makeup with respect to one or more partial facial features. The similarity calculation is performed on the makeup of the corresponding parts in the middle, and the subsequent operation steps that need to be prompted are adjusted in real time according to the similarity calculation result. Through the above method, the pertinence of the interaction between the user and the corresponding makeup auxiliary device during the makeup process can be refined, and the efficiency and flexibility of the interaction method can be improved.

An embodiment of the present invention provides an interaction method based on a smart mirror, wherein the method includes:

determining a target makeup, where the target makeup is one or more partial facial makeup;

Prompt steps for the target makeup look; and

Before the prompt action of the operation step is completed, periodically perform the following adjustment operations:

Collect the current facial features of the user; and

Calculate the similarity between the current facial feature of the user and the makeup of the corresponding part of the target makeup, and adjust the operation steps about the target makeup that need to be prompted in real time according to the similarity.

Further, the determined target makeup includes:

Determine whether a selection makeup instruction is received;

Wherein, in the case of receiving the selection makeup instruction, the selected makeup is used as the target makeup;

In the case of not receiving the instruction to select makeup, perform the following operations:

The current facial features of the user are compared with multiple pre-stored makeup looks, and the closest makeup look is used as the target makeup look.

Further, the process of comparing the current facial features of the user with multiple pre-stored makeup looks, and using the closest makeup look as the target makeup look includes:

Determine the eye shadow smear boundary;

obtaining the skin color within the boundaries of the eye shadow application;

Get multiple pre-stored eye shadow makeup colors;

Comparing the color within the eye shadow smearing boundary with the color of each of the pre-stored eye shadow makeup looks respectively, and using the pre-stored eye shadow makeup look corresponding to the highest similarity whose similarity is higher than the eye shadow color preset value as the target eye shadow makeup look; and

Trigger the operation steps for prompting the target eye shadow makeup.

acquiring the inner corner eyeliner, the outer upper corner eyeliner and the outer lower corner eyeliner, and determining the target eyeliner makeup according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner and the outer lower corner eyeliner; and

Trigger to prompt the operation steps about the target eyeliner makeup.

Further, determining the target eyeliner makeup according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner and the outer lower eyeliner eyeliner includes:

If the inner corner eyeliner is not obtained and the outer lower eyeliner is not obtained, the target eyeliner makeup is determined according to the first mapping relationship, and the first mapping relationship is the correspondence between the type of the outer upper corner eyeliner and the target eyeliner makeup ;

If the inner corner eyeliner is obtained and the outer lower eyeliner is not obtained, then the target eyeliner makeup is determined according to a second mapping relationship, and the second mapping relationship is the correspondence between the type of the outer upper corner eyeliner and the target eyeliner makeup;

If the inner corner eyeliner is not acquired, and the outer lower corner eyeliner is acquired, the target eyeliner makeup is determined according to a third mapping relationship, and the third mapping relationship is the correspondence between the type of the outer upper corner eyeliner and the target eyeliner makeup.

Further, the process of calculating the similarity between the current facial feature of the user and the makeup look of the corresponding part in the target makeup look, and adjusting the operation steps about the target makeup look that needs to be prompted later according to the similarity degree in real time includes:

Determine the skin color of the face setting area;

Determine the eye shadow smear boundary;

Obtain the skin color outside the eye shadow smearing boundary. If the deviation between the skin color outside the eye shadow smearing boundary and the skin color of the face setting area is greater than the preset value of the color deviation, a prompt about repairing the eye shadow smearing out of bounds operation is displayed. step.

Obtain the eye image of the user with their eyes closed;

determining a fitted curve for the actual eyeliner in the eye image;

Determine the distance between the actual eyeliner and the fitting curve, and if the distance is greater than the preset value of the line distance, an operation step for improving the smoothness of the eyeliner is prompted.

Determine one or more of the following user's eyelash characteristics when the user's eyes are closed: eyelash length, eyelash curling degree, eyelash density, and eyelash adhesion;

If any one of the user's eyelash features does not meet the preset conditions of the corresponding eyelash feature, a corresponding operation step for repairing the eyelashes is prompted.

Further, the method also includes:

After the prompt action of the operation step is completed, the similarity between the current facial feature of the user and the makeup look of the corresponding part in the target makeup look is converted into a corresponding value, and the display terminal is controlled to display the corresponding value.

An embodiment of the present invention further provides an interaction device, and the device includes:

A processor, a memory and a camera, the camera is used for collecting the current facial information of the user, the memory is used for storing at least one instruction, the instruction is loaded and executed by the processor to implement the above interaction method.

An embodiment of the present invention also provides a smart mirror, the smart mirror comprising:

the above-mentioned interactive device; and

A display screen, the display screen is connected with the interaction device.

The interaction method, the interaction device, and the smart mirror provided by the embodiments of the present invention can periodically compare the current facial features of the user with the determined facial features during the operation guidance process of the target makeup with respect to one or more partial facial features through the above technical solutions. Similarity calculation is performed on the makeup of the corresponding part of the target makeup, and the subsequent operation steps that need to be prompted are adjusted in real time according to the similarity calculation result. Through the above method, the pertinence of the interaction between the user and the corresponding makeup auxiliary device during the makeup process can be refined, and the efficiency and flexibility of the interaction method can be improved.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic flowchart of an interaction method based on a smart mirror provided by an embodiment of the present invention;

2A is a schematic flowchart of determining a target eye shadow makeup according to an embodiment of the present invention;

2B is a schematic diagram of an eye shadow smearing range provided by an embodiment of the present invention;

3A is a schematic flowchart of determining target eyeliner makeup according to an embodiment of the present invention;

3B is a schematic diagram of eyeliner types provided by an embodiment of the present invention;

3C is a schematic diagram of a first mapping relationship provided by an embodiment of the present invention;

3D is a schematic diagram of a second mapping relationship provided by an embodiment of the present invention;

3E is a schematic diagram of a third mapping relationship provided by an embodiment of the present invention;

4 is a schematic diagram of a quantitative result about eyelash makeup provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an interaction device based on a smart mirror provided by an embodiment of the present invention.

detailed description

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

FIG. 1 shows a schematic flowchart of an interaction method based on a smart mirror provided by an embodiment of the present invention. As shown in FIG. 1 , the interaction method based on a smart mirror includes:

Step 101 determines a target makeup, the target makeup is one or more partial facial makeup;

Step 102 prompts the operation steps about the target makeup; and

Step 103 collects the user's current facial features; calculates the similarity between the user's current facial features and the makeup of the corresponding part in the target makeup, and adjusts the operation steps about the target makeup that need to be prompted in real time according to the similarity

Step 104 confirms whether the operation steps are prompted to complete, and performs corresponding operations according to the confirmation result:

Collect the current facial features of the user; and

The technical solution of the method embodiment shown in FIG. 1 will be described in detail below using specific embodiments.

About determining the target makeup

The target makeup can be one or more partial facial makeups, such as target eye shadow makeup, target eyeliner makeup, target eyelash makeup, and the like.

The methods for determining the target makeup include an active method and a passive method, wherein the active method is that the user selects a pre-stored makeup as the target makeup. The makeup corresponding to the instruction is used as the target makeup.

The passive method is to automatically match the target makeup for the user when it is determined that the control end of the smart mirror has not received the user's instruction for selecting makeup. For example, after the user activates the corresponding function of the smart mirror (for example, the auxiliary makeup function), if the user does not actively select the target makeup within a set time (such as 30s), that is, the control end of the smart mirror does not receive the target makeup within 30s. When the user selects the makeup instruction, the target makeup is automatically matched for the user.

The above operation of automatically matching the target makeup for the user may be: comparing the current facial features of the user with multiple pre-stored makeups, and using the closest makeup as the target makeup.

For example, FIG. 2A shows a schematic diagram of a process for determining a target eye shadow makeup provided by an embodiment of the present invention. As shown in FIG. 2A , the process for determining the target eye shadow makeup may be:

Step 201: Determine the eye shadow smear boundary;

Step 202 obtains the skin color within the eye shadow smearing boundary;

Step 203 obtains multiple recommended eye shadow makeup colors to be determined;

Step 204 compares the color within the eye shadow smearing boundary with each of the to-be-determined recommended eye shadow makeup colors, and compares the to-be-determined eye shadow corresponding to the highest similarity whose similarity is higher than the second preset similar eye shadow preset value. The identified recommended eyeshadows are used as target eyeshadows.

Specifically, the eye shadow smearing boundary is determined according to the current user's facial features, and the eye shadow smearing boundary can be calculated and determined by a preset algorithm based on the lower edge of the eyebrows of the user and the upper and lower contour edges of the eyes, and the first dotted line 201 shown in FIG. The eye shadow application range (ie, the area S1 within the first dashed line) is determined according to the first dashed line.

In the case of determining the eye shadow application range applicable to the current user, obtain the skin color within the eye shadow application range and multiple pre-stored eye shadow makeup colors, and compare the similarity between the skin color within the eye shadow application range and the multiple eye shadow makeup colors, The eye shadow makeup with the highest similarity to the skin color within the eye shadow application range of the current user is used as the target eye shadow makeup.

FIG. 3A shows a schematic flowchart of determining a target eyeliner makeup provided by an embodiment of the present invention. As shown in FIG. 3A , the process of determining the target eyeliner makeup may be to compare the user’s current facial features with a plurality of pre-stored eyeliner makeups, respectively. Among them, the closest eyeliner makeup is used as the target eyeliner makeup.

Specifically, the inner corner eyeliner, the outer upper corner eyeliner, and the outer lower corner eyeliner are obtained according to the current user facial features, and the target eyeliner makeup is determined according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner, and the outer lower eyeliner eyeliner. The above-mentioned determination of the target eyeliner makeup according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner and the outer lower eyeliner eyeliner includes: if the inner corner eyeliner is not obtained, and the outer lower eyeliner is not obtained, then according to the first A mapping relationship determines the target eyeliner makeup, and the first mapping relationship is the corresponding relationship between the type of the outer upper corner eyeliner and the target eyeliner makeup; if the inner eyeliner is obtained, and the outer lower eyeliner is not obtained, then according to the first The second mapping relationship determines the target eyeliner makeup, and the second mapping relationship is the corresponding relationship between the type of the outer upper corner eyeliner and the target eyeliner makeup; if the inner corner eyeliner is not obtained, and the outer lower eyeliner is obtained, then according to the first Three mapping relationships determine the target eyeliner makeup, and the third mapping relationship is the correspondence between the type of eyeliner in the outer upper corner of the eye and the target eyeliner makeup.

As shown in FIG. 3B , the pre-stored multiple eyeliner makeups may include European and American eyeliners 301 , dog eyeliners 302 , rising eyeliners 303 , drooping eyeliners 304 , scheming eyeliners 305 , and parallel eyeliners 306 .

Determining the target eyeliner makeup according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner, and the outer lower eyeliner eyeliner includes: determining the target eyeliner makeup according to the first mapping relationship shown in FIG. 3C . Specifically, the first mapping is the correspondence between the type of eyeliner in the upper outer corner of the eye and the makeup look of the target eyeliner. Therefore, the target can be determined according to the type of eyeliner in the upper outer corner of the eye when the inner corner eyeliner and the large outer lower corner eyeliner are not obtained. Eyeliner makeup.

Specifically, the type of the outer upper canthus eyeliner can be determined by the direction of the outer upper corner of the eye and the length. In the case that the outer upper corner of the eyeline is upward, the type of the outer upper corner of the eye is determined to be upward, and the upward eyeliner is used as the target eyeliner Makeup; in the case where the outer upper corner eyeliner is downward, determine the type of outer upper corner eyeliner as downward, and take the drooping eyeliner as the target eyeliner makeup; the outer upper corner eyeliner is parallel and the outer upper corner eyeliner is smaller than the first eyeliner. In the case of a set length, determine the type of eyeliner on the outer upper corner of the eye as a parallel segment, and use the parallel eyeliner as the target eyeliner makeup; when the outer upper corner of the eyeliner is smaller than the second set length, determine the type of eyeliner in the outer upper corner of the eye. The eyeliner is extremely short, and the eyeliner 305 is used as the target eyeliner makeup, wherein the second set length is smaller than the first set length.

Then, the target eyeliner makeup can be determined according to the first mapping relationship shown in FIG. 3C , specifically: when the inner corner eyeliner is not obtained and the large outer lower corner eyeliner is not obtained, the upward type outer upper corner eyeliner corresponds to the upward eyeliner 303, The downward type outer upper corner eyeliner corresponds to the drooping eyeliner 304 , the extremely short type outer upper corner eyeliner corresponds to the scheming eyeliner 305 , and the parallel short type outer upper corner eyeliner corresponds to the parallel eyeliner 306 .

Determining the target eyeliner makeup according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner and the outer lower eyeliner eyeliner further includes: determining the target eyeliner makeup according to the second mapping relationship shown in FIG. 3D . The precondition for triggering the second mapping relationship to determine the target eyeliner makeup is that the inner corner eyeliner is acquired, and the outer lower eyeliner is not acquired. Therefore, if the inner corner eyeliner is obtained and the outer lower eyeliner is not obtained, the target eyeliner makeup is determined according to the second mapping relationship, and the second mapping relationship is the correspondence between the type of the outer upper corner eyeliner and the target eyeliner makeup . As shown in FIG. 3D , in the case where the inner canthus eyeliner is obtained and the outer lower eyeliner is not obtained, the upward type outer upper corner eyeliner corresponds to the European and American eyeliner 301 .

Determining the target eyeliner makeup according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner and the outer lower eyeliner eyeliner further includes: determining the target eyeliner makeup according to the third mapping relationship shown in FIG. 3E . The precondition for triggering the third mapping relationship to determine the target eyeliner makeup is that the inner corner eyeliner is not acquired, and the outer lower eyeliner is acquired. Therefore, if the inner corner eyeliner is not obtained, and the outer lower eyeliner is obtained, the target eyeliner makeup is determined according to the third mapping relationship, and the third mapping relationship is the correspondence between the type of the outer upper corner eyeliner and the target eyeliner makeup. As shown in FIG. 3E , in the case where the inner corner eyeliner is not obtained, and the outer lower eyeliner is obtained, the downward type outer upper corner eyeliner corresponds to the dog eyeliner 302 .

It should be noted that the variability of eyelash makeup is low, therefore, only general eyelash makeup and corresponding operation steps can be pre-stored, and when the user actively selects the eyelash makeup, the pre-stored eyelash makeup can be directly used as the target eyelashes. makeup.

Wherein, in the case where the user does not actively select the target makeup, after comparing the similarity between the user's facial features and the pre-stored makeup, the most similar makeup for each part of the face is selected and combined as the target makeup. For example, if the pre-stored makeup includes eyeshadow makeup, eyeliner makeup, and eyelash makeup, if the user does not actively select the target makeup, the eyeshadow makeup and eyeliner makeup with the closest similarity to the user are provided as the target eyeshadow makeup and target eyeliner makeup respectively. User, there is no matching action for eyelash makeup, you can directly provide the pre-stored general eyelash makeup as the target eyelash makeup to the user, and then generate a corresponding query interface to determine whether the user needs the makeup of each part, and then delete the instruction according to the user's instruction. Get the ultimate target makeup look.

Guidance on Achieving Your Target Look

In the case of determining the target makeup, the display screen based on the smart mirror prompts the user about the operation steps of the target makeup, and before the prompt action of the operation step is completed, the following adjustment operations are periodically performed periodically:

Collect the current facial features of the user; and

In the case where the target makeup is the target eye shadow makeup, the user is prompted step by step how to operate, and the user's face can be imaged within a set time (for example, within 5s) after each step is prompted, and the collected user's face Similarity calculation is performed between the eye shadow applied by the user and the target eye shadow makeup in the image, and the subsequent operation steps for the target makeup that need to be prompted are adjusted in real time according to the similarity.

Specifically, the similarity calculation process may include:

The step of determining the skin color of the face setting area may be the same as the color acquisition action in the above-mentioned determination of the target eye shadow makeup operation, that is, the color of the user's forehead may be acquired.

Determine the eye shadow smear boundary. If the eye shadow smear boundary has been obtained in the previous determination of the target makeup operation, the previously obtained eye shadow smear boundary is directly used. Otherwise, an operation similar to determining the eye shadow smear boundary is performed to obtain the eye shadow smear boundary.

Get the skin color outside the eye shadow smearing boundary. If the deviation between the skin color outside the eye shadow smearing boundary and the skin color in the set area of the face (such as the forehead) is greater than the preset value of the color deviation, it is determined that the eye shadow smearing is out of range, and the The display screen of the smart mirror displays the information about the over-range of eye shadow smear to the user, and prompts the operation steps for repairing the over-range of eye shadow smear.

Wherein, after each color deviation comparison, a makeup result quantification operation is performed. Specifically, as shown in FIG. 4 , the quantification items about eye shadow makeup results include: smear range, color usage, degree of fit with the target eye shadow and eye shadow for both eyes symmetry.

Smear range result quantification

Determine the size of the area where the deviation between the skin color outside the smearing boundary and the skin color of the set area is greater than the preset value of the color deviation, and this area is the smearing over-range area, and perform quantitative numerical processing according to the relationship between the smearing over-range area and the set unit area, Specifically, based on the original value (for example, 40), for the over-range smear per unit area, 1 is deducted from the original value, and finally the quantized value of the smear range result is obtained.

Color is quantified using results

The color distribution range of the target eye shadow makeup is further determined according to the determined eye shadow application range. Since the color distribution of each eye shadow in the multiple pre-stored eye shadows is fixed, it can be determined on the basis of the eye shadow application range as shown in FIG. 2B The second dashed line 202 (used to determine the dark area S2 of the eye shadow) and the third dashed line 203 (used to determine the brightened area S3 of the eye shadow), further, it is determined whether there are light colors and dark colors relative to each other in the skin color within the smearing boundary color, transition color, and brightening color, according to the determination result and based on the preset color use result quantization mechanism, numerical processing is performed on the original value of the color use result, so as to obtain the color use result quantization value.

Quantification of fit and eyeshadow symmetry results

According to the captured user's eye image, determine the eye shadow symmetry of both eyes and the degree of fit with the target eye shadow, and perform numerical processing on the original values of the degree of fit and eye shadow symmetry based on the preset quantification mechanism of the symmetry result of the fit degree , to obtain the quantized value of the fit symmetry result.

Eye shadow quantification results

The comprehensive quantification result of eye shadow is the sum of the above three, that is, eye shadow comprehensive quantization value=smear range result quantization value+color use result quantization value+fit degree symmetry result quantization value.

In the case where the target makeup is eyeliner makeup, the user will be prompted step by step for the operation of how to achieve the target eyeliner makeup, and images of the user's face can be collected within a set time (for example, within 5s) after the completion of each step prompt. Similarity calculation is performed between the eyeliner outlined by the user and the target eyeliner makeup in the obtained face image of the user, and the subsequent operation steps about the target makeup that need to be prompted are adjusted in real time according to the similarity.

Specifically, the similarity calculation process may include:

Obtain the eye image of the user's closed eyes;

determine a fitted curve with respect to the actual eyeliner in the eye image;

The distance between the fitting curve and the actual curve is calculated according to a preset algorithm, and the distance obtained by the calculation represents the smoothness of the actual eyeliner. Wherein, when the distance is greater than the preset value of the line distance, it is determined that the eyeline fluency is low, and the information about the low eyeline fluency is displayed to the user through the display screen of the smart mirror, and the operation steps for improving the eyeline fluency are prompted.

Wherein, after each eyeliner fluency calculation, a result quantification is performed. Specifically, as shown in FIG. 4 , the result quantification items about eyeliner makeup include: the degree of fit with the target eyeliner, the symmetry of the eyeliner in both eyes, and the fluency of the eyeliner. and cleanliness.

Similarly, the above items about eyeliner makeup are numerically processed based on the corresponding mechanism, and the quantified values of each item are summed to obtain the comprehensive quantified value of the eyeliner.

In the case where the target makeup is eyelash makeup, the user is prompted step by step for the operation of how to achieve the target eyelash makeup, and the user's face can be imaged within a set time (for example, within 5s) after the completion of each step prompt, and the collected Similarity calculation is performed between the user-processed eyelashes and the target eyelash makeup in the obtained user face image, and the subsequent operation steps about the target makeup that need to be prompted are adjusted in real time according to the similarity.

Specifically, the similarity calculation process may include:

If any one of the user's eyelash features does not meet the preset conditions of the corresponding eyelash feature, a corresponding operation step for repairing the eyelashes is prompted. For example, when it is determined that there is adhesion of the eyelashes, the user is notified of the result through the display screen, and the user is prompted how to repair the eyelashes, so as to eliminate the current adhesion problem.

After completing the prompted operation steps about the target eyelash makeup, the user's eyelash makeup is quantified. As shown in FIG. 4 , the quantified items about the eyelash makeup include: eyelash length, eyelash curling degree, eyelash density and The adhesion of eyelashes is numerically processed based on the corresponding mechanism, and the quantified value of each item is summed to obtain the comprehensive quantified value of eyelashes.

The comprehensive quantified value of the user's eye makeup is the sum of the local makeup volume values. For example, if the target makeup selected by the user includes eyeshadow makeup, eyeliner makeup, and eyelash makeup, the comprehensive quantitative value of eye makeup = the comprehensive quantitative value of eyeshadow + Comprehensive quantitative value of eyeliner + comprehensive quantitative value of eyelashes. After calculating and obtaining the comprehensive quantitative value of the user's eye makeup, it can be displayed on the smart mirror to prompt the user for the quantitative value information corresponding to the current makeup result.

In the embodiment, after the smart mirror is activated, the image taken by the camera on the user's head can be displayed in real time in the corresponding area on the display screen, and specifically, the user's face can be displayed in close-up. The AR technology superimposes the operation demonstration screen on the real-time shooting screen of the user's face, so that the user can intuitively feel the operation steps about the target makeup and enhance the user's experience. Among them, the real-time shooting picture of the user's face can be processed correspondingly to avoid the corresponding occlusion problem. For example, when the user holds the eyeliner and outlines the eyeliner according to the prompted operation steps, the eyeliner often occurs. The problem of blocking the eye picture, so , when such or similar problems occur, the occluder can be hidden to prevent the occlusion problem from affecting the user's view of the overall effect.

FIG. 5 shows a schematic structural diagram of an interactive device based on a smart mirror provided by an embodiment of the present invention. As shown in FIG. 5 , the device may include a processor 10, a memory 20, and a camera 30, where the camera 30 is used to monitor the user's head. The part is photographed to collect the current facial information of the user, and the memory 20 is used to store at least one instruction, and the instruction is loaded and executed by the processor 10 to realize the following interaction method based on a smart mirror:

Prompt steps for the target makeup look; and

Collect the current facial features of the user; and

Further, the determined target makeup includes:

Determine whether a selection makeup instruction is received;

Determine the eye shadow smear boundary;

obtaining the skin color within the boundaries of the eye shadow application;

Get multiple pre-stored eye shadow makeup colors;

Trigger the operation steps for prompting the target eye shadow makeup.

Trigger to prompt the operation steps about the target eyeliner makeup.

Determine the skin color of the face setting area;

Determine the eye shadow smear boundary;

Obtain the eye image of the user with their eyes closed;

determining a fitted curve for the actual eyeliner in the eye image;

Further, the method also includes:

An embodiment of the present invention further provides a smart mirror, which includes the above-mentioned smart mirror-based interaction device and a display screen, and the display screen is connected to the interaction device.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above may refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided by the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined. Either it can be integrated into another system, or some features can be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware, or may be implemented in the form of hardware plus software functional units.

The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (Processor) to execute the methods described in the various embodiments of the present invention. some steps. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by program instructions related to hardware. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps including the above method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place , or distributed over at least two network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. Scope.

Claims

An interaction method, characterized in that the method comprises:

determining a target makeup, where the target makeup is one or more partial facial makeup;

Prompt steps for the target makeup look; and

Before the prompt action of the operation step is completed, periodically perform the following adjustment operations:

Collect the current facial features of the user; and

Calculate the similarity between the current facial feature of the user and the makeup of the corresponding part of the target makeup, and adjust the operation steps about the target makeup that need to be prompted in real time according to the similarity.
The method according to claim 1, wherein the determining the target makeup comprises:

Determine whether a selection makeup instruction is received;

Wherein, in the case of receiving the selection makeup instruction, the selected makeup is used as the target makeup;

In the case of not receiving the instruction to select makeup, perform the following operations:

The current facial features of the user are compared with multiple pre-stored makeup looks, and the closest makeup look is used as the target makeup look.
The method according to claim 2, wherein the step of comparing the current facial features of the user with a plurality of pre-stored makeup looks, and using the closest makeup look as the target makeup look comprises:

Determine the eye shadow smear boundary;

obtaining the skin color within the boundaries of the eye shadow application;

Get multiple pre-stored eye shadow makeup colors;

Comparing the color within the eye shadow smearing boundary with the color of each of the pre-stored eye shadow makeup looks respectively, and using the pre-stored eye shadow makeup look corresponding to the highest similarity whose similarity is higher than the eye shadow color preset value as the target eye shadow makeup look; and

Trigger the operation steps for prompting the target eye shadow makeup.
The method according to claim 2, wherein the step of comparing the current facial features of the user with a plurality of pre-stored makeup looks, and using the closest makeup look as the target makeup look comprises:

acquiring the inner corner eyeliner, the outer upper corner eyeliner and the outer lower corner eyeliner, and determining the target eyeliner makeup according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner and the outer lower corner eyeliner; and

Trigger to prompt the operation steps about the target eyeliner makeup.
The method according to claim 4, wherein the determining the target eyeliner makeup according to the eyeliner types of the inner corner eyeliner, the outer upper corner eyeliner and the outer lower eyeliner eyeliner comprises:

If the inner corner eyeliner is not obtained and the outer lower eyeliner is not obtained, the target eyeliner makeup is determined according to the first mapping relationship, and the first mapping relationship is the correspondence between the type of the outer upper corner eyeliner and the target eyeliner makeup ;

If the inner corner eyeliner is obtained and the outer lower eyeliner is not obtained, then the target eyeliner makeup is determined according to a second mapping relationship, and the second mapping relationship is the correspondence between the type of the outer upper corner eyeliner and the target eyeliner makeup;

If the inner corner eyeliner is not acquired, and the outer lower corner eyeliner is acquired, the target eyeliner makeup is determined according to a third mapping relationship, and the third mapping relationship is the correspondence between the type of the outer upper corner eyeliner and the target eyeliner makeup.
The method according to claim 1, characterized in that, calculating the similarity between the current facial feature of the user and the makeup of the corresponding part in the target makeup, and adjusting the follow-up prompts about the target in real time according to the similarity. The process of makeup operation steps includes:

Determine the skin color of the face setting area;

Determine the eye shadow smear boundary;

Obtain the skin color outside the eye shadow smearing boundary. If the deviation between the skin color outside the eye shadow smearing boundary and the skin color of the face setting area is greater than the preset value of the color deviation, a prompt about repairing the eye shadow smearing out of bounds operation is displayed. step.
The method according to claim 1, characterized in that, calculating the similarity between the current facial feature of the user and the makeup of the corresponding part in the target makeup, and adjusting the follow-up prompts about the target in real time according to the similarity. The process of makeup operation steps includes:

Obtain the eye image of the user with their eyes closed;

determining a fitted curve for the actual eyeliner in the eye image;

Determine the distance between the actual eyeliner and the fitting curve, and if the distance is greater than the preset value of the line distance, an operation step for improving the smoothness of the eyeliner is prompted.
The method according to claim 1, characterized in that, calculating the similarity between the current facial feature of the user and the makeup of the corresponding part in the target makeup, and adjusting the follow-up prompts about the target in real time according to the similarity. The process of makeup operation steps includes:

Determine one or more of the following user's eyelash characteristics when the user's eyes are closed: eyelash length, eyelash curling degree, eyelash density, and eyelash adhesion;

If any one of the user's eyelash features does not meet the preset conditions of the corresponding eyelash feature, a corresponding operation step for repairing the eyelashes is prompted.
The method according to claim 1, wherein the method further comprises:

After the prompt action of the operation step is completed, the similarity between the current facial feature of the user and the makeup look of the corresponding part in the target makeup look is converted into a corresponding numerical value, and the display terminal is controlled to display the corresponding numerical value.
An interactive device, characterized in that the device comprises:

A processor, a memory and a camera, the camera is used to collect the current facial information of the user, the memory is used to store at least one instruction, the instruction is loaded and executed by the processor to achieve any one of claims 1-9 A described interaction method.
A smart mirror, characterized in that the smart mirror comprises:

The interactive device of claim 10; and

A display screen, the display screen is connected with the interaction device.