JP7408562B2 - Program, information processing device, quantification method, and information processing system - Google Patents

Description

The present invention relates to a program, an information processing device, a quantification method, and an information processing system.

It is known that the finish of makeup varies depending on the cosmetics used and makeup techniques. Conventionally, many makeup users have had few opportunities to see other people's makeup techniques. In recent years, many videos of people applying makeup (makeup videos) have been posted on websites such as video sites. Such makeup videos have increased opportunities for users to check makeup techniques (for example, see Non-Patent Document 1).

"Lessons with videos", [online], Shiseido Co., Ltd., [searched on April 20, 2018], Internet <URL: https://www.shiseido.co.jp/beauty/dictionary/lesson/index.html 〉

By viewing the makeup video described above, the user can see the cosmetics used by the person in the makeup video and the makeup actions. However, it has been difficult for users to skillfully imitate the makeup actions of a person in a makeup video just by looking at them. Note that if makeup actions can be quantified from makeup videos, it will be convenient because it will be easy to compare the makeup actions of the imitated user and the makeup actions of the person in the makeup video.

An object of an embodiment of the present invention is to provide a program that can quantify makeup actions of a person appearing in video data.

In order to solve the above-mentioned problems, one embodiment of the present invention is a program for causing a computer to function to analyze the makeup movements of a person appearing in makeup or skin care video data , a first detection means for detecting a face area in which the person's face is captured from video data; a second detection unit for detecting a hand area in which the person's hand is captured from the video data; and the detected face area and the hand area. Based on the coordinates and speed of the hand of the person appearing in the video data acquired from the movements of the face area and the hand area, and the coordinates and speed of the face, the fineness of the hand movement and the coordination of the hand and elbow are determined. The present invention is characterized in that it is a program for functioning as an output means for quantifying and calculating the fineness of hand movements and facial movements, and outputting the results .

According to an embodiment of the present invention, it is possible to quantify makeup actions of a person appearing in video data.

FIG. 1 is a configuration diagram of an example of an information processing system according to an embodiment. FIG. 1 is a configuration diagram of an example of an information processing system according to an embodiment. FIG. 1 is a hardware configuration diagram of an example of a computer according to the present embodiment. It is a scatter diagram of an example of a measurement result. It is a scatter diagram of an example of a measurement result. FIG. 1 is a functional block diagram of an example of an information processing system according to the present embodiment. FIG. 3 is a configuration diagram of an example of a region detection section. FIG. 3 is an image diagram of an example of a process for detecting a face area of a person in a frame image. 3 is a flowchart of an example of facial feature point detection processing. FIG. 3 is an image diagram of an example of a process for detecting a hand region of a person in a frame image. FIG. 3 is a configuration diagram of an example of a region detection section. FIG. 6 is an image diagram of an example of processing by a face area detection unit and a hand area detection unit. FIG. 3 is an image diagram of an example of processing by a force field calculation unit and a channel calculation unit.

Next, embodiments of the present invention will be described in detail.

[First embodiment]
<System configuration>
1A and 1B are configuration diagrams of an example of an information processing system according to this embodiment. The information processing system in FIG. 1A includes a single information processing device 1. The information processing system shown in FIG. The information processing device 1 is a PC operated by a user, a smartphone, a tablet, a dedicated device for quantifying makeup actions for home use or business use, or the like.

Further, in the information processing system shown in FIG. 1B, one or more client terminals 2 and a server device 3 are connected via a network 4 such as the Internet. The client terminal 2 is a terminal device such as a PC, a smartphone, or a tablet operated by a user, or a dedicated device for quantifying makeup actions for home or business use. The server device 3 performs processing related to quantifying makeup actions performed on the client terminal 2.

In this way, the present invention is applicable not only to the client-server type information processing system shown in FIG. 1B but also to the single information processing device 1 shown in FIG. 1A. Note that the information processing systems shown in FIGS. 1A and 1B are merely examples, and it goes without saying that there are various system configuration examples depending on the usage and purpose. For example, the server device 3 in FIG. 1B may be configured to be distributed among multiple computers.

<Hardware configuration>
The information processing device 1, client terminal 2, and server device 3 in FIGS. 1A and 1B are realized, for example, by a computer with a hardware configuration as shown in FIG. 2. FIG. 2 is a hardware configuration diagram of an example of a computer according to this embodiment.

The computer in FIG. 2 includes an input device 501, an output device 502, an external I/F 503, a RAM 504, a ROM 505, a CPU 506, a communication I/F 507, an HDD 508, and the like, each of which is interconnected by a bus B.

The input device 501 is a keyboard, mouse, or the like used for input. The output device 502 includes a display such as a liquid crystal or organic EL that displays a screen, a speaker that outputs sound data such as voice and music, and the like. The communication I/F 507 is an interface that connects the computer to the network 4. The HDD 508 is an example of a nonvolatile storage device that stores programs and data.

External I/F 503 is an interface with an external device. The computer can read from and/or write to the recording medium 503a via the external I/F 503. The recording medium 503a includes a DVD, an SD memory card, a USB memory, and the like.

The CPU 506 is an arithmetic unit that implements control and functions of the entire computer by reading programs and data from a storage device such as the ROM 505 and the HDD 508 onto the RAM 504 and executing processing. The information processing device 1, client terminal 2, and server device 3 according to this embodiment can implement various functions by executing programs on a computer with the above-described hardware configuration.

Note that the hardware configuration shown in FIG. 2 is one example, and it goes without saying that there are various configuration examples depending on the usage and purpose. For example, the computer in FIG. 2 may have the input device 501 have a camera function that can shoot moving images.

<Study on quantifying makeup movements>
As a method for quantifying makeup actions, for example, there is a method using a sensor. In the method of quantifying makeup movements using sensors, the person applying makeup is asked to wear a sensor and perform the makeup movements. In the method of quantifying makeup actions using sensors, it is possible to quantify the makeup actions of a person based on the data output from the sensor, but it is not possible to quantify the makeup actions of a person in a makeup video that has already been filmed.

If makeup videos that have already been filmed can be analyzed and the makeup movements of the people in those videos can be quantified, makeup videos from websites such as video sites can be used, and there is no need to wear sensors, etc., so it can be done naturally. We can expect to be able to quantify makeup movements. Therefore, in the present embodiment, in order to quantify the makeup actions of people in the makeup videos that have already been shot, we investigated the object to be acquired from the makeup video and the quantification of the makeup actions using that object.

《Understanding what to obtain from makeup videos》
In order to understand what was captured from the makeup videos, we used motion capture to measure the subjects' movements while applying makeup. The measurement sites were a total of six locations: the tip of the middle finger of the right hand, the base of the middle finger, the center of the back of the hand, the center of the wrist, the elbow, and the forehead. The objects of analysis are the coordinates (displacement), velocity, acceleration, and angular velocity of the measurement site. The principal components of the subject's movements analyzed in this way can be estimated to be the main movement elements during the makeup action.

FIGS. 3A and 3B are scatter diagrams of examples of measurement results. FIG. 3A is a scatter diagram illustrating an example of the fineness of hand movements and the coordination of hands and elbows for each makeup action. FIG. 3B is a scatter diagram showing an example of the fineness of hand movements and facial movements for each makeup action. As shown in FIGS. 3A and 3B, the main components of the subject's movements were the fineness of the hand movements, the coordination of the hands and elbows, and the movements of the face. Therefore, in this embodiment, the objects to be acquired from the makeup video are the coordinates and speed of the hand and the coordinates and speed of the face.

《Quantification of makeup movements using acquired objects》
Image recognition using a convolutional neural network (hereinafter referred to as CNN) is a method for acquiring the coordinates and speed of the hands and the coordinates and speed of the face of a person in a makeup video during the makeup action. Image recognition using CNN can detect face areas and hand areas from two-dimensional images, so by tracking the face and hand areas detected from frame images of makeup videos, it is possible to detect facial and hand areas from two-dimensional images. It is possible to obtain the coordinates and speed of the hand and the coordinates and speed of the face while the person in the photo is applying makeup. Note that details of image recognition using CNN of this embodiment will be described later.

<Software configuration>
《Functional block》
The software configuration of the information processing system according to this embodiment will be described. Note that the information processing device 1 shown in FIG. 1A will be described here as an example. FIG. 4 is a functional block diagram of an example of an information processing system according to this embodiment. The information processing device 1 implements an operation reception section 10, an area detection section 12, a quantification section 14, a post-processing section 16, and a video data storage section 18 by executing programs.

The operation reception unit 10 receives various operations from the user. The video data storage unit 18 stores makeup videos. Note that the video data storage section 18 may be provided outside the information processing device 1. The area detection unit 12 receives input of a makeup video stored in the video data storage unit 18 or a makeup video captured by the camera function. The area detection unit 12 detects, for each frame image constituting the input makeup video, a face area and a hand area of a person appearing in the frame image, as will be described later.

The quantification unit 14 acquires the coordinates and speed of the hand of the person in the makeup video and the coordinates and speed of the face during the makeup movement of the person from the face area and hand area detected by the area detection unit 12. Quantifying the makeup movements of people in videos. The post-processing unit 16 post-processes the processing results by the area detection unit 12 and the quantification unit 14 and outputs the post-processed results to the output device 502 or the like.

For example, the post-processing unit 16 performs post-processing of enclosing a face area and a hand area of a person appearing in a makeup video in a rectangle while the person is applying makeup. Additionally, the post-processing unit 16 calculates the fineness of hand movements, the coordination between hands and elbows, and the movement of the face from the coordinates and speed of the hands of the person in the makeup video and the coordinates and speed of the face. Perform post-processing such as visual representation.

Further, the post-processing unit 16 can output the comparison result by quantifying and comparing the makeup actions of the people in the two makeup videos. For example, a user who uses the information processing system according to the present embodiment can quantify and compare his or her makeup actions with those of a makeup artist or other person who is skilled at applying makeup, thereby identifying the differences between his or her own makeup techniques. It becomes easier to understand. Thereby, the information processing system according to this embodiment can provide a service that improves the user's makeup technique.

The area detection unit 12 shown in FIG. 4, which detects a face area and a hand area of a person in a frame image of a makeup video, is configured as shown in FIG. 5, for example. FIG. 5 is a configuration diagram of an example of the area detection section. The area detection unit 12 in FIG. 5 has a configuration including a framing unit 20, a face area detection unit 22, a hand area detection unit 24, and a facial feature point detection unit 26.

The framing unit 20 provides the input makeup video to the face area detection unit 22 and the hand area detection unit 24 in units of frame images. The face area detection unit 22 has a face area learning model including a face part area learning model.

Note that the face area learning model possessed by the face area detection unit 22 is created by machine learning using a two-dimensional image in which the hand area overlaps the face area as training data. Teacher data in which the hand region overlaps the face region is created from a two-dimensional image in which the hand is in the foreground of the face. The training data in which the hand region overlaps the face region is an annotated (created as training data) hand region training dataset, and the image of the annotated face region training dataset is changed so that the hand is in the foreground. It can also be created by pasting.

The face area detection unit 22 uses a face area learning model learned by performing CNN using training data in which a part of the face area is hidden by the hand area (occlusion environment) as a learning data set. Achieving robust face area detection against overlap between face and hand areas.

The hand region detection unit 24 has a hand region learning model including a fingertip position region learning model. The hand region learning model possessed by the hand region detection unit 24 is created using a two-dimensional image of a hand applying makeup as training data.

The training data for hands while applying makeup includes an annotated hand region learning dataset that specializes in the shape of hands while applying makeup, and an annotated fingertip position area learning dataset that specializes in the position of fingertips while applying makeup. Created by The hand area detection unit 24 detects the positions of hands and fingertips that have many variations in shape during makeup movements by using a hand area learning model learned by performing CNN using the above-mentioned teacher data as a learning data set. Achieve highly accurate hand region detection.

Further, the facial feature point detection unit 26 has a facial feature point learning model including a facial part feature point learning model. The facial feature point detection unit 26 detects facial feature points of the entire face by using a facial feature point learning model. Thereafter, the facial feature point detection unit 26 detects facial feature points of facial parts for each region by using the facial part feature point learning model. The facial feature point detection unit 26 detects the facial feature points of the eyes included in the facial parts, and corrects the positions of the facial feature points of parts other than the eyes (including the contour) from the positions of the facial feature points of the eyes. , achieves highly accurate facial feature point detection even in low resolution or occluded environments.

<Processing>
《Face area detection and facial feature point detection》
The process by which the area detection unit 12 detects a face area of a person appearing in a frame image is performed as shown in FIG. 6, for example. FIG. 6 is an image diagram of an example of a process for detecting a face area of a person in a frame image.

The face area detection unit 22 of the area detection unit 12 uses the above-described face area learning model to detect a face area in which a person's face appears in the frame image 1000 as a rectangle 1002. The face area detection unit 22 detects a facial part area from the area of the rectangle 1002 using the above-described facial part area learning model, and corrects the rectangular ratio of the rectangle 1002 to a rectangle 1004 with the detected nose as the center.

The facial feature point detection unit 26 detects facial feature points from the rectangular area 1004 as in the rectangular area image 1006 using the above-described facial feature point learning model. Further, the facial feature point detection unit 26 detects feature points of facial parts for each region from the rectangular area image 1006 as in the rectangular area image 1008 using the above-described facial part feature point learning model.

Note that the area detection unit 12 in FIG. 5 can realize highly accurate facial feature point detection even in a low resolution or shielded environment by performing processing as shown in the flowchart in FIG. FIG. 7 is a flowchart of an example of facial feature point detection processing.

In step S11, the facial feature point detecting unit 26 of the area detecting unit 12 detects facial feature points from the facial area (the entire face) detected by the facial area detecting unit 22 using the above-described facial feature point learning model. , estimate the head pose.

Proceeding to step S12, the facial feature point detection unit 26 takes into consideration the head posture estimated in step S11, and detects the eyes detected by the face area detection unit 22 using the facial part feature point learning model. Correct the position estimation of the eye and improve the accuracy of the eye position estimation.

Proceeding to step S13, the facial feature point detection unit 26 takes into account the estimated positions of the eyes corrected in step S12 and uses the above-described facial part feature point learning model to determine the feature points (including contours) of facial parts other than the eyes. Detects and corrects the estimated positions of facial parts other than eyes. The process in the flowchart of FIG. 7 is effective, for example, when the outline of the face is obscured by a hand.

《Hand area detection》
The process by which the area detection unit 12 detects a hand area of a person appearing in a frame image is performed, for example, as shown in FIG. 8. FIG. 8 is an image diagram of an example of a process for detecting a hand region of a person in a frame image.

The hand area detection unit 24 of the area detection unit 12 uses the above-described hand area learning model to detect a hand area in which the left and right hands of the person in the frame image 1100 are captured in a rectangle 1102. Further, the hand area detection unit 24 uses the above-described fingertip position area learning model to determine the areas 1112 and 1114 of the left and right hands of the person in the frame image 1110, the fingertip positions 1116 and 1116 of the left and right hands from the area of the rectangle 1102. 1118 is detected.

"output"
The information processing device 1 according to the present embodiment calculates, for example, the fineness of hand movements, the coordination between hands and elbows, based on the coordinates and speed of the hands of the person in the makeup video and the coordinates and speed of the face during the makeup movement. , can calculate and output facial movements. The fineness of hand movements, the coordination of hands and elbows, and the output of facial movements are useful for research on makeup movements.

Furthermore, since the information processing device 1 according to the present embodiment can quantify and compare the makeup actions of people in two makeup videos, it is possible to compare the makeup actions of a user who wants to learn makeup techniques with the makeup actions of a person who is good at makeup, such as a makeup artist. You can quantify and compare makeup actions. The comparison result may be presented to the user as a score, or the difference in makeup action between the user and a person who is good at makeup, such as a makeup artist, may be visually presented to the user. Further, based on the comparison result, the information processing device 1 according to the present embodiment may present a makeup technique to the user so that the user's makeup behavior approaches that of a makeup artist or other person skilled in makeup.

For example, the information processing device 1 according to the present embodiment can provide a user with a lecture on the application area determination and application range when applying a makeup product such as blush/foundation over a wide area. Further, the information processing device 1 according to the present embodiment can judge whether the movements are correct or incorrect when using a makeup product such as eyeliner/eyeshadow/concealer that requires a difficult technique and give a lecture to the user. Furthermore, the information processing device 1 according to the present embodiment can also give a lecture to the user on how to apply hair wax (hair product), a lecture on how to apply a skin care product, or a lecture on a massage method.

In addition, the information processing device 1 according to the present embodiment may apply makeup in a manner that suits the user's facial features, since users with sharp bones and users with round bones may apply makeup in different ways, such as how to apply blush. You can also give recommendations and lectures on techniques to achieve the makeup.

[Second embodiment]
The information processing system according to the second embodiment is the same as the information processing system according to the first embodiment except for some parts, so the description will be omitted as appropriate. The information processing system of the second embodiment has a configuration including the area detection section 12 shown in FIG. 9 in place of the area detection section 12 shown in FIG. FIG. 9 is a configuration diagram of an example of the area detection section. The region detection section 12 in FIG. 9 includes a framing section 50, a skin color region extraction section 52, a region division section 54, a face region detection section 56, a hand region detection section 58, a force field calculation section 60, and a channel calculation section 62. It is.

The framing unit 50 provides the input makeup video to the skin color area extraction unit 52 in units of frame images. The skin color area extraction unit 52 extracts a skin color area from the frame image. The region dividing section 54 divides the skin color region extracted by the skin color region extracting section 52 into candidate blobs, and further performs labeling of the candidate blobs. The region dividing section 54 provides the labeled candidate blobs to the face region detecting section 56 and the hand region detecting section 58 .

The face area detection unit 56 classifies candidate blobs of the face area (detects the face area) based on the provided label of the candidate blob (features of the divided skin color areas). Furthermore, based on the label of the candidate blob (features of the divided skin color region) provided from the region dividing section 54 and the candidate blob of the face region classified by the face region detecting section 56, the hand region detecting section 58 performs the following operations. Classify hand region candidate blobs (detect hand region).

As shown in FIG. 10, the face area detection unit 56 and the hand area detection unit 58 in FIG. 9 first classify candidate blobs of the face area, exclude candidate blobs of the face area, The hand region detection unit 58 classifies candidate blobs of the hand region. Therefore, in this embodiment, erroneous detection of the hand area can be prevented.

If the face area detection unit 56 is unable to classify the candidate blob of the face area, the force field calculation unit 60 determines that interference between the face area and the hand area has occurred, and performs the following processing. The force field calculation unit 60 is provided with candidate blobs of the face region and hand region of the previous frame (t-1) and labeled candidate blobs of the current frame (t).

As shown in FIG. 11, the force field calculation unit 60 sets a large number of channels within the candidate blob image using a force field. The channel calculation unit 62 calculates the movement distance from the previous frame for each channel, and assumes that the candidate blob of the channel with a large movement distance is the candidate blob of the hand that is moving. Can classify candidate blobs.

Note that the force field calculation unit 60 and channel calculation unit 62 of the region detection unit 12 cluster the movements that are typical of the hand region in addition to the size of the movement distance, thereby preventing false detection of candidate blobs in the hand region and face region. can be further prevented.

(summary)
As described above, according to the present embodiment, it is possible to quantify the makeup actions of a person in a shot makeup video without wearing a sensor or the like, and it is possible to provide and teach makeup techniques. The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, in this embodiment, two-dimensional video data has been described as an example, but three-dimensional video data may also be used. According to this embodiment, it is possible to quantify makeup movements of a person in 3D video data through data analysis similar to 2D video data or analysis that combines 2D video data analysis with 3D information. , it becomes possible to provide and teach makeup techniques.

Although the present invention has been described above based on examples, the present invention is not limited to the above-mentioned examples, and various modifications can be made within the scope of the claims. This application claims priority to Basic Application No. 2018-199739 filed with the Japan Patent Office on October 24, 2018, and the entire contents thereof are incorporated herein by reference.

1 Information processing device 2 Client terminal 3 Server device 4 Network 10 Operation reception unit 12 Area detection unit 14 Quantification unit 16 Post-processing unit 18 Video data storage unit 20 Framing unit 22 Face area detection unit 24 Hand area detection unit 26 Facial features Point detection section 50 Framing section 52 Skin color region extraction section 54 Region division section 56 Face region detection section 58 Hand region detection section 60 Force field calculation section 62 Channel calculation section

Claims (10)

A program for operating a computer to analyze the makeup movements of a person in makeup or skin care video data,
The computer,
a first detection means for detecting a face area in which the person's face is captured from the video data;
a second detection means for detecting a hand region in which the person's hand is captured from the video data;
Based on the detected face area and the hand area, the hand movement is determined from the hand coordinates and speed of the person appearing in the video data acquired from the movements of the face area and the hand area, and the coordinates and speed of the face during the makeup movement. output means for quantifying and outputting the fineness of hand and elbow coordination, or the fineness of hand movements and facial movements;
A program to function as The first detection means includes:
face area detection means for detecting the face area of the person from the video data using a face area learning model;
facial part area detection means for detecting a facial part area from the facial area using a facial part feature point learning model;
has
The second detection means includes:
hand area detection means for detecting the hand area of the person from the video data using the hand area learning model of the makeup action;
Fingertip position area detection means for detecting a fingertip position area from the hand area using the fingertip position area learning model of the makeup action;
2. The program according to claim 1, comprising: 3. The computer program product according to claim 2 , wherein the facial area learning model is a convolutional neural network trained using teacher data in which a part of the facial area is hidden by a hand area as a facial area learning data set. The hand region learning model is a convolutional neural network that is trained using training data of the shape of a hand while applying makeup as a hand region learning dataset, and the fingertip position region learning model is trained using training data of the position of a fingertip while applying makeup. 3. The program according to claim 2 , wherein the program is a convolutional neural network trained using a fingertip position region learning data set. The computer,
extraction means for extracting a skin color region from the video data;
dividing means for dividing the skin color region into divided regions;
further function as
The first detection means detects a face area in which the face of the person is reflected from the skin color area based on the feature amount of the divided area,
The second detection means detects a hand region in which the person's hand is captured from the skin color region excluding the skin color region detected as a face region in which the person's face is captured, based on the feature amount of the divided region. The program according to claim 1, characterized in that: The computer,
When the first detection means cannot detect a face area in which the person's face is reflected from the skin color area, the first detection means detects the movement distance of the divided area between frame images forming the video data, and detects the movement distance of the divided area between the frame images forming the video data. a third detection means that assumes that the divided area is a hand area in which the person's hand is captured;
6. The program according to claim 5 , further functioning as a program. The output means outputs an image visually representing a difference in the makeup actions of the people shown in the two video data, based on a comparison result of an output calculated by quantifying the makeup actions of the people shown in the two video data. 2. The program according to claim 1, wherein the program outputs an output. An information processing device that analyzes makeup movements of a person appearing in makeup or skin care video data, the information processing device comprising:
a first detection means for detecting a facial area in which the person's face is captured from the video data;
a second detection means for detecting a hand region in which the person's hand is captured from the video data;
Based on the detected face area and the hand area, the hand movement is determined from the hand coordinates and speed of the person appearing in the video data acquired from the movements of the face area and the hand area, and the coordinates and speed of the face during the makeup movement. output means for quantifying and outputting the fineness of the hand and the hand-elbow coordination, or the fineness of the hand movement and the facial movement;
An information processing device having: A quantification method executed in an information processing device that analyzes makeup movements of a person in makeup or skin care video data, the method comprising:
a first detection step of detecting a facial area in which the person's face is captured from the video data;
a second detection step of detecting a hand region in which the person's hand is captured from the video data;
Based on the detected face area and the hand area, the hand movement is determined from the hand coordinates and speed of the person appearing in the video data acquired from the movements of the face area and the hand area, and the coordinates and speed of the face during the makeup movement. an output step of quantifying and outputting the fineness of the hand and the coordination of the hand and the elbow, or the fineness of the hand movement and the movement of the face;
A quantification method with An information processing system comprising: a client terminal that receives operations from a user; and a server device that analyzes makeup actions of a person in makeup or skin care video data based on the operations that the client terminal receives from the user. The server device includes:
receiving means for receiving information on an operation that the client terminal has accepted from the user;
a first detection means for detecting a facial area in which the person's face is captured from the video data based on an operation received from the user;
a second detection means for detecting a hand region in which the person's hand is captured from the video data based on an operation received from the user;
Based on the detected face area and the hand area, the hand movement is determined from the hand coordinates and speed of the person appearing in the video data acquired from the movements of the face area and the hand area, and the coordinates and speed of the face during the makeup movement. output means for quantifying and outputting the fineness of the hand and the hand-elbow coordination, or the fineness of the hand movement and the facial movement;
transmitting means for transmitting the output of the output means to the client terminal;
An information processing system with

Images