JP2019000132A - Makeup support system, makeup support method, and makeup support model - Google Patents

Abstract

To provide a makeup support system, a makeup support method, and a makeup support model that facilitate proposal of makeup that satisfies a user.SOLUTION: A makeup support system 10 includes an image input part (a terminal device 30) for receiving input of an input image, which is a digital picture of a predetermined part of a living body to make up, and a generation part 40. The generation part 40 corrects a pixel value of the input image so that coordinates of a feature amount derived from the pixel value of the input image fall within a specified range in a feature amount space, and generates a normalized image indicating a reference state of the predetermined part from the input image.SELECTED DRAWING: Figure 1

Description

  The present invention generally relates to a makeup support system, a makeup support method, and a makeup support model, and more particularly to a makeup support system, a makeup support method, and a makeup used for assisting a user in determining makeup. Up support model.

  Patent document 1 discloses a skin condition measurement analysis information management system. The skin condition measurement analysis information management system of Patent Literature 1 can provide a user who can obtain an analysis result of his / her skin condition from measurement data of the skin condition measurement apparatus to the user of the skin condition measurement apparatus. Further, the skin condition measurement analysis information management system of Patent Document 1 can provide collected data to a cosmetic manufacturer or sales company (cosmetic company).

Japanese Patent Laid-Open No. 2006-115363

  According to the skin condition measurement analysis information management system of Patent Document 1, it is possible to collect and use a lot of skin condition measurement data. However, about the makeup suitable for the user (customer), the staff of the store (salesperson, beauty staff) needs to determine. For this reason, whether or not the user can propose satisfactory makeup depends greatly on the individual skills of the staff.

  An object of the present invention is to propose a makeup support system, a makeup support method, and a makeup support model that make it easy to propose makeup that satisfies a user.

  A makeup support system according to an aspect of the present invention includes an image input unit that receives an input of an input image that is a digital photograph of a predetermined part of a human body to be makeup, and a generation unit. The generation unit corrects the pixel value of the input image so that the coordinates of the feature amount derived from the pixel value of the input image are within a specified range in the feature amount space, and the predetermined portion is obtained from the input image. The normalized image indicating the reference state is generated.

  The makeup support method according to one aspect of the present invention includes a first step and a second step. In the first step, input of an input image that is a digital photograph of a predetermined part of a human body to be made up is received. In the second step, the pixel value of the input image is corrected so that the coordinates of the feature amount derived from the pixel value of the input image are within a specified range in the feature amount space, and the predetermined portion is extracted from the input image. A normalized image indicating the reference state is generated.

  The makeup support model of one aspect according to the present invention is a learned model in which a relationship between an input image and a normalized image is learned by learning data including a relationship between the input image and the normalized image for a plurality of persons. including. The input image is a digital photograph of a predetermined part of a human body to be made up. The normalized image is obtained by correcting the pixel value of the input image so that the coordinates of the feature amount derived from the pixel value of the input image are within a specified range in the feature amount space. It is an image which shows a state.

  According to the makeup support system, makeup support method, and makeup support model of the aspect of the present invention, there is an effect that it is easy to propose makeup that satisfies the user.

FIG. 1 is a block diagram of a makeup support system according to an embodiment. 2A and 2B are schematic views of a photo selection screen displayed by the makeup support system. FIG. 3 is a schematic diagram of an illustration selection screen displayed in the makeup support system. FIG. 4 is a schematic diagram of a makeup selection screen displayed by the makeup support system. FIG. 5 is a flowchart of a makeup support method executed by the makeup support system.

1. Embodiment 1.1 Overview The makeup support system 10 (see FIG. 1) of the present embodiment is used to propose makeup suitable for a user (customer). As an example, the makeup support system 10 may be used by a store staff (salesperson, beauty staff) to make a makeup proposal that satisfies a user to a user who has visited the store. The makeup satisfied by the user is assumed to be, for example, a makeup that realizes an ideal appearance of the user, a makeup that makes the user feel happy, a makeup that makes the user feel like herself, and the like.

  In the makeup support system 10, a user image is used as user information. The user's image is a digital photograph of a predetermined part of the human body to be made up. The predetermined part may be the whole body or a part of the human body to be made up. Examples of the predetermined part being a part of the human body include the head, hands, and feet. Further, the predetermined part may be, for example, the entire head, or a part of the head (for example, face, head). Further, the predetermined part may be the entire face or a part of the face (for example, eyes, eyebrows, forehead, nose, lips, chin, cheeks). The predetermined portion is often determined by the user in advance, but can be changed by consultation with store staff. The digital photograph of the predetermined part may be a digital photograph of only the predetermined part. However, since it is usually not practical to obtain a digital photograph of only the predetermined part, it may be a digital photograph of the human body including the predetermined part. That's fine. For example, if the predetermined part is the head, the digital photograph of the head may be a digital photograph of the upper body that is a part including the head.

  As described above, there are various types of predetermined parts to be made up. There are various types of makeup, such as face makeup, hair makeup, and eye makeup, in relation to the type of site (predetermined site) to which makeup is applied. In this embodiment, various types of makeup may not be distinguished and may be collectively referred to as makeup. Hair makeup may also include hair arrangements that change hairstyles, haircuts that cut hair, hair coloring that changes hair color, and the like.

  The makeup support system 10 outputs an image (result image) indicating a state of the predetermined part after makeup using the input image (input image) of the predetermined part of the user. The makeup support system 10 uses a makeup pattern that specifies the contents of makeup when generating a result image. The makeup pattern is information that specifies the contents of makeup. For example, makeup content may include information on one or more makeup processes (eg, base, cheek, mouth, eyes, eyebrows, etc.). The information on the makeup process can be composed of information on the makeup item (cosmetics) used in the process and a parameter indicating a use location (for example, a range where the makeup item is applied). Examples of makeup items include makeup base, concealer, foundation, powder, highlighter, teak, eyeliner, eye shadow, mascara, lipstick, gloss, and teak. Thus, the makeup pattern includes parameters indicating information on one or more makeup processes. The makeup pattern is, for example, TPO (for example, business, formal, dress, casual), impression (for example, fair skin, dark black, natural, clean, gentle), fashion (for example, popular actress or model-like makeup) ) And the like can be prepared in various ways. The makeup pattern may be generated with reference to a history of makeup performed by a makeup artist. Therefore, the user selecting a favorite result image is equivalent to the user selecting a favorite makeup pattern.

  The makeup support system 10 outputs information (makeup information) for applying makeup having the content specified by the makeup pattern finally selected by the user. The makeup information is obtained, for example, by converting the contents of makeup represented by parameters in the makeup pattern into expressions (sentences etc.) that can be understood by a person (for example, staff). The staff of the store presents the makeup method and necessary cosmetics to the user with reference to the makeup information output from the makeup support system 10. In this way, by using the makeup support system 10, the staff can easily propose makeup that satisfies the user. In particular, the makeup satisfied by the user varies from user to user, and the user himself / herself is often not clearly grasped. The makeup support system 10 presents the state after makeup of a predetermined part as a specific image called a result image. In this way, the makeup support system 10 can propose an image that specifically shows the state after the makeup of the user when the user has not yet clearly grasped his / her ideal regarding the state after the makeup. To.

  Further, in the makeup support system 10, the result image shows the state of the predetermined part after makeup with an illustration. By using the illustration, for example, a portion that the user does not want to see much at a predetermined portion (for example, a portion where the user feels a complex) can be made inconspicuous. Therefore, a reduction in psychological burden when the user views his / her own image (result image) can be expected. In order to generate such a result image, makeup support system 10 performs an illustration process. The illustration processing may include, for example, image gradation conversion (linear gradation or non-linear correction), image gradation flattening (smoothing), image edge enhancement, and the like. In addition, when the predetermined part includes a face, a process for adjusting the arrangement of facial parts (for example, eyes, eyebrows, nose, mouth) and a process for simplifying the shape of the part may be included. The specific content of the illustration processing is determined according to the touch (taste) of the illustration. Illustration touches vary depending on, for example, combinations of the degree of image simplification, the degree of contour / shape enhancement, the degree of color tone enhancement, and the like.

1.2 Configuration Hereinafter, a specific configuration of makeup support system 10 will be described. As shown in FIG. 1, makeup support system 10 includes one or more cameras 20, one or more terminal devices 30, and a generation unit 40. One or more terminal devices 30 and the generation unit 40 are connected to a network 60 such as the Internet.

  In the makeup support system 10, the generation of the result image is performed by the generation unit 40. The terminal device 30 is used to give a user image (input image), which is information used for generating a result image by the generation unit 40, to the generation unit 40. In the makeup support system 10, a makeup pattern is used to generate a result image. The generation unit 40 also determines or changes the makeup pattern.

  The camera 20 is used, for example, in a store that sells cosmetics. The camera 20 is used for obtaining the input image described above. The camera 20 is a so-called digital camera, for example. In the present embodiment, the input image generated by the camera 20 is a color image.

  The image of the user (predetermined part) captured by the camera 20 is transmitted to the terminal device 30 as an input image. In the present embodiment, the camera 20 is used to obtain a plurality of input images. The plurality of input images are images having different directions in which the predetermined part is imaged. That is, the camera 20 images a predetermined part of the user from various angles. For example, by rotating the camera 20 360 ° around the user in a horizontal plane, a predetermined part (for example, the head) of the user is photographed from all directions of 360 °. Therefore, a plurality of input images obtained by imaging a predetermined part from different directions are output from the camera 20 to the terminal device 30. For example, twelve input images can be obtained by photographing in 30 ° increments from the front of the head.

  The terminal device 30 is used, for example, in a store that sells cosmetics. As illustrated in FIG. 1, the terminal device 30 includes an input unit 31, an output unit 32, a communication unit 33, and a processing unit 34. For example, the terminal device 30 is configured by a computer system having a CPU and a memory as main components, and the computer system functions as the terminal device 30 when the CPU executes a program stored in the memory. The program is recorded in advance in the memory of the terminal device 30 here, but may be provided through a telecommunication line such as the Internet or recorded in a recording medium such as a memory card.

  The input unit 31 has a function of accepting a human operation. The output unit 32 is a thin display device such as a liquid crystal display or an organic EL (electroluminescence) display. Here, the input unit 31 is, for example, a touch pad installed on the display surface of the output unit 32. That is, the output unit 32 and the input unit 31 constitute a touch panel. The input unit 31 is not limited to a touch panel, and may be a keyboard, a pointing device, a mechanical switch, or the like.

  The communication unit 33 is connected to the network 60 to enable communication between the terminal device 30 and the generation unit 40.

  The processing unit 34 has a function of creating a screen to be displayed on the output unit 32 and controlling the output unit 32 to display the created screen on the output unit 32. That is, the terminal device 30 implements a function as the processing unit 34 for controlling the output unit 32 by installing dedicated application software and starting the application software.

  The processing unit 34 is configured to display a photo selection screen 500 as shown in FIGS. 2A and 2B. In the photo selection screen 500, the processing unit 34 displays a predetermined number of input images 510 obtained from the camera 20 (two images in FIGS. 2A and 2B). The processing unit 34 accepts selection of the input image 510 by the input unit 31 while the photo selection screen 500 is displayed. That is, the processing unit 34 enables the input unit 31 to select an input image 510 used for generating a result image from a plurality of input images 510. Thereby, the user can select the input image 510 that he likes most from the plurality of input images 510. The processing unit 34 transmits the input image 510 selected by the input unit 31 to the generation unit 40 through the communication unit 33. The generation unit 40 performs processing for generating a result image based on the input image 510 received from the terminal device 30.

  The processing unit 34 is configured to display an illustration selection screen 600 as shown in FIG. The illustration selection screen 600 is a screen for selecting the illustration image touch of the result image. On the illustration selection screen 600, the processing unit 34 displays a plurality (four in FIG. 3) of sample images 610, 620, 630, and 640 as shown in FIG. The plurality of sample images 610, 620, 630, and 640 each illustrate a touch of a different illustration.

  For example, the sample images 610, 620, 630, and 640 correspond to touches of illustrations having different combinations of the degree of simplification of the image and the degree of contour / shape enhancement / color tone enhancement. The sample images 610 and 620 are less simplified than the sample images 630 and 640 (the illustrations are closer to photographs). The sample images 610 and 630 have lower contour / shape enhancement levels and higher color / tone enhancement levels than the sample images 620 and 640. In particular, the sample image 610 is an image close to a photograph, but wrinkles and fine wrinkles are removed. The sample image 620 is an image close to a caricature, and the contour is emphasized compared to a photograph. The sample image 630 is an image obtained by further simplifying the sample image 620. The sample image 640 is an image obtained by further simplifying the sample image 610.

  Sample images 610, 620, 630, and 640 are generated using a user input image 510. However, it is not always necessary to use the user input image 510, and an image prepared in advance may be used. For example, the sample image may be an image generated using a digital photograph that is different from the user.

  The processing unit 34 accepts selection of the sample images 610, 620, 630, and 640 by the input unit 31 while the illustration selection screen 600 is displayed. That is, the processing unit 34 enables the input unit 31 to select an illustration touch. Thereby, the user can select the touch of the most favorite illustration with reference to the plurality of sample images 610, 620, 630, and 640. The processing unit 34 transmits information on the illustration touch selected by the input unit 31 (result of selection of the illustration touch) to the generation unit 40 through the communication unit 33. The generation unit 40 performs a process of generating a result image based on the illustration touch information received from the terminal device 30.

  The processing unit 34 is configured to display a makeup input screen that accepts input of a makeup pattern. On the makeup input screen, the processing unit 34 displays, for example, a plurality of makeup patterns prepared in advance. In addition, the processing unit 34 accepts selection of a plurality of makeup patterns by the input unit 31. In this way, the processing unit 34 receives an input of a makeup pattern. On the makeup screen, the processing unit 34 accepts direct input of a makeup pattern by the input unit 31. In this case, the processing unit 34 may display an individual change button for individually changing the parameters of the makeup pattern, or may display a batch change button for collectively changing two or more parameters of the makeup pattern. Also good. The batch change button, for example, has a tendency to change the makeup pattern (for example, “smooth”, “gorgeous”, “young”, “adult”, “soft”, “beautiful”, “natural” Etc.) may be displayed.

  The processing unit 34 transmits the makeup pattern input by the input unit 31 to the generation unit 40. On the makeup input screen, the processing unit 34 can simultaneously input a predetermined number (for example, four) of makeup patterns. The generation unit 40 generates a result image from the input image based on the makeup pattern received from the processing unit 34.

  The processing unit 34 is configured to display a makeup selection screen 700 as shown in FIG. On the makeup selection screen 700, the processing unit 34 displays a plurality (four in FIG. 4) of result images 710, 720, 730, and 740 obtained from the generation unit 40. While the makeup selection screen 700 is displayed, the processing unit 34 accepts selection of the result images 710, 720, 730, and 740 by the input unit 31. Therefore, the user can select a favorite result image from the plurality of result images 710, 720, 730, and 740 generated by the generation unit 40.

  When any of the plurality of result images 710, 720, 730, and 740 is selected by the input unit 31, the processing unit 34 generates makeup information based on the selected result image, or the selected result The selection of whether to generate another result image based on the image is accepted. Here, when it is selected to generate another result image based on the selected result image, the processing unit 34 displays a makeup input screen. In this case, the processing unit 34 may display a makeup pattern corresponding to the selected result image in advance on the makeup input screen. On the other hand, when generation of makeup information is selected based on the selected result image, the processing unit 34 applies makeup of the content specified by the makeup pattern corresponding to the selected result image. Is obtained from the generation unit 40 and displayed on the output unit 32. If the user does not want to select any of the plurality of result images 710, 720, 730, and 740, the makeup input screen may be displayed again.

  As described above, in the terminal device 30, when a makeup pattern is input using the input unit 31, a result image corresponding to the input makeup pattern is displayed on the output unit 32. Therefore, the makeup pattern can be changed so that a result image desired by the user (a result image that the user likes) can be obtained. Further, the terminal device 30 can display makeup information corresponding to the makeup pattern on the output unit 32.

  As described above, the terminal device 30 generates information on the input image, the result of selecting the touch of the illustration, the result of selecting the makeup pattern, and the result of selecting the result image in the makeup assisting system 10. Used to give to. That is, the terminal device 30 functions as an image input unit that receives input of an input image that is a digital photograph of a predetermined part of a human body to be makeupd. In addition, the terminal device 30 functions as a makeup pattern input unit that receives input of a makeup pattern.

  Next, the generation unit 40 will be described. As illustrated in FIG. 1, the generation unit 40 includes a communication unit 41, a model storage unit 42, and a processing unit 43. The generation unit 40 is configured by, for example, a computer system having a CPU and a memory as main components, and the computer system functions as the generation unit 40 when the CPU executes a program stored in the memory. Here, the program is recorded in advance in the memory of the generation unit 40, but may be provided through a telecommunication line such as the Internet or in a recording medium such as a memory card.

  The communication unit 41 is connected to the network 60 to enable communication with the terminal device 30.

  The model storage unit 42 stores a makeup support model used for processing in the processing unit 43. The makeup support model is a learned model that is used to generate a result image from an input image.

  In the present embodiment, the makeup support model generates an image (hereinafter referred to as a normalized image) indicating a reference state of a predetermined part for applying a makeup pattern in the process of generating a result image from an input image. For example, the reference state is a state to be subjected to a plurality of makeups having different directions. In other words, the reference state is a state in which the predetermined part can be easily converted to the state after makeup. This facilitates the process of generating an image of a predetermined part after makeup from the normalized image. As an example, the reference state assumes a state in which no makeup is applied, or a state in which only processing common to a plurality of makeups having different directions is performed. For example, when the predetermined part is a face, a state where no makeup is applied corresponds to a bare face. It can be said that the state where only the process common to the assumed makeup is performed is a state where it is not necessary to remove makeup in advance for any of a plurality of makeups. As an example, such a state may correspond to a state in which only base makeup has been completed.

  The makeup support model generates a result image from the normalized image based on the makeup pattern. The result image shows a state after the makeup of the predetermined portion obtained by applying makeup having the content specified by the makeup pattern to the reference state of the predetermined portion represented by the normalized image. By generating the result image from the normalized image, even if the user visiting the store has already applied some makeup to the predetermined part, the state of the predetermined part of the user when visiting the store may affect the result image. Can be reduced.

  Here, the makeup pattern can be input using the terminal device 30 as described above. This makes it easier to propose makeup that satisfies the user.

  In the present embodiment, the makeup support model includes a normalization model and a derived model.

  The normalized model is used to generate a normalized image to be given to the derived model from the user input image. More specifically, the normalization model normalizes from the input image by correcting the pixel value of the input image so that the coordinates of the feature amount derived from the pixel value of the input image are within a specified range in the feature amount space. Used to generate an image.

  Here, the feature amount may be, for example, a pixel value of the input image, or may be a value derived by applying an m × n filter to the pixels constituting the input image. m and n are integers of 2 or more, and m and n may be the same. The feature amount space is a space defined by coordinate axes for each feature amount derived from the input image. If the number of feature amounts is 100, the feature amount space is a 100-dimensional space. The feature amount coordinates are defined by values on the coordinate axes of the feature amounts derived from the input image. The specified range can be determined by a normalized image (that is, a reference state of a predetermined part). In the present embodiment, the specified range is a range including the center of gravity (or the center) of the distribution of the feature amount coordinates of the input image included in the learning data. This means that the state of the predetermined part is brought close to a state where makeup is not performed. By setting the specified range in this way, a normalized image in which the influence of the user's makeup at the time of visiting the store is reduced from a predetermined part can be generated. This facilitates the process of generating an image of a predetermined part after makeup from the normalized image. Note that the smaller the size of the specified range, the smaller the difference between the normalized images other than the face shape.

  In the present embodiment, the normalized model is a learned model in which the relationship between the input image and the normalized image is learned by learning data including the relationship between the input image and the normalized image for a plurality of people. By using such a normalization model, the accuracy of the normalized image is improved.

  As a model for generating a normalized model, for example, a neural network which is a kind of hierarchical model is used. For example, in a neural network, each pixel value of the input image is given as a variable to the input layer, and a variable from the output layer is given as each pixel value of the normalized image. When the predetermined part is a face, there is a possibility that the shape of the face (the contour of the face, the shape and arrangement of the facial parts) does not change significantly in the input image and the normalized image. In addition, there is a possibility that the part that can be changed by makeup on the face (part where lipstick is applied, part where cheek is applied, part where eye shadow is applied, etc.) is determined to some extent in relation to facial parts. Therefore, if a structure of one or more intermediate layers (hidden layers) of the neural network is designed based on such a prospect, a learned model with high accuracy can be obtained.

  The derived model is used to generate a result image from the normalized image. In addition, the derived model uses a makeup pattern when generating a result image. That is, the derived model is used to generate a result image from the normalized image and the makeup pattern. More specifically, the derived model derives a feature value from the pixel value of the normalized image, and corrects the pixel value of the normalized image so that the coordinates of the derived feature value become values according to the makeup pattern. Thus, it is used to generate a result image from the normalized image and the makeup pattern.

  In the present embodiment, the derived model is a learning in which the relationship between the input information including the normalized image and the makeup pattern and the result image is learned by the learning data including the relationship between the input information and the result image for a plurality of persons. It is a finished model. By using such a derived model, the accuracy of the result image is improved.

  As a model for generating a derived model, for example, a neural network which is a kind of hierarchical model is used. For example, in a neural network, each pixel value of a normalized image and a numerical value defined by a makeup pattern are given as variables to the input layer, and a variable from the output layer is given as each pixel value of the result image. For example, when the predetermined part is a face, there is a possibility that the shape of the face (the contour of the face, the shape and arrangement of the facial parts) does not change significantly in the normalized image and the result image. Also, facial parts that can be changed by makeup determined according to the makeup pattern (lipstick-applied parts, cheeked parts, eye-shadowed parts, etc.) are determined to some extent in relation to facial parts. There is a prospect. Therefore, if a structure of one or more intermediate layers (hidden layers) of the neural network is designed based on such a prospect, a learned model with high accuracy can be obtained.

  As described above, the result image is an image showing a predetermined portion by an illustration. Therefore, it can be expected that the psychological burden when the user views his / her own result image is reduced. In order to obtain such a result image, an image showing a predetermined part as an illustration is used as a normalized image of learning data of a normalized model. In this case, the illustration process is executed in the process of generating the normalized image from the input image by the normalized model. Therefore, the makeup support model has a combination of a normalized model and a derived model for each touch of an illustration. Illustrations are easier to obtain simple features than digital photographs. Therefore, improvement in the accuracy of the result of the makeup support model can be expected.

  In the makeup support model, the process of generating a result image from an input image includes processing (a process for generating a normalized image) related to items (common items) executed in common regardless of the makeup pattern, and a makeup pattern. It is divided into processing relating to items (unique items) that are executed independently for each (processing for generating a derived image). Separately learned models (normalized model and derived model) are used for the common term and the unique term. Therefore, when a plurality of result images are generated, overlapping processes can be omitted, the total calculation amount can be reduced, and the load can be reduced and the processing speed can be increased.

  The processing unit 43 is configured to generate a result image from the input image using the makeup support model stored in the model storage unit 42 and transmit the result image to the terminal device 30. More specifically, the processing unit 43 generates a normalized image based on the input image obtained through the communication unit 41 and the result of selection of an illustration touch using the normalized model. In addition, the processing unit 43 generates a result image based on the normalized image and the input result of the makeup pattern obtained through the communication unit 41 using the derived model. Then, the processing unit 43 transmits the result image to the terminal device 30 through the communication unit 41.

  The processing unit 43 is configured to generate makeup information for applying makeup having the content specified by the makeup pattern. More specifically, the processing unit 43 generates makeup information based on the selection result of the makeup pattern obtained from the terminal device 30 through the communication unit 41, and outputs the makeup information to the terminal device 30 through the communication unit 41. The processing unit 43 can output the makeup content for realizing the state after the makeup of the predetermined part shown in the result image as makeup information. This makes it easier to propose makeup that satisfies the user.

1.3 Operation Next, an example of the operation of the makeup support system 10 of the present embodiment will be briefly described with reference to FIG. The following example shows a procedure in which a staff member in a store uses the makeup support system 10 to make a makeup proposal that the user is satisfied with to a user who has visited the store.

  First, in the store, the user is photographed using the camera 20 (S100). A plurality of digital photographs of the user obtained by the camera 20 are input to the terminal device 30 as input images.

  The terminal device 30 displays a photo selection screen 500 as shown in FIGS. 2A and 2B and accepts selection of the input image 510 (S110). Accordingly, the user can select the most favorite digital photograph from a plurality of digital photographs (input image 510). The terminal device 30 transmits the input image finally selected by the user to the generation unit 40.

  The terminal device 30 displays an illustration selection screen 600 as shown in FIG. 3 and accepts selection of an illustration touch (S120). Thereby, the user can select the touch of the most favorite illustration with reference to the plurality of sample images 610, 620, 630, and 640. The terminal device 30 transmits information on the touch of the illustration selected by the user (result of selection of the illustration touch) to the generation unit 40.

  Through steps S110 and S120, the terminal device 30 gives the generation unit 40 the result of selecting the input image and the illustration touch. As a result, the generation unit 40 uses the normalization model to generate a normalization image based on the input image obtained from the terminal device 30 and the result of selection of the illustration touch (S130).

  The terminal device 30 displays a makeup input screen and accepts input of a makeup pattern (S140). The makeup input screen is prepared assuming that the store staff listens to the user's opinions and inputs the makeup pattern, rather than the user inputting the makeup pattern. The makeup pattern input on the makeup input screen is transmitted from the terminal device 30 to the generation unit 40.

  The generation unit 40 uses the derived model to generate a result image based on the normalized image generated using the normalization model and the makeup pattern obtained from the terminal device 30 (S150). Then, the generation unit 40 transmits the generated result image to the terminal device 30.

  The terminal device 30 displays a makeup selection screen 700 as shown in FIG. 4 (S160). The makeup selection screen 700 displays result images 710, 720, 730, and 740 received by the terminal device 30 from the generation unit 40. While the makeup selection screen 700 is displayed, the terminal device 30 accepts selection of the result images 710, 720, 730, and 740 by the input unit 31. Therefore, the user can select a favorite result image from the plurality of result images 710, 720, 730, and 740.

  When any one of the plurality of result images 710, 720, 730, and 740 is selected by the input unit 31, the terminal device 30 receives a selection as to whether or not to change the makeup pattern (S170).

  When the change of the makeup pattern is selected (S170: Yes), the terminal device 30 displays the makeup input screen and accepts the input of the makeup pattern (S140). In this case, the terminal device 30 displays in advance a makeup pattern corresponding to the selected result image on the makeup input screen. The store staff changes the makeup pattern corresponding to the selected result image while listening to the user's opinion, and creates a new makeup pattern to suit the user's preference.

  When the change of the makeup pattern is not selected (S170: No), the terminal device 30 transmits a makeup pattern corresponding to the selected result image to the generation unit 40. The generation unit 40 generates makeup information based on the makeup pattern selection result received from the terminal device 30 and outputs the makeup information to the terminal device 30 (S180). And the terminal device 30 displays the makeup information received from the production | generation part 40 (S190). The staff of the store presents the makeup method and necessary cosmetics to the user with reference to the makeup information displayed on the terminal device 30.

  In this way, by using the makeup support system 10, the staff can easily propose makeup that satisfies the user.

  Note that the flowchart of FIG. 5 is merely an example of the operation of the makeup support system 10 of the above-described embodiment, and does not intend to limit the order of steps S100 to S190 to the order of FIG. For example, step S110 and step S120 may be interchanged. Moreover, step S130 may be between step S140 and step S150. In this case, step S130 can be omitted after step S140 is executed once.

1.4 Summary In the makeup support system 10 described above, the generation unit 40 sets the pixel value of the input image so that the coordinates of the feature amount derived from the pixel value of the input image are within the specified range in the feature amount space. It correct | amends and produces | generates the normalization image which shows the reference | standard state of a predetermined part from an input image. In other words, the makeup support system 10 executes the following makeup support method. The makeup support method includes a first step and a second step. In the first step, an input of an input image 510 that is a digital photograph of a predetermined part of a human body to be made up is received. In the second step, the pixel value of the input image 510 is corrected so that the coordinates of the feature amount derived from the pixel value of the input image 510 are within the specified range in the feature amount space. A normalized image indicating the reference state is generated.

  By using the normalized image, it is possible to generate an image (for example, result images 710 to 740) indicating a state after makeup of a predetermined part. That is, it is possible to show the user the state after the makeup of the predetermined part with an image. Therefore, in the makeup support system 10 and the makeup support method of the present embodiment, it is easy to propose makeup that satisfies the user.

  The makeup support system 10 uses the makeup support model to support makeup proposals that satisfy the user. The makeup support model includes a learned model in which the relationship between the input image 510 and the normalized image is learned by learning data including the relationship between the input image 510 and the normalized image for a plurality of people. The input image 510 is a digital photograph of a predetermined part of the human body that is the subject of makeup. The normalized image has a reference state of a predetermined part obtained by correcting the pixel value of the input image 510 so that the coordinates of the feature amount derived from the pixel value of the input image 510 are within the specified range in the feature amount space. It is the image shown. Therefore, according to the makeup support model, it is easy to propose makeup that satisfies the user.

  Furthermore, in the makeup support system 10 described above, the generation unit 40 generates the result images 710 to 740 from the normalized image based on the makeup pattern that specifies the makeup content. The result image shows a state after the makeup of the predetermined portion obtained by applying makeup having the content specified by the makeup pattern to the reference state of the predetermined portion represented by the normalized image. In other words, the makeup support method further includes a third step. In the third step, result images 710 to 740 are generated from the normalized image based on the makeup pattern that specifies the makeup contents. The result image shows a state after the makeup of the predetermined portion obtained by applying makeup having the content specified by the makeup pattern to the reference state of the predetermined portion represented by the normalized image.

  Since the result images 710 to 740 show the state after the makeup of the predetermined portion, the user can convey his / her ideal as a specific image to the staff with reference to the result image. Therefore, the makeup support system 10 and the makeup support method make it easier to propose makeup that satisfies the user.

  In the makeup support system 10 described above, the generation unit 40 generates makeup information for applying makeup having the content specified by the makeup pattern. In other words, the makeup support method further includes a fourth step. In the fourth step, makeup information for applying makeup having the contents specified by the makeup pattern is generated.

  With reference to the makeup information, the staff can present a makeup method and necessary cosmetics to the user. Therefore, it is possible to more easily propose makeup that satisfies the user.

2. The embodiment of the present invention is not limited to the above embodiment. The above embodiment can be variously modified according to the design or the like as long as the object of the present invention can be achieved. Below, the modification of the said embodiment is enumerated.

  For example, makeup support system 10 only needs to have an image input unit (terminal device 30) that accepts input of an input image that is a digital photograph of a predetermined part of a human body to be made up, and the input image itself. It is not necessary to have the function of generating That is, makeup support system 10 may not include camera 20. For example, when a user brings his / her favorite digital photograph to the store, the digital photograph may be used as an input image.

  Moreover, in the said embodiment, the production | generation part 40 produces | generates a result image from a normalization image based on the makeup pattern input into the makeup pattern input part (terminal device 30). However, the generation unit 40 may be configured to generate a result image from the normalized image based on the makeup pattern determined based on the normalized image. That is, the generation unit 40 may automatically determine a makeup pattern based on the feature amount derived from the pixel value of the normalized image. A learned model can be used to automatically determine the makeup pattern. For example, a learned model may be generated using a relationship between a user's normalized image and a makeup pattern that tends to satisfy the user.

  In the above embodiment, the processing unit 34 displays the photo selection screen 500. However, when the input image to be used is determined in advance, the processing unit 34 does not necessarily display the photo selection screen 500.

  In the above embodiment, the processing unit 34 displays the illustration selection screen 600. However, the result image may display the state after makeup of the predetermined part as a photograph instead of an illustration. As described above, when the illustration process is not performed (when the state after the makeup of the predetermined part is not displayed as an illustration), the processing unit 34 does not need to display the illustration selection screen 600.

  In the above embodiment, the processing unit 34 does not display the normalized image on the output unit 32, but may display the normalized image on the output unit 32.

  Note that the normalized image is an image generated for the purpose of keeping variations in input images of various users within a predetermined range in order to improve the accuracy of the result image. Therefore, the reference state does not necessarily need to be a state where it is not necessary to remove makeup in advance for any of a plurality of makeups as in the above embodiment. That is, since the reference state changes depending on how the normalized image is selected, it may be a pre-selected state after makeup.

  Further, in the above-described embodiment, an image showing a predetermined part as an illustration is used as a normalized image of learning data of a normalized model. As a result, the illustration process is executed in the process in which the normalized image is generated from the input image by the normalized model. Instead of this, an image showing a predetermined portion as an illustration may be used as an input image of learning data for the normalized model. In this case, either the terminal device 30 or the generation unit 40 may execute the illustration processing of the input image. Or the image which shows a predetermined part with an illustration may be used as a result image of the data for learning of a derived model. Thus, the illustration process is executed in the process of generating the result image from the normalized image by the derived model. Instead of this, an image showing a predetermined part as an illustration may be used as a normalized image of the learning data for the derived model. In this case, the generation unit 40 may perform illustration processing of the normalized image. Alternatively, the generation unit 40 may execute a result image illustration process.

  In the above embodiment, the makeup pattern includes a parameter indicating information on one or more makeup processes, but may further include a parameter indicating a situation around a predetermined portion as an additional parameter. When the predetermined part is a face, examples of the situation around the predetermined part include clothes and a hairstyle. This is because the impression of a person's face depends on clothes and hairstyle. The information on clothes may be, for example, information on clothes themselves or information indicating an impression of clothes. The information indicating the impression of clothes may include, for example, information indicating the color of clothes, style (formal, casual, etc.), usage, and the like. Similarly, the hairstyle information may be, for example, information on the hairstyle itself, or information indicating an impression of the hairstyle. The information indicating the impression of the hairstyle can include, for example, information indicating the color of the hair, the pattern (long, short, bob, etc.) and the like.

  In the above embodiment, the processing unit 34 generates a result image using the derived model. However, the result image may be generated by processing the normalized image with the terminal device 30. This assumes a case where the staff uses the terminal device 30 to process the normalized image and generate a result image while listening to the user's request. In this case, the terminal device 30 is required to have a function of processing a normalized image and generating a result image according to an input from the input unit 31. Such a function can be realized by a known image processing program.

  In other words, the makeup support model does not necessarily include the derived model, and may include at least the normalization model. In the above embodiment, the normalization model is prepared for each touch of the illustration, but a plurality of normalization models may be prepared according to the category. Examples of categories include the price range, gender, age, and makeup application (eg, time, place, purpose) of cosmetics used for makeup. This also applies to the derived model.

  In the above embodiment, makeup support system 10 includes one or more cameras 20, one or more terminal devices 30, and a generation unit 40. The one or more terminal devices 30 and the generation unit 40 are configured by a computer system including a plurality of computers. However, makeup support system 10 may at least realize terminal device 30 and generation unit 40 with a single computer. That is, makeup support system 10 may be configured by a computer system having a CPU and a memory as main components, for example. Then, the computer system may function as the makeup support system 10 by the CPU executing the program stored in the memory. Here, the program is recorded in advance in the memory of the makeup support system 10, but may be provided through a telecommunication line such as the Internet or in a recording medium such as a memory card. The computer system may be configured by a single computer or a system configured by a plurality of computers. For example, makeup support system 10 may be configured by at least one of a plurality of computers. A plurality of computers may share the functions of the makeup support system 10. Moreover, at least a part of the functions of the makeup support system 10 may be realized by, for example, cloud (cloud computing). The computer used here does not need to be a dedicated device, and may be a smartphone, a tablet computer, a personal computer, or the like. Therefore, makeup support system 10 may be realized by a computer installed in one place, or may be realized by a plurality of computers arranged in a plurality of locations.

  The makeup support system 10 of the above embodiment can propose an image that specifically shows the state after the makeup of the user when the user has not yet clearly grasped his / her ideal regarding the state after the makeup. To. Therefore, the user can convey his / her ideal as a specific image to the staff with reference to the image. That is, the makeup support system 10 assists in transmitting the user's ideals to the staff. The technical idea of proposing such a user's ideal as a specific image can be applied to various scenes. For example, it can be applied to scenes such as clothing coordination, building remodeling, and product ordering. Examples of the product include clothes, furniture (for example, a makeup shelf), a car, and the like. Here, the case of renovation of a building will be briefly described. In this case, as an input image, a digital photograph of a place (for example, a room) to be reformed is used. In the normalized image, a state in which furniture, wallpaper, or the like is removed from a location to be reformed is used as the reference state. And as a result image, the image which shows the state after reform of the place used as the object of remodeling is used. The state after the renovation is a state in which, for example, furniture or wallpaper desired by the user is installed. Even in this case, the learned model can be used. The process of generating the result image from the input image is a process related to an item (common item) that is executed in common regardless of the content of the reform, and an item that is executed independently for each content of the reform (original item) It may be divided into processing related to. Common items include basic room and furniture shapes. Independent items include materials (wood and metal), colors, and patterns. Similar to the above embodiment, different learned models may be used for the common term and the unique term.

3. Aspect As is apparent from the embodiment and the modification described above, the makeup support system (10) of the first aspect accepts input of an input image that is a digital photograph of a predetermined part of a human body to be makeupd. An image input unit (terminal device 30) and a generation unit (40) are provided. The generation unit (40) corrects the pixel value of the input image (510) so that the coordinates of the feature amount derived from the pixel value of the input image (510) are within a specified range in the feature amount space. , Configured to generate a normalized image from the input image (510). The normalized image shows a reference state of the predetermined part. According to the 1st aspect, the proposal of the makeup which a user is satisfied becomes easy.

  The makeup support system (10) of the second aspect can be realized by a combination with the first aspect. In the second aspect, the generation unit (40) is configured to generate the normalized image from the input image (510) using a normalized model. The normalized model is learned by learning the relationship between the input image (510) and the normalized image by learning data including the relationship between the input image (510) and the normalized image for a plurality of people. It is a model. According to the second aspect, the accuracy of the normalized image is improved.

  The makeup support system (10) of the third aspect can be realized by a combination with the second aspect. In the third aspect, the specified range is a range including the centroid of the feature value coordinate distribution of the input image (510) included in the learning data. According to the 3rd aspect, the process which produces | generates the image of the state after the makeup of a predetermined part from a normalization image becomes easy.

  The makeup support system (10) of the fourth aspect can be realized by a combination with any one of the first to third aspects. In the fourth aspect, the reference state is a state to be subjected to a plurality of makeups having different directions. According to the 4th aspect, the process which produces | generates the image of the state after the makeup of a predetermined part from a normalization image becomes easy.

  The makeup support system (10) of the fifth aspect can be realized in combination with the fourth aspect. In the fifth aspect, the reference state is a state in which it is not necessary to remove makeup in advance for any of the plurality of makeups. According to the 5th aspect, the process which produces | generates the image of the state after the makeup of a predetermined part from a normalization image becomes easy.

  The makeup support system (10) of the sixth aspect can be realized by a combination with any one of the first to fifth aspects. In the sixth aspect, the generation unit (40) is configured to generate a result image (710-740) from the normalized image based on a makeup pattern that specifies the contents of makeup. The result image shows a state after the makeup of the predetermined portion obtained by applying makeup of the content specified by the makeup pattern to the reference state of the predetermined portion represented by the normalized image. According to the 6th aspect, the proposal of the makeup which a user is satisfied becomes easier.

  The makeup support system (10) of the seventh aspect can be realized in combination with the sixth aspect. In a seventh aspect, the makeup support system (10) further includes a makeup pattern input unit (terminal device 30) that receives input of the makeup pattern. The generation unit (40) is configured to generate the result image (710-740) from the normalized image based on the makeup pattern input to the makeup pattern input unit (terminal device 30). Is done. According to the 7th aspect, the proposal of the makeup which a user is satisfied becomes easier.

  The makeup support system (10) of the eighth aspect can be realized by a combination with the sixth aspect. In the eighth aspect, the generation unit (40) is configured to generate the result image (710-740) from the normalized image based on a makeup pattern determined based on the normalized image. Is done. According to the eighth aspect, it is easier to propose makeup that satisfies the user.

  The makeup support system (10) of the ninth aspect can be realized by a combination with any one of the sixth to eighth aspects. In the ninth aspect, the generation unit (40) is configured to generate the result image (710-740) from the normalized image using a derived model. The derived model has a relationship between the input information including the normalized image and the makeup pattern and the result image (710 to 740), and the input information and the result image (710 to 740) for a plurality of persons. This is a learned model learned by learning data including the relationship. According to the ninth aspect, the accuracy of the result image is improved.

  The makeup support system (10) of the tenth aspect can be realized by a combination with any one of the sixth to ninth aspects. In the tenth aspect, the generation unit (40) is configured to generate makeup information for applying makeup having the content specified by the makeup pattern. According to the 10th aspect, the proposal of the makeup suitable for a user becomes still easier.

  The makeup support system (10) of the eleventh aspect can be realized by a combination with any one of the sixth to tenth aspects. In the eleventh aspect, the result image is an image showing the state after the makeup of the predetermined portion as an illustration. According to the eleventh aspect, a reduction in psychological burden when the user views his / her own result image can be expected.

  The makeup support method according to the twelfth aspect includes a first step and a second step. In the first step, an input of an input image (510) which is a digital photograph of a predetermined part of a human body to be made up is received. In the second step, the pixel value of the input image (510) is corrected so that the coordinates of the feature amount derived from the pixel value of the input image (510) are within a specified range in the feature amount space, and the input It is configured to generate a normalized image from the image (510). The normalized image shows a reference state of the predetermined part. According to the twelfth aspect, it is easy to propose makeup that satisfies the user.

  The makeup support model of the thirteenth aspect learns the relationship between the input image (510) and the normalized image by learning data including the relationship between the input image (510) and the normalized image for a plurality of people. Trained models. The input image (510) is a digital photograph of a predetermined part of the human body to be made up. The normalized image is obtained by correcting the pixel value of the input image (510) so that the coordinates of the feature amount derived from the pixel value of the input image (510) are within a specified range in the feature amount space. FIG. 6 is an image showing a reference state of the predetermined part. According to the thirteenth aspect, it is easy to propose makeup that satisfies the user.

10 makeup support system 30 terminal device (image input unit, makeup pattern input unit)
40 generator

Claims (13)

An image input unit for receiving input of an input image which is a digital photograph of a predetermined part of a human body to be made up;
The pixel value of the input image is corrected so that the coordinates of the feature amount derived from the pixel value of the input image are within a specified range in the feature amount space, and the reference state of the predetermined part is indicated from the input image A generator for generating a normalized image;
Comprising
Makeup support system. The generation unit is configured to generate the normalized image from the input image using a normalized model,
The normalized model is a learned model in which the relationship between the input image and the normalized image is learned by learning data including the relationship between the input image and the normalized image for a plurality of people.
The makeup support system according to claim 1. The specified range is a range including a centroid of a feature value coordinate distribution of the input image included in the learning data.
The makeup support system according to claim 2. The reference state is a state to be subjected to a plurality of makeup having different directions.
The makeup support system according to any one of claims 1 to 3. The reference state is a state where it is not necessary to remove makeup in advance for any of the plurality of makeups.
The makeup support system according to claim 4. Based on a makeup pattern that specifies makeup content, the generation unit generates a content specified by the makeup pattern from the normalized image to a reference state of the predetermined portion represented by the normalized image. Configured to generate a result image indicating a state after makeup of the predetermined portion obtained by applying makeup;
The makeup support system according to any one of claims 1 to 5. A makeup pattern input unit for receiving input of the makeup pattern;
The generation unit is configured to generate the result image from the normalized image based on the makeup pattern input to the makeup pattern input unit.
The makeup support system according to claim 6. The generating unit is configured to generate the result image from the normalized image based on a makeup pattern determined based on the normalized image.
The makeup support system according to claim 6. The generating unit is configured to generate the result image from the normalized image using a derived model,
The derived model has learned the relationship between the input information including the normalized image and the makeup pattern and the result image by learning data including the relationship between the input information and the result image for a plurality of persons. A trained model,
The makeup support system according to any one of claims 6 to 8. The generating unit is configured to generate makeup information for applying makeup of the content specified by the makeup pattern;
The makeup support system according to any one of claims 6 to 9. The result image is an image showing the state after makeup of the predetermined part in an illustration,
The makeup support system according to any one of claims 6 to 10. A first step of accepting input of an input image that is a digital photograph of a predetermined part of a human body to be made up;
The pixel value of the input image is corrected so that the coordinates of the feature amount derived from the pixel value of the input image are within a specified range in the feature amount space, and the reference state of the predetermined part is indicated from the input image A second step of generating a normalized image;
including,
Makeup support method. Including a learned model in which a relationship between an input image and a normalized image is learned by learning data including a relationship between the input image and the normalized image for a plurality of persons;
The input image is a digital photograph of a predetermined part of a human body to be made up,
The normalized image is obtained by correcting the pixel value of the input image so that the coordinates of the feature amount derived from the pixel value of the input image are within a specified range in the feature amount space. It is an image showing the state,
Makeup support model.