JP2022133792A - Makeup simulation system, makeup simulation method, makeup simulation program, and makeup simulation device - Google Patents

Abstract

To provide a makeup simulation system, a makeup simulation method, a makeup simulation program, and a makeup simulation device capable of increasing reproduction accuracy of makeup simulation.SOLUTION: A makeup simulation device includes: a first spectral reflectivity calculation unit 23b that calculates first spectral reflectivity of an application target before makeup from a first RGB value of the application target before makeup; a first color signal calculation unit 23c that calculates a first color signal of the application target before makeup using the first spectral reflectivity; a second color signal calculation unit 23e that calculates a second color signal of cosmetics using second spectral reflectivity of the cosmetics; a third color signal calculation unit 23f that calculates a third color signal of a makeup state in which the cosmetics are applied to the application target from the first color signal and the second color signal; a second RGB value calculation unit 23g that calculates a second RGB value of the makeup state from the third color signal; and an object generation unit 23a that generates a three-dimensional object in a makeup state from the second RGB value.SELECTED DRAWING: Figure 4

Description

The present invention relates to a makeup simulation system, a makeup simulation method, a makeup simulation program, and a makeup simulation apparatus for reproducing a state in which cosmetics are applied to an application target.

2. Description of the Related Art A makeup simulation system is known that displays a simulation image of a user's face after applying cosmetics using a face image. For example, Patent Literature 1 describes a makeup simulation method that generates a simulation image of a face with cosmetics applied by superimposing a pattern for each cosmetic on a user's face image without makeup. .

JP 2007-257194 A

When makeup is applied to the face, the area to which the makeup is applied has a color that is a combination of the color of the skin and the color of the makeup. That is, when the skin color of the user is different, the regions coated with the cosmetic have different colors even when the same cosmetic is applied. However, in the above-described method of simply superimposing a cosmetic pattern on a user's face image, the skin color is not reflected in the simulation image, and the simulation image and the appearance when the user actually wears makeup are different. divergence becomes large. Therefore, with the above method, it is difficult to perform makeup simulation with high reproducibility.

A makeup simulation system for solving the above-described problems is a first spectral reflectance calculation that calculates a first spectral reflectance of an application target before makeup from a first RGB value before makeup of an application target to which a cosmetic is applied. part, the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristic of the application object before the makeup, the spectral distribution of the illumination light, and a geometric condition. and a second spectral reflectance of the cosmetic and a second reflection characteristic that defines the reflection characteristic of the cosmetic. a second color signal calculation unit that calculates a second color signal of the cosmetic using a second three-dimensional reflection characteristic model represented by a parameter, the spectral distribution of the illumination light, and the geometric condition; a third color signal calculation unit for calculating a third color signal for the application object in a makeup state in which the cosmetic is applied to the application object before the makeup from the color signal and the second color signal; a second RGB value calculation unit that calculates second RGB values of the application target in the makeup state from the color signal; and an object generation unit that generates a three-dimensional object including the application target in the makeup state from the second RGB values. , provided.

A makeup simulation method for solving the above-described problems includes a step of calculating a first spectral reflectance of the application object before makeup from first RGB values before makeup on the application object to which the cosmetic is applied; Using a first three-dimensional reflection characteristic model expressed by a spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristic of the application target before the makeup, and a spectral distribution and geometric conditions of the illumination light, the before makeup a second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristic of the cosmetic, the spectral distribution of the illumination light, and the a step of calculating a second color signal of the cosmetic using a second three-dimensional reflection characteristic model represented by a geometric condition; a step of calculating a third color signal of the application object in a makeup state in which the cosmetic is applied to an object; a step of calculating second RGB values of the application object in the makeup state from the third color signal; generating a three-dimensional object including the application target in the makeup state from the second RGB values.

A makeup simulation program for solving the above-described problems provides a computer with a process of calculating a first spectral reflectance of an application target before makeup from first RGB values before makeup on an application target on which cosmetics are applied; Using a first three-dimensional reflection characteristic model represented by the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristic of the application object before the makeup, and the spectral distribution and geometric conditions of the illumination light A process of calculating a first color signal on the application target before the makeup, a second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining a reflection characteristic of the cosmetic, and the spectrum of the illumination light a process of calculating a second color signal of the cosmetic using a second three-dimensional reflection characteristic model represented by the distribution and the geometric condition; a process of calculating a third color signal of the application object in a makeup state in which the cosmetic is applied to the application object; and calculating a second RGB value of the application object in the makeup state from the third color signal. and a process of generating a three-dimensional object including the application target in the makeup state from the second RGB values.

A makeup simulation apparatus for solving the above-described problem is a first spectral reflectance calculation that calculates a first spectral reflectance of an application target before makeup from a first RGB value before makeup of an application target to which a cosmetic is applied. part, the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristic of the application object before the makeup, the spectral distribution of the illumination light, and a geometric condition. and a second spectral reflectance of the cosmetic and a second reflection characteristic that defines the reflection characteristic of the cosmetic. a second color signal calculation unit that calculates a second color signal of the cosmetic using a second three-dimensional reflection characteristic model represented by a parameter, the spectral distribution of the illumination light, and the geometric condition; a third color signal calculation unit for calculating a third color signal for the application object in a makeup state in which the cosmetic is applied to the application object before the makeup from the color signal and the second color signal; a second RGB value calculation unit that calculates second RGB values of the application target in the makeup state from the color signal; and an object generation unit that generates a three-dimensional object including the application target in the makeup state from the second RGB values. , provided.

According to each of the above configurations, the first color signal is calculated using the first three-dimensional reflection characteristic model, and the second color signal is calculated using the second three-dimensional reflection characteristic model. As the color signal of the object, a third color signal in which the color of the object to be applied is reflected in the cosmetic can be calculated. Then, by calculating the second RGB values from the third color signal, a three-dimensional object of the application target in the makeup state is generated. As a result of the simulation, it is possible to obtain a three-dimensional object of the application target in the makeup state in which the color of the application target, the shadow due to the illumination light, and the gloss are reflected. Therefore, makeup simulation can be performed with higher reproducibility than when simply superimposing the color of the cosmetic on the object to be applied. In addition, since it is possible to reproduce shadows and luster caused by the illumination light in the makeup state, it is possible to reproduce the texture of the state in which the cosmetic is applied to the application target. Then, for example, when counseling the user, by using the makeup simulation, it is possible to propose the optimum cosmetics to the user.

In the makeup simulation system, the first conversion function for converting the first RGB values into the first spectral reflectance is stored according to the color of the object to be applied before the makeup. wherein the first spectral reflectance calculation unit calculates the first spectral reflectance using the first conversion function corresponding to the color of the object to be applied before applying makeup. good. According to the above configuration, by calculating the first spectral reflectance based on the first conversion function corresponding to the color of the coating object, the first spectral reflectance is the spectral reflectance closer to the actual coating object. can be calculated. Therefore, makeup simulation can be performed with higher reproducibility.

In the makeup simulation system, a second conversion function for calculating the second spectral reflectance from the first spectral reflectance, the second conversion function corresponding to the color of the object to be applied before the makeup. and a second spectral reflectance for calculating the second spectral reflectance using the first spectral reflectance and the second conversion function corresponding to the color of the object to be applied before the makeup. and a ratio calculation unit, wherein the second color signal calculation unit calculates the second color signal using the second three-dimensional reflection characteristic model including the second spectral reflectance calculated by the second spectral reflectance calculation unit. A two-color signal may be calculated. According to the above configuration, by calculating the second spectral reflectance based on the first spectral reflectance and the second conversion function corresponding to the color of the application object, the cosmetic is actually applied to the application object. It is possible to calculate the second spectral reflectance that is closer to the spectral reflectance of the cosmetic in a state in which it has been blended with the application target. Therefore, makeup simulation can be performed with higher reproducibility.

In the above makeup simulation system, the cosmetics database stores the second conversion function and the second reflection characteristic parameter for each product of the cosmetics, and the second spectral reflectance calculation unit stores the cosmetics The second spectral reflectance is calculated using the second conversion function for each product, and the second color signal calculation unit calculates the second spectral reflectance calculated by the second spectral reflectance calculation unit, The second color signal may be calculated using the second three-dimensional reflection characteristic model including the second reflection characteristic parameter for each cosmetic product. According to the above configuration, by calculating the second color signal using the second spectral reflectance and the second reflection characteristic parameter for each cosmetic product, the color and reflection characteristic of each cosmetic product are reflected. Can generate 3D objects. Therefore, makeup simulation can be performed with higher reproducibility.

In the above-described makeup simulation system, the cosmetic database includes spectral reflectance of a glittering agent contained in the cosmetic, a reflection characteristic parameter of the glittering agent that defines the reflection characteristics of the glittering agent, and a size of the glittering agent. and dispersion information that defines the dispersion state of the brightening agent, and the second color signal calculation unit uses the second three-dimensional reflection characteristic model to store the shape information that defines the The second color signal is calculated, and the spectral reflectance of the glittering agent and the reflection characteristics of the glittering agent are estimated from the shape information and the dispersion information of the glittering agent in an area where the glittering agent is scattered. A color signal of the brightening agent is calculated using a third three-dimensional reflection characteristic model expressed by parameters, the spectral distribution of the illumination light, and the geometric condition, and the third color signal calculation unit calculates the first From the color signal, the second color signal, and the color signal of the glitter agent, the third A color signal may be calculated. According to the above configuration, the color signal of the brightening agent can be calculated using the third three-dimensional reflection characteristic model including the spectral reflectance of the brightening agent contained in the cosmetic. Then, by calculating the second RGB values from the first color signal, the second color signal, and the third color signal calculated based on the color signal of the brightening agent, the luster of the brightening agent in the makeup state was reproduced. Can generate 3D objects. In addition, by calculating the color signal of the glittering agent in the area where the glittering agent is estimated from the shape information and the dispersion information, the particle shape of the glittering agent contained in the actual cosmetic and the dispersion state of the glittering agent can be obtained. Reproducible. Therefore, makeup simulation can be performed with higher reproducibility.

In the makeup simulation system described above, at least one of the second spectral reflectance and the second reflection characteristic parameter may be an actual measurement value measured for the cosmetic. According to the above configuration, as at least one of the second spectral reflectance and the second reflection characteristic parameter, which are parameters constituting the second three-dimensional reflection characteristic model, by using the actual measurement value for the cosmetic, various cosmetics can be obtained. can be easily simulated. For example, when a new cosmetic is developed, the present system can perform a simulation by actually measuring the second spectral reflectance and the second reflection characteristic parameter of the new cosmetic.

In the makeup simulation system, the spectral distribution of the illumination light and the spatial distribution of the illumination light source that generates the illumination light, the spatial distribution in all directions centering on the three-dimensional object, are used as an illumination environment. and the first color signal calculator calculates the first color signal using the first three-dimensional reflection characteristic model including the spectral distribution of the illumination light for each lighting environment. and the second color signal calculator may calculate the second color signal using the second three-dimensional reflection characteristic model including the spectral distribution of the illumination light for each illumination environment. According to the above configuration, by calculating the first color signal and the second color signal based on the illumination spectral distribution for each lighting environment, a cubic color image that reflects lighting effects such as shadows and gloss due to illumination light for each lighting environment is obtained. You can generate the original object. Therefore, makeup simulation can be performed with higher reproducibility.

In the makeup simulation system, the first spectral reflectance calculator, the first color signal calculator, the second color signal calculator, the third color signal calculator, the second RGB value calculator, and the object A GPU functioning as at least one of the generators may be further provided. According to the above configuration, by executing processing in the makeup simulation system using the GPU, image processing can be performed in real time, and makeup simulation can be performed more favorably.

According to the configuration described above, by reflecting the color of the application target on the cosmetic, it is possible to improve the reproduction accuracy of the makeup simulation compared to simply superimposing the color of the cosmetic on the application target.

1 is a schematic diagram showing the overall configuration of a makeup simulation system; FIG. 3 is a block diagram showing the configuration of a user terminal; FIG. FIG. 3 is a block diagram showing the configuration of a server; FIG. 3 is a functional block diagram of a GPU included in the server; FIG. 4 is a schematic diagram showing an application image in which a first three-dimensional object is displayed; FIG. 4 is a schematic diagram showing an application image in which a cosmetics selection area is displayed; 4 is a flowchart showing a processing procedure in the makeup simulation system; Schematic diagram showing a geometric model of reflection in the makeup simulation system. FIG. 9 is a schematic diagram showing the reflected light distribution in the geometric model of reflection shown in FIG. 8 ; FIG. 3 is a schematic diagram showing interference light generated in a multilayer film of glittering agents contained in cosmetics. FIG. 4 is a schematic diagram showing an application image in which a second three-dimensional object is displayed;

[First Embodiment]
A first embodiment of a makeup simulation system, a makeup simulation method, a makeup simulation program, and a makeup simulation apparatus will be described below with reference to FIGS. 1 to 11. FIG.

〔overall structure〕
As shown in FIG. 1 , makeup simulation system 1 includes user terminal 10 and server 20 . The user terminal 10 is an information processing device owned by the user 2, and is, for example, a smart phone. Note that the user terminal 10 may be an information processing device such as a tablet, a laptop computer, or a desktop computer.

The server 20 is a server computer that executes various processes in the makeup simulation system 1, and is an example of a makeup simulation device for performing makeup simulation. Specifically, the server 20 generates a first three-dimensional object 30a, which is the three-dimensional object 30 of the face in the non-makeup state, which is the state before makeup. The server 20 also generates a second three-dimensional object 30b, which is the three-dimensional object 30 of the face in the makeup state, which is the state after makeup.

The user terminal 10 and the server 20 are connected via a network 3, which is a communication line. The network 3 is, for example, a mobile communication system such as 4G or 5G for mobile phones, or a wireless LAN communication system such as Wi-Fi (registered trademark).

A three-dimensional object display program as an application program is installed in the user terminal 10 . The three-dimensional object display program is a program that causes the user terminal 10 to execute processing for displaying the first three-dimensional object 30a and the second three-dimensional object 30b. The three-dimensional object display program is downloaded from the application delivery device via the network and installed in the user terminal 10 .

The user terminal 10 has a touch panel 13 . The touch panel 13 displays a first three-dimensional object 30a, which is a three-dimensional object without makeup, and a second three-dimensional object 30b, which is a three-dimensional object 30 with makeup, as a simulation result.

[User terminal]
As shown in FIG. 2 , the user terminal 10 includes a control section 11 , a memory 12 , a touch panel 13 , an imaging section 14 and a communication section 15 . The control unit 11 is, for example, a CPU and controls the overall operation of the user terminal 10 .

The memory 12 includes a main memory and a data storage section. The main memory included in the memory 12 is, for example, a RAM, which temporarily stores data and programs of the user terminal 10 . The data storage unit included in the memory 12 is a non-volatile memory, such as a flash memory. A data storage unit included in the memory 12 stores programs for executing various processes in the user terminal 10, such as a three-dimensional object display program.

The touch panel 13 is a display that enables touch operation input by combining a display device such as a liquid crystal panel or an organic EL panel and a position input device. In other words, the touch panel 13 is a display unit on which the three-dimensional object 30 is displayed in the user terminal 10 and an operation unit for inputting operations to the server 20 .

The imaging unit 14 is, for example, an RGB camera including an imaging element such as a CCD or CMOS. The imaging unit 14 detects light such as visible light, and outputs image data composed of RGB values of red, green, and blue. Hereinafter, the RGB values of the image acquired by the imaging unit 14 are referred to as first RGB values.

The imaging unit 14 acquires a face image of the user 2 in the no-makeup state composed of the first RGB values. The first three-dimensional object 30a is generated based on the first RGB values that constitute the facial image of the user 2 without makeup acquired by the imaging unit 14 .

The communication unit 15 communicates with the server 20 via the network 3 . Specifically, the communication unit 15 transmits a signal from the user terminal 10 to the server 20, and receives data generated by the server 20 such as the first three-dimensional object 30a and the second three-dimensional object 30b.

〔server〕
As shown in FIG. 3 , the server 20 includes a control section 21 , a memory 24 and a communication section 25 . The control part 21 is provided with CPU22 and GPU23 as an example. The CPU 22 controls overall operations other than the process of generating the three-dimensional object 30 in the server 20 . The GPU 23 executes processing for generating the three-dimensional object 30 in the server 20 .

The memory 24 includes a main memory and a data storage section. The main memory included in the memory 24 is, for example, a RAM, which temporarily stores data and programs of the server 20 . A data storage unit included in the memory 24 is a non-volatile memory, such as a flash memory such as an HDD or an SSD.

A data storage unit included in the memory 24 stores programs for executing various processes in the server 20, such as a makeup simulation program. The makeup simulation program is a program that causes the server 20 to execute processing for generating the first three-dimensional object 30a and the second three-dimensional object 30b. Processing in the makeup simulation program is controlled by the GPU 23 provided in the control unit 21 .

The memory 24 includes a spectral conversion database 24a as a data storage unit, a lighting environment database 24b, and a cosmetics database 24c. In the first three-dimensional object 30a, the spectral conversion database 24a converts the first RGB values constituting the application target to which the cosmetic is applied to the first spectral reflectance, which is the spectral reflectance of the application target in the non-makeup state. Store a first transformation function M for transforming the rate S ₁ (λ). In this embodiment, the spectral conversion database 24a includes a first conversion function for converting the first RGB values into the first spectral reflectance S ₁ (λ) for each of bare skin and lips on the face, which is an example of the application target. store M.

The first conversion function M is obtained by actually measuring the RGB values and spectral reflectances of the application target for a large number of people, and determining the correspondence relationship between the RGB values and the spectral reflectance of the application target. It is a transformation matrix obtained by statistically analyzing each The first conversion function M is, for example, a 61×3 conversion matrix that divides the visible wavelength range from 400 nm to 700 nm into 5 nm intervals.

As a specific example, the spectral conversion database 24a stores the first type skin, the second type skin, and the third type skin according to the amount of melanin contained in the skin, which is an example of the application target. The first conversion function M is stored according to the skin color, divided into two types. The first type of skin is skin in the first range where the amount of melanin pigment is less than the second and third types, and has a high relative reflectance of each wavelength when the skin is exposed to light. be. The second type of skin is skin having a melanin pigment amount in a second range that is higher than the first range, and having a medium relative reflectance at each wavelength when the skin is exposed to light. The third type of skin is skin in the third range in which the amount of melanin pigment is higher than in the second range, and has a low relative reflectance at each wavelength when the skin is exposed to light. Note that the spectral conversion database 24a may, for example, divide the RGB values of the skin into predetermined ranges and store the first conversion function M for each division of the RGB values of the skin.

The illumination environment database 24b stores light source information, which is a combination of the illumination spectral distribution E(λ), which is the spectral distribution of the illumination light applied to the three-dimensional object 30, and the spatial distribution of the illumination light source that generates the illumination light, as the illumination environment. memorize each The illumination light source here is, for example, a direct light source that directly irradiates the three-dimensional object 30 with illumination light, and an object existing in the illumination environment that reflects the illumination light from the direct light source toward the three-dimensional object 30. Although there is a reflected light source, these are not particularly distinguished in this embodiment. Therefore, the illumination environment database 24b stores the spatial distribution of the illumination light source including the direct light source and the reflected light source, and the illumination spectral distribution E ( λ). The spatial distribution of illumination light sources is used as information for defining geometric conditions such as the incident angle of illumination light with respect to the three-dimensional object 30 .

The illumination environment here is a combination of the illumination spectral distribution E(λ) and the spatial distribution of the illumination light source for reproducing the illumination effect in a specific scene for the three-dimensional object 30 . Specific examples of lighting environments include indoor environments where artificial direct light sources are applied, and outdoor environments where natural direct light sources are applied. It is divided into bands. Artificial direct light sources include, for example, incandescent lamps, fluorescent lamps, and LED lights, and natural direct light sources include, for example, the sun and the moon.

The illumination spectral distribution E (λ) and the spatial distribution of the illumination light source are obtained, for example, by acquiring omnidirectional spectral images in the actual illumination environment, and measuring every 5 nm in the visible wavelength range (eg, 400 to 700 nm). It can be obtained by sampling the spectral distribution in all directions. In this case, the illumination environment database 24b stores the spectral illumination distribution E(λ) for each pixel of the spectral image. Specifically, the position of each pixel in the spectral image serves as information specifying the spatial distribution of the illumination light sources distributed in the omnidirectional space around the three-dimensional object 30 . The spectral illumination distribution E(λ) assigned to each pixel becomes the spectral illumination distribution E(λ) of the illumination light with which the three-dimensional object 30 is irradiated. In other words, by regarding each pixel in the spectral image as an illumination light source and assigning the illumination spectral distribution E(λ) to each pixel, the spatial distribution of the illumination light source in all directions and the illumination of the illumination light emitted from each illumination light source can be obtained. A spectral distribution E(λ) can be obtained. According to such a method, in an actual lighting environment, the illumination light sources of the entire scene can be collectively processed without distinguishing between each direct light source and each reflected light source. Therefore, even when there are a plurality of illumination light sources whose boundaries of illumination light cannot be distinguished, it is possible to reproduce the illumination environment.

The cosmetic database 24c stores the second conversion function T(λ) for each cosmetic product. The second conversion function T(λ) is a conversion matrix for calculating the second spectral reflectance S ₂ (λ), which is the spectral reflectance of the cosmetic, based on the first spectral reflectance S ₁ (λ). be. The cosmetic here is a cosmetic applied to an object to be applied, and includes, for example, foundation, blush, eye color, lipstick, and the like. That is, the cosmetic database 24c stores the second conversion function T(λ) for each type of cosmetic, such as foundation and blush, and for each variation such as the color and application of each cosmetic.

More specifically, the cosmetic database 24c stores the second conversion function T(λ) according to the color of the application target for each cosmetic product. For example, if the cosmetic database 24c is a cosmetic that is applied to the skin, the second conversion data for each of the first type skin, the second type skin, and the third type skin is stored for each cosmetic product. Store the function T(λ).

For example, the second conversion function T(λ) is obtained by, for a large number of people, the spectral reflectance S _{a of the subject's application target and the spectral reflectance S b of} the subject's application target with the cosmetic applied. are actually measured, and statistically calculated for each color of the application target of the person to be measured from the amount of change in the spectral reflectance _Sa and the spectral reflectance _Sb .

In addition, the cosmetics database 24c stores reflection characteristic parameters of cosmetics for each cosmetic product. Although the details will be described later, the reflection characteristic parameter is a parameter that depends on the surface properties of an object such as the roughness of the object surface. It is a parameter that defines characteristics. The reflection characteristic parameter of the cosmetic can be obtained, for example, from actual measurements of the actual cosmetic using a measuring instrument such as a gloss meter.

In addition, the cosmetics database 24c may store spectral reflectance and reflection characteristic parameters of brightening agents such as pearls and glitters contained in cosmetics such as eye color. The spectral reflectance of the brightening agent is calculated using a reflection component that is independent of the spectral reflectance of the skin itself, because the color of the object to which the cosmetic is applied does not easily affect the color of the cosmetic. Therefore, calculation processing by the second conversion function T(λ) is not performed. Therefore, as the spectral reflectance of the brightening agent, an actual measurement value measured by a measuring device is stored.

The communication unit 25 communicates with the user terminal 10 via the network 3 . Specifically, the communication unit 25 receives a signal from the user terminal 10 and transmits data such as the first three-dimensional object 30 a and the second three-dimensional object 30 b generated by the server 20 to the user terminal 10 .

[GPU]
As shown in FIG. 4, the GPU 23 included in the control unit 21 includes an object generation unit 23a, a first spectral reflectance calculator 23b, a first color signal calculator 23c, a second spectral reflectance calculator 23d, a second color signal It functions as a calculator 23e, a third color signal calculator 23f, and a second RGB value calculator 23g.

The object generator 23a generates a first three-dimensional object 30a, which is the three-dimensional object 30 of the face without makeup, based on the first RGB values of the face image without makeup. The object generator 23a also generates a second three-dimensional object 30b, which is the three-dimensional object 30 of the face in the makeup state, based on the second RGB values calculated by the second RGB value calculator 23g, which will be described later.

The first spectral reflectance calculator 23b calculates the first spectral reflectance S ₁ (λ) based on the first RGB values and the first conversion function M. Based on a first three-dimensional reflection characteristic model, which is a three-dimensional reflection characteristic model of an object to be applied before makeup, the first color signal calculation unit 23c calculates a A first color signal C ₁ (λ), which is a color signal, is calculated.

The three-dimensional reflection characteristic model here is a mathematical model that describes the process of light reflection of an object, and the color and texture of the object are determined by the spectral reflectance and reflection characteristic parameters of the object that constitute the three-dimensional reflection characteristic model. Defined. Then, by giving the spectral distribution of the illumination light and geometric conditions such as the incident angle of the illumination light to the three-dimensional reflection characteristic model, color signals when the illumination light is reflected by the object are calculated. That is, the first three-dimensional reflection characteristic model is a mathematical model that describes the process of light reflection in an application target without makeup.

In addition, the color signal here is the spectral distribution (energy spectral distribution) of the electromagnetic wave that the illumination light reflects off an object and enters the human eye or visual system such as a camera. The color signal is converted into color information such as RGB values that humans actually perceive using an RGB color matching function or the like. Note that the first color signal C ₁ (λ) corresponds to a color signal when the illumination light is reflected on the application target without makeup.

The second spectral reflectance calculator 23d calculates a second spectral reflectance S ₂ (λ) based on the first spectral reflectance S ₁ (λ) and the second conversion function T(λ). The second color signal calculator 23e calculates a second color signal, which is a color signal when the illumination light is reflected on the cosmetic, based on a second three-dimensional reflection characteristic model, which is a three-dimensional reflection characteristic model of the cosmetic. Calculate C ₂ (λ). The second three-dimensional reflection characteristic model is a mathematical model describing the process of light reflection in cosmetics. In the following description, the reflection characteristic parameter for the non-makeup application object will be referred to as the first reflection characteristic parameter, and the reflection characteristic parameter for the cosmetic will be referred to as the second reflection characteristic parameter.

Based on the first color signal C ₁ (λ) and the second color signal C ₂ (λ), the third color signal calculator 23f calculates the third color signal C ₃ (λ), which is the color signal of the second three-dimensional object 30b. ) is calculated. The third color signal C ₃ (λ) corresponds to a color signal when the illumination light is reflected on the applied object in the makeup state, that is, the applied object and the cosmetic applied to the applied object. . The second RGB value calculator 23g calculates second RGB values, which are the RGB values of the second three-dimensional object 30b, based on the third color signal C ₃ (λ) calculated by the third color signal calculator 23f. The second RGB values correspond to the RGB values of the coated object in the makeup state.

[App image]
As shown in FIG. 5, the touch panel 13 of the user terminal 10 displays an application image 40, which is an application image displayed by the three-dimensional object display program. The application image 40 has a three-dimensional object display area 41 and an operation area 42 . The three-dimensional object display area 41 is an area in which the first three-dimensional object 30a or the second three-dimensional object 30b is displayed.

The operation area 42 includes, for example, a switching object 42a, a makeup area selection object 42b, a cosmetics selection object 42c, a lighting selection object 42d, and an operation object 42e. The switching object 42a is an object that switches between ON and OFF of the simulation result for each cosmetic.

The makeup area selection object 42b is an object for manually specifying the makeup area 31, which is the area of the three-dimensional object 30 to which cosmetics are applied. As an example of a method of specifying the makeup area 31, in the first three-dimensional object 30a displayed in the three-dimensional object display area 41, an area to be applied with cosmetics is specified by tap operation input. As the makeup area 31, for example, the entire facial skin is designated for foundation or blush, the skin around the eyes is designated for eye color, and the lips are designated for lips. Note that the makeup area 31 may be configured to be automatically set using an image recognition function or the like.

The cosmetic selection object 42c is an object for selecting a cosmetic product to be simulated. As shown in FIG. 6, a cosmetic selection area 43 is displayed in the three-dimensional object display area 41 by tapping the cosmetic selection object 42c. In the cosmetics selection area 43, cosmetics registered in the cosmetics database 24c are displayed for each product as product objects 43a. By tapping the product object 43a, a cosmetic product to be applied in the makeup simulation is selected.

As a specific example, in the case of foundation, the product object 43a of the desired product to be simulated is selected from among the product objects 43a displayed for each type of foundation, such as liquid foundation and powder foundation, and for each color.

The lighting selection object 42d is an object for selecting the lighting environment in the makeup simulation. A lighting selection area (not shown) is displayed in the three-dimensional object display area 41 by tapping the cosmetics selection object 42c. In the lighting selection area, various lighting environments registered in the lighting environment database 24b are displayed as lighting objects for each lighting environment. A lighting environment to be applied in the makeup simulation is selected by tapping the lighting object.

The operation object 42 e is an object for moving, rotating, enlarging, or reducing the first three-dimensional object 30 a or the second three-dimensional object 30 b displayed in the three-dimensional object display area 41 .

[Action of the first embodiment]
Next, the action of the makeup simulation system 1 will be described.
As shown in FIG. 7, in step S1, the control unit 11 of the user terminal 10 causes the imaging unit 14 to perform processing for acquiring a face image of the user 2 without makeup. Then, the control unit 11 executes a process of generating facial image data of the user 2 in the non-makeup state composed of the first RGB values and transmitting the data to the server 20 .

The face image acquired here may be, for example, a single image or a plurality of images taken at different angles, as long as the first three-dimensional object 30a can be generated. may be an image obtained from

Further, when photographing a face image in step S1, it is affected by various factors such as lighting in the photographing environment. From the viewpoint of acquiring a face image that is more suitable for makeup simulation, for example, as a correction for reducing the influence of the shooting environment, brightness and saturation are applied to the photographed face image by an image processing program or the like stored in the memory 12. You can adjust the degree.

In step S2, the object generation unit 23a of the server 20 generates the first three-dimensional object 30a based on the first RGB values forming the facial image of the user 2 without makeup transmitted from the user terminal 10. to run. Further, the object generation unit 23a transmits data of the generated first three-dimensional object 30a to the user terminal 10 and executes processing for displaying the data on the touch panel 13. FIG. Thereby, the first three-dimensional object 30 a is displayed in the three-dimensional object display area 41 of the application image 40 on the touch panel 13 .

Next, the user 2 performs a tap operation input on the object in the operation area 42 displayed on the touch panel 13, sets the makeup area 31 to which the cosmetic is applied, and further simulates the cosmetic product and lighting environment. to select. The control unit 11 of the user terminal 10 transmits to the server 20 a signal specifying the makeup area 31 set by the user 2 and the cosmetic product and lighting environment selected by the user 2 . It should be noted that, for example, a plurality of different types of cosmetics may be selected, such as blush and foundation.

In step S3, the first spectral reflectance calculator 23b executes a process of calculating the first spectral reflectance S ₁ (λ) in the makeup area 31 based on the first RGB values forming the first three-dimensional object 30a. do. Specifically, the first spectral reflectance calculator 23b reads the first conversion function M according to the color of the application target in the makeup area 31 from the spectral conversion database 24a, and calculates the _first spectral reflectance S1 in the makeup area 31. (λ) is calculated.

The first conversion function M read here may be automatically determined, for example, according to the first RGB values forming the makeup area 31 of the first three-dimensional object 30a, or the user 2 may select a color such as a color code. It may be determined by selecting a color close to the object to be coated from a sample.

The first spectral reflectance S ₁ (λ) uses the first conversion function M and R ₁ , G ₁ , and B ₁ , which are the values of each element in the first RGB values, in the visible wavelength range from 400 nm to 700 nm. It can be approximated as a discrete 61-dimensional spectral reflection vector s divided at intervals of 5 nm. The first spectral reflectance S ₁ (λ) can be expressed by the following formulas (1) and (2).

By calculating the first spectral reflectance S ₁ (λ) based on the first conversion function M corresponding to the color of the coating object, the first spectral reflectance S closer to the actual spectral reflectance of the coating object ₁ (λ) can be calculated. In addition, for example, when the application target is skin on the face, it is generally difficult to directly measure the spectral reflectance of the entire skin on the face because the measurement range of a measuring instrument such as a spectrophotometer is small. . On the other hand, as in the present embodiment, by calculating the first spectral reflectance S ₁ (λ) from the first RGB values of the face image in the non-makeup state, Spectral reflectance can be obtained.

Note that in the present embodiment, for example, in areas where cosmetics are not applied, such as hair and accessories such as pierced earrings, in areas other than the makeup area 31, the first spectral reflectance S ₁ (λ) is calculated from the first RGB values. no action is taken. This can simplify calculation processing when calculating the first spectral reflectance S ₁ (λ) from the first RGB values.

In step S4, the first color signal calculator 23c expresses with the first spectral reflectance S ₁ (λ), the first reflection characteristic parameter, the illumination spectral distribution E(λ) of the illumination light, and the geometric conditions of the illumination light. A process of calculating the first color signal C ₁ (λ) is executed based on the first three-dimensional reflection characteristic model obtained.

Specifically, the first color signal calculator 23c reads the illumination spectral distribution E(λ) and the spatial distribution of the illumination light source in the illumination environment selected by the user 2 from the illumination environment database 24b. Then, the first color signal calculator 23c uses the first three-dimensional reflection characteristic model to perform processing for calculating the first color signal C ₁ (λ).

The first three-dimensional reflection characteristic model uses the first spectral reflectance S ₁ (λ), the illumination spectral distribution E (λ), the first diffuse reflection function D _a , and the first specular reflection function G _a as follows: can be expressed by the following formula (3). Note that the first diffuse reflection function D _a is a function that defines the diffuse reflection in the application target without makeup. The first specular reflection function Ga is _a function that defines the specular reflection on the application target without makeup. In the first three-dimensional reflection characteristic model, the first term on the right side represents the diffuse reflection component, and the second term on the right side represents the specular reflection component.

Here, the geometric model of reflection in this embodiment will be described with reference to FIGS. 8 and 9. FIG. As shown in FIG. 8, the virtual space 50 includes a virtual point 51, a virtual light source 52, and a virtual viewpoint 53. FIG. A virtual point 51 is a point on the surface S of the first three-dimensional object 30 a that receives light from the virtual light source 52 . The virtual light source 52 irradiates the virtual point 51 with illumination light. A virtual viewpoint 53 is a viewpoint from which the virtual point 51 is observed.

Also, the illumination direction vector L is a vector directed from the virtual point 51 to the virtual light source 52 . A normal vector N is a vector in the normal direction at the virtual point 51 on the surface S. A line-of-sight direction vector V is a vector directed from the virtual point 51 to the virtual viewpoint 53 . The illumination direction vector L, normal vector N, and line-of-sight direction vector V are parameters for defining geometric conditions of illumination light in the three-dimensional reflection characteristic model. The angle formed by the illumination direction vector L and the normal vector N is defined as the incident angle _θi , and the angle formed between the normal vector N and the line-of-sight direction vector V is defined as the light receiving angle _θr .

In general, when an object receives illumination light, the color signal C ₀ (λ) generated on the surface of the object is obtained by combining the spectral reflectance S ₀ (λ) of the object and the spectral distribution E ₀ (λ) of the illumination light. is expressed as C ₀ (λ)=E ₀ (λ)S ₀ (λ).

In this embodiment, in addition to simply calculating the color signal of the object from the spectral reflectance of the object and the spectral distribution of the illumination light, by using a three-dimensional reflection characteristic model that takes into account diffuse reflection and specular reflection on the surface of the object, It reproduces the gloss and texture of the object surface more accurately.

Specifically, when the virtual point 51 receives the light from the virtual light source 52, the diffusely reflected light that occurs in all directions at the virtual point 51 and the specular reflection that occurs when the incident angle _θi and the light receiving angle _θr approximate each other A reflected light composed of light is generated.

As shown in FIG. 9, the intensity distribution of the reflected light generated at the virtual point 51 is represented by the reflected light distribution RD, which is the intensity distribution obtained by adding the diffuse reflected light intensity α and the specular reflected light intensity I. The intensity of the reflected light generated at the virtual point 51 shows different values depending on the light receiving angle _θr . Specifically, the distance from the virtual point 51 to the reflected light distribution RD in FIG. 9 represents the intensity of the reflected light at the light receiving angle θr.

The diffuse reflection light intensity α is a reflection characteristic parameter for defining the intensity of diffuse reflection light. The diffuse reflected light intensity α is a substantially constant value regardless of the incident angle _θi and the light receiving angle _θr . The specular reflected light intensity I is a reflection characteristic parameter for defining the relative intensity of the specular reflected light with respect to the diffuse reflected light intensity α. The specular reflected light intensity _I increases as the value of the light receiving angle _θr approaches the incident angle θi, and reaches its maximum value when the incident angle _θi and the light receiving angle _θr match.

For example, when the light-receiving angle _θr is a light-receiving angle _θr1 that is significantly different from the incident angle _θi , the intensity of the reflected light observed at the virtual viewpoint 53a is the value of the diffuse reflected light intensity α. On the other hand, when the light-receiving angle _θr is a light-receiving angle _θr2 that approximates the incident angle _θi , the intensity of the reflected light observed at the virtual viewpoint 53b is the diffuse reflection light intensity α and the specular reflection light intensity I is the sum of

The first diffuse reflection function D _a can be expressed by the following equation (4) using the first diffuse reflection light intensity α ₁ and the illumination direction vector L and normal vector N as geometric conditions of the illumination light. can. The first diffusely reflected light intensity α1 is a _first reflection characteristic parameter for defining the intensity of the diffusely reflected light on the application target without makeup. Here, the first diffuse reflection light intensity α ₁ =1 is treated. Note that the geometric conditions of the illumination light used in the following calculations are calculated from the spatial distribution of the illumination light source, the shape of the three-dimensional object 30, the viewpoint position for observing the three-dimensional object 30, and the like.

Further, in the reflected light distribution RD, the range in which specularly reflected light occurs is represented by a specularly reflected parameter m. The specular reflection parameter m is a parameter that depends on the surface properties of an object such as the roughness of the object surface, and is a reflection characteristic parameter that defines the spread of gloss. The larger the specular reflection parameter m, the larger the range in which the specular reflection occurs.

The first specular reflection function G _a is composed of the first specular reflection light intensity I1 _, the _first specular reflection parameter m1, the illumination direction vector L, the normal vector N, and the viewing direction as geometric conditions of the illumination light. Using vector V, it can be represented by the following equations (5) to (7). Equation (6) is a Beckmann function. Also, the first specular reflected light intensity I1 is a _first reflection characteristic parameter for defining the intensity of the diffusely reflected light on the non-makeup application object. The first specular reflection parameter m1 is a first reflection characteristic parameter for defining the spread of gloss on an unmade _- up application target. Arbitrary values are set in advance for the _first specular reflection light intensity I1 and the _first specular reflection parameter m1.

Note that the first specular reflection light intensity I1 and the _first specular reflection parameter m1 can also be obtained, for example, from _a plurality of images obtained by changing the incident angle of the illumination light for the face of the user 2 without makeup. can. For example, in step S1, when acquiring the face image of the user 2 without makeup, by acquiring a plurality of images obtained by changing the incident angle of the illumination light with respect to the face of the user 2, the first specular reflection The light intensity I1 and the _first specular parameter _m1 can also be determined.

In this way, by calculating the first color signal C ₁ (λ) using the first three-dimensional reflection characteristic model, the color of the application target in the non-makeup state, and the shadow and gloss caused by the illumination light can be reproduced in the first order. It can be reproduced as a one-color signal C ₁ (λ).

In step S5, the second spectral reflectance calculator 23d calculates the _{second spectral reflectance S 2 (} λ) is calculated. Specifically, the second spectral reflectance calculator 23d reads out the second conversion function T(λ) according to the color of the application target in the makeup area 31 from the cosmetics database 24c, and calculates the second spectral reflection in the makeup area 31. A process for calculating the ratio S ₂ (λ) is executed.

Note that the second conversion function T(λ) read here may be determined together with the first conversion function M in step S3, for example, and the user 2 may use a color sample such as a color code to may be determined by choosing a color close to The second spectral reflectance S ₂ (λ) can be expressed by Equation (8) below using the first spectral reflectance S ₁ (λ) and the second conversion function T(λ).

By calculating the second spectral reflectance S ₂ (λ) based on the first spectral reflectance S ₁ (λ) and the second conversion function T (λ) corresponding to the color of the object to be coated, the actual It is possible to calculate the second spectral reflectance S ₂ (λ) that is closer to the spectral reflectance of the cosmetic in the state where the cosmetic is applied to the application target and has become familiar.

In step S6, the second color signal calculator 23e expresses the second spectral reflectance S ₂ (λ), the second reflection characteristic parameter, the illumination spectral distribution E(λ) of the illumination light, and the geometric conditions of the illumination light. A second color signal C ₂ (λ) is calculated based on the obtained second three-dimensional reflection characteristic model.

Specifically, the second color signal calculator 23e reads the second reflection characteristic parameter of the cosmetic selected by the user 2 from the cosmetic database 24c. Further, the second color signal calculator 23e reads the illumination spectral distribution E(λ) and the spatial distribution of the illumination light source in the illumination environment selected by the user 2 from the illumination environment database 24b. Then, the second color signal calculation unit 23e uses the second three-dimensional reflection characteristic model to execute the process of calculating the second color signal C ₂ (λ).

The second three-dimensional reflection characteristic model uses the second spectral reflectance S ₂ (λ), the illumination spectral distribution E (λ), the second diffuse reflection function D _b , and the second specular reflection function G _b as follows: can be represented by the following formulas (9) to (11). The second diffuse reflection function _Db is a function that defines diffuse reflection in cosmetics. The second specular reflection function G _b is a function that defines specular reflection in cosmetics. Also, B(N, V, L, m ₂ ) included in Equation (11) is a Beckmann function. In Equation (9), the first term on the right side represents the diffuse reflection component, and the second term on the right side represents the specular reflection component.

In the second three-dimensional reflection characteristic model, the _second diffuse reflection light intensity α2 is a parameter that is set for each cosmetic product, and is the second reflection characteristic for defining the intensity of the diffuse reflection light of the cosmetic. is a parameter. Here, the second diffuse reflected light intensity α ₂ =1 is assumed. The second specular reflected light intensity _I2 is a parameter set for each cosmetic product, and is a second reflection characteristic parameter for defining the intensity of the specular reflected light of the cosmetic. The second specular reflection parameter _m2 is a parameter set for each cosmetic product, and is a second reflection characteristic parameter for defining the spread of gloss of the cosmetic.

Further, when a plurality of cosmetics such as foundation and blush are selected as the cosmetics to be simulated, in step S6, the second color signal calculator 23e calculates the second color signal C ₂ ( λ) is calculated.

In this way, by calculating the second color signal C ₂ (λ) using the second three-dimensional reflection characteristic model, the color of the cosmetic and the shadow and gloss caused by the illumination light can be obtained by the second color signal C ₂ . (λ).

In step S6, the second color signal calculation unit 23e calculates a third three-dimensional reflection property model, which is a three-dimensional reflection property model of the luster agent, for the luster agent such as pearl and glitter contained in cosmetics such as eye color. can also be used to calculate the color signal C _G (λ) of the glitter. In this case, the second color signal C ₂ (λ) for the base portion of the cosmetic containing the brightening agent, that is, the portion of the cosmetic other than the brightening agent is calculated by the second three-dimensional reflection characteristic model, and the color of the brightening agent is calculated. A signal C _G (λ) is calculated by the third three-dimensional reflection characteristic model.

As shown in FIG. 10, the brightening agent 60 contained in the cosmetic has a multilayer film structure in which a plurality of thin films 61 are laminated. When the light from the virtual light source 52 is applied to the brightening agent 60 , interference light depending on the thickness d of each thin film 61 is observed at the virtual viewpoint 53 . Therefore, by calculating the color signal C _G (λ) of the glittering agent for the interference light from such a multilayer film, the interference color of the glittering agent such as rainbow colors can be reproduced. Information about the multi-layer film structure such as the number and thickness d of the thin films 61 in the brightening agent 60 is stored in the cosmetics database 24c.

The third three-dimensional reflection characteristic model uses the spectral reflectance S _G of the glittering agent, the reflection characteristic parameter of the glittering agent, the illumination spectral distribution E (λ), and the geometric conditions of the illumination light, and the following formula (12) , (13). In the third three-dimensional reflection characteristic model, the color signal C _G (λ) is calculated only for the specular reflection component of the brightening agent, and the diffuse reflection component is ignored. Also, the glitter agent has a spectral reflectance _SG that depends on the geometric conditions of the illumination light. B(N, V, L, m _G ) included in equation (13) is the Beckmann function.

The glitter specular function _GG is the function that defines the specular reflection in the glitter. The specular reflected light intensity _IG is a reflection characteristic parameter of a glitter, and is a parameter that defines the intensity of specular reflected light for each glitter. The specular reflection parameter _mG is a reflection characteristic parameter of glitter, and is a parameter that defines the spread of gloss for each glitter.

Further, the process of calculating the color signal C _G (λ) of the brightening agent is executed in a region in which the brightening agent is scattered within the makeup region 31 . Specifically, the area in which the brightening agent is scattered in the makeup area 31 contains shape information that defines the particle shape size of the brightening agent, and the amount of the brightening agent in the area where the cosmetic is actually applied. It can be estimated based on dispersion information that defines the dispersion state. By estimating areas where glittering agents are scattered in the makeup area 31 and calculating the color signal C _G (λ) of the glittering agent in the area, the grain shape of the glittering agent contained in the actual cosmetic and The dispersed state of glittering agents can be reproduced. The shape information and dispersion information of the brightener are stored in advance in the cosmetics database 24c.

When calculating the color signal C _G (λ) of the brightening agent, correction may be performed to increase the spectral intensity at a specific wavelength in the color signal C _G (λ) of the brightening agent. Further, the third three-dimensional reflection characteristic model may include parameters such as an orientation parameter that defines the orientation of the brightening agent and the thickness d of each thin film 61 .

In step S7, the third color signal calculator 23f calculates the third color signal, which is the color signal of the second three-dimensional object 30b, based on the first color signal C ₁ (λ) and the second color signal C ₂ (λ). A process for calculating C ₃ (λ) is executed. Note that in a region where a plurality of cosmetics are superimposed in the makeup region 31, the third color signal C ₃ (λ) is calculated by adding the second color signals C ₂ (λ) of the respective cosmetics. be done.

As a specific example, the third color signal calculator 23f calculates the first color signal C ₁ (λ), the second color signal C _2f (λ), C _2c (λ) and the following equation (14) using weighting factors w ₁ to w ₃ to calculate the third color signal C ₃ (λ). The second color signal C _2f (λ) is the color signal of foundation, which is an example of cosmetics. The second color signal C _2c (λ) is the color signal of cheek, which is an example of cosmetics. The weighting coefficients w ₁ to w ₃ are coefficients for weighting each color signal, and are appropriately determined according to the thickness of the cosmetics in the simulation, the order of applying the cosmetics in layers, and the like.

Thus, by calculating the third color signal C ₃ (λ) based on the first color signal C ₁ (λ) and the second color signal C ₂ (λ), the color signal , the third color signal C ₃ (λ) in which the color of the object to be applied is reflected in the cosmetic can be calculated.

Even when simulating a state in which a plurality of cosmetics are superimposed, the _{third color signal C 3 ( λ} ), it is possible to calculate a color signal in which the color of the object to be applied and the color of the underlying cosmetic are reflected in the cosmetic on the outermost surface.

_In step S7, the _third color signal calculator _23f calculates the , to calculate the third color signal C ₃ (λ).

As a specific example, the third color signal calculation unit 23f calculates a third A process for calculating the color signal C ₃ (λ) is executed. The second color signal C _2e (λ) is the color signal of the portion of the eye color excluding the brightening agent. A color signal C _Gp (λ) is a color signal of pearl, which is an example of a brightening agent contained in eye color. The color signal C _Gl (λ) is the color signal of lame, which is an example of glittering agent contained in eye color.

In this way, by calculating the makeup state color signal based on the first color signal C ₁ (λ), the second color signal C ₂ (λ), and the brightening agent color signal C _G (λ), It is possible to calculate the third color signal C ₃ (λ) reflecting the color and gloss of the brightener contained in the cosmetic.

In step S8, the second RGB value calculator 23g calculates the second RGB values, which are the RGB values of the second three-dimensional object 30b, based on the third color signal C ₃ (λ) calculated by the third color signal calculator 23f. Execute the calculation process. Specifically, the second RGB value calculation unit 23g executes processing for calculating the second RGB values based on the following equation (16) using the third color signal C ₃ (λ) and RGB color matching functions. Note that R ₂ , G ₂ , and B ₂ on the left side of Equation (16) are the values of the respective elements in the second RGB values.

In step S9, the object generation unit 23a executes processing for generating the second three-dimensional object 30b based on the second RGB values calculated by the second RGB value calculation unit 23g. Further, the object generation unit 23a transmits data of the generated second three-dimensional object 30b to the user terminal 10 and executes processing for displaying the data on the touch panel 13. FIG.

As shown in FIG. 11 , the second three-dimensional object 30 b is displayed in the three-dimensional object display area 41 of the application image 40 on the touch panel 13 . In the makeup area 31 of the second three-dimensional object 30b, the color of the skin, which is an example of the object to be applied, the color of the cosmetic used as the base, and the skin in the makeup state reflecting the shadows of the illumination light are displayed. . In addition, in the glossy area 32, which is an area in which the light receiving angle _θr approximates the incident angle _θi in the decorative area 31, a state in which gloss due to specular reflection is reproduced is displayed. With the above processing, the makeup simulation is completed.

By selecting another cosmetic while the second three-dimensional object 30b is displayed, the processes of steps S3 to S8 are executed again, and the second three-dimensional object 30b to which another cosmetic is applied is displayed. can be generated and displayed. At this time, for example, the processing of steps S3 and S4 may be omitted by storing the once calculated first color signal C ₁ (λ) in the main memory of the memory 24 . In this way, by repeatedly simulating various cosmetics, it is possible to search for cosmetics that meet one's wishes.

By calculating the first color signal C ₁ (λ) and the second color signal C ₂ (λ) based on the illumination spectral distribution E(λ) for each illumination environment and the spatial distribution of the illumination light source, the first 2. The lighting environment is also reflected in the three-dimensional object 30b. Therefore, it is possible to perform a makeup simulation that reflects lighting effects of various actual lighting environments. Further, by using the illumination spectral distribution E(λ) calculated from the measured values of the spectral distribution in the actual illumination environment, various illumination environments can be easily reproduced.

The processing of steps S2 to S8 is executed by the GPU 23 provided in the server 20, so that the processing of steps S2 to S8 can be performed at high speed, enabling real-time makeup simulation.

Further, for example, by using the makeup simulation system 1 to simulate various cosmetics for the user 2, it is also possible to provide counseling for proposing the optimum cosmetics that meet the needs of the user 2.

[Effect of the first embodiment]
According to the said 1st Embodiment, the effect enumerated below can be acquired.
(1-1) Calculate the first color signal C ₁ (λ) using the first three-dimensional reflection characteristic model, and calculate the second color signal C ₂ (λ) using the second three-dimensional reflection characteristic model. Thus, the third color signal C ₃ (λ) in which the color of the application target is reflected in the cosmetic can be calculated as the color signal of the application target in the makeup state. Then, by calculating the second RGB values from the third color signal C ₃ (λ), it is possible to generate the second three-dimensional object 30b including the application target in the makeup state. As a result of the simulation, it is possible to obtain the second three-dimensional object 30b in a makeup state in which the color of the application target is reflected in the cosmetics. Therefore, makeup simulation can be performed with higher reproducibility than when simply superimposing the color of the cosmetic on the area of the application target. In addition, since the luster of the illumination light can be reproduced, the texture of the second three-dimensional object 30b can be reproduced in a state where the cosmetic is applied to the application target.

(1-2) By calculating the first spectral reflectance S ₁ (λ) based on the first conversion function M corresponding to the color of the skin and lips as the application target, the spectrum of the actual skin and lips A first spectral reflectance S ₁ (λ) close to the reflectance can be calculated.

(1-3) Calculate the second spectral reflectance S ₂ (λ) based on the first spectral reflectance S ₁ (λ) and the second conversion function T(λ) corresponding to the color of the object to be coated. Thus, it is possible to calculate the second spectral reflectance S ₂ (λ) that is closer to the spectral reflectance of the cosmetic in the state where the cosmetic is actually applied to the application target and is familiar with the application target.

(1-4) By calculating the second color signal C ₂ (λ) using the second spectral reflectance S ₂ (λ) and the second reflection characteristic parameter for each cosmetic product, the cosmetic A second three-dimensional object 30b can be generated that reflects the color and reflection characteristics of each product.

(1-5) By using the third three-dimensional reflection characteristic model, it is possible to calculate the color signal C _G (λ) of the brightener contained in the cosmetic. Then, from the first color signal C ₁ (λ), the second color signal C ₂ (λ), and the third color signal C ₃ (λ) calculated based on the color signal C _G (λ) of the luster agent, By calculating the second RGB values, it is possible to generate the second three-dimensional object 30b that reproduces the luster of the glitter in the makeup state. In addition, by calculating the color signal _CG (λ) of the brightening agent in the area where the brightening agent is estimated from the shape information and the dispersion information, the particle shape and the brightness of the brightening agent contained in the actual cosmetic are obtained. The dispersed state of the agent can be reproduced.

(1-6) By calculating the first color signal C ₁ (λ) and the second color signal C ₂ (λ) based on the illumination spectral distribution E(λ) for each illumination environment and the spatial distribution of the illumination light source, The lighting environment is also reflected in the second three-dimensional object 30b as a simulation result. Therefore, it is possible to perform makeup simulation with high reproducibility that reflects the lighting environment.

(1-7) By using the GPU 23 to execute the processing of steps S2 to S8 in the makeup simulation system 1, image processing can be performed in real time, and makeup simulation can be performed more favorably.

(1-8) By using the makeup simulation system 1 to simulate various cosmetics for the user 2, it is also possible to provide counseling for proposing the optimum cosmetics that meet the user's 2 needs. .

[Second embodiment]
In the first embodiment, the third color signal C ₃ (λ) is calculated based on the first color signal C ₁ (λ) and the second color signal C ₂ (λ) calculated by spectral reflectance-based calculation processing. In addition, the example of executing the process of calculating the second RGB values based on the third color signal C ₃ (λ) has been described. Instead of the above process, the second RGB values can also be obtained by performing an RGB value-based calculation process. As a second embodiment, the operation of performing calculation processing based on RGB values will be described below.

Note that the RGB value-based calculation processing for obtaining the second RGB values is performed in a region of the makeup region 31 that does not require high reproduction accuracy. That is, in areas where high reproducibility is required, the second RGB values are calculated by the spectral reflectance-based calculation processing described in the first embodiment. Further, in the second embodiment, the RGB value-based calculation processing for obtaining the second RGB values is executed by the second RGB value calculation section 23g as the processing after step S2 shown in FIG.

First, the second RGB value calculator 23g executes a process of calculating third RGB values based on the first RGB values, the first diffuse reflection function D _a , the first specular reflection function G _a , and the RGB values of the illumination light. do. The third RGB value is the RGB value of the coating object in a state in which lighting effects such as shadow and gloss are reflected by irradiating the coating object with illumination light. The third RGB values R ₃ , G ₃ , B ₃ can be represented by the following equation (17). Also, the RGB values R _L , G _L , and B _L of the illumination light can be expressed by the following equation (18) using the illumination spectral distribution E(λ) and RGB color matching functions. Note that the RGB values of the illumination light are stored in the illumination environment database 24b for each illumination environment.

Next, the second RGB value calculator 23g executes a process of calculating fourth RGB values based on the first RGB values and the cosmetic conversion coefficients. The fourth RGB value is the RGB value of the cosmetic in a state where the cosmetic is applied to the application target and is familiar with the application target. That is, the fourth RGB values do not reflect lighting effects such as shadows and gloss caused by the lighting light in the lighting environment selected in step S2. Also, the cosmetic conversion coefficients are coefficients calculated based on the second conversion function T(λ), and are coefficients for converting the first RGB values into the fourth RGB values. Cosmetic conversion coefficients are stored in advance in the cosmetic database 24c for each cosmetic. The fourth RGB values R ₄ , G ₄ , B ₄ can be represented by the following equation (19). Also, the cosmetic transformation coefficients _RT, GT, _BT can be expressed by the following equation (20) using the second transformation function _T (λ) and the RGB color matching function.

Next, the second RGB value calculator 23g executes a process of calculating fifth RGB values based on the fourth RGB values, the second diffuse reflection function D _b , the second specular reflection function G _b , and the RGB values of the illumination light. do. The fifth RGB value is the RGB value of the cosmetic that reflects illumination effects such as shadows and gloss by irradiating the cosmetic with illumination light. The fifth RGB values R ₅ , G ₅ , B ₅ can be represented by the following equation (21).

Then, the second RGB value calculation unit 23g executes processing for calculating the second RGB values based on the third RGB value, the fifth RGB value, and the weighting factor w. As a specific example, the second RGB value calculation unit 23g performs a process of calculating the second RGB values based on the following formula (22) in the area where the foundation and cheeks are superimposed in the makeup area 31. R _5f , G _5f , B _5f are the fifth RGB values of the foundation. R _5c , G _5c , B _5c are the fifth RGB values of cheeks.

It should be noted that it is difficult to perform calculation processing based on RGB values for brightening agents contained in cosmetics. Therefore, the sixth RGB values that reproduce the interference color of the brightening agent are obtained by calculating the color signal C _G (λ) of the brightening agent based on the spectral reflectance using the above equation (12), and then calculating the color signal of the brightening agent. It is calculated from C _G (λ) and RGB color matching functions. Further, the process of calculating the sixth RGB values is executed by the second RGB value calculator 23g. The sixth RGB values R ₆ , G ₆ , B ₆ can be represented by the following equation (23).

Then, the second RGB value calculation unit 23g calculates the second RGB values in the state where the cosmetic containing the brightening agent is applied, based on the third RGB value, the fifth RGB value, the sixth RGB value, and the weighting factor w. can also be executed. As a specific example, the second RGB value calculation unit 23g calculates the second RGB values based on the following formula (24) in the makeup region 31, in the region where the foundation and the eye color including pearl and glitter as glitter agents are superimposed. A process for calculating R ₂ , G ₂ and B ₂ is executed. R _5e , G _5e , and B _5e are the fifth RGB values of the portion of the eye color excluding the brightening agent. R _6p , G _6p , and B _6p are the sixth RGB values of pearl, which is an example of brightener contained in eye color. R _6l , G _6l , and B _6l are the sixth RGB values of lame, which is an example of glittering agent contained in eye color.

The second RGB values calculated by the RGB value-based calculation processing as described above are used by the object generation unit 23a to execute processing for generating the second three-dimensional object 30b in step S9 shown in FIG. .

When calculating the second RGB values by RGB value-based calculation processing, three-dimensional calculation processing is performed for the three colors of red, green, and blue. On the other hand, when calculating the second RGB values based on the color signal calculated from the spectral reflectance as in the first embodiment, the 61-dimensional calculation processing is performed by dividing the visible light region of 400 nm to 700 nm by 5 nm. done.

That is, by calculating the second RGB values by the RGB value-based calculation process, the calculation process in the makeup simulation can be simplified more than the process of calculating the second RGB values based on the color signal calculated from the spectral reflectance. Therefore, by performing calculation processing based on RGB values in the makeup region 31 where high reproduction accuracy is not required, the calculation processing can be simplified without significantly lowering the reproduction accuracy of the makeup simulation.

[Effect of Second Embodiment]
According to the said 2nd Embodiment, the following effects can be acquired.
(2-1) Calculating the second RGB values by the RGB value-based calculation process can simplify the calculation process in the makeup simulation compared to the process of calculating the second RGB values based on the color signal calculated from the spectral reflectance. . Therefore, by performing calculation processing based on RGB values in the makeup region 31 where high reproduction accuracy is not required, the calculation processing can be simplified without significantly lowering the reproduction accuracy of the makeup simulation.

In addition, the above-described second embodiment can also be implemented with appropriate modifications as follows.
- The configuration in which the second RGB values are calculated based on the third RGB values and the fifth RGB values calculated using the RGB value-based calculation process has been exemplified. However, without being limited to this, either the third RGB value or the fifth RGB value may be obtained by spectroscopic-based calculation. Specifically, values obtained by converting the first color signal C ₁ (λ) into RGB values using an RGB color matching function may be used instead of the third RGB values. Also, instead of the fifth RGB values, values obtained by converting the second color signal C ₂ (λ) into RGB values using an RGB color matching function may be used. According to such a configuration, only arbitrary calculation processing that does not require precision can be calculated based on RGB values, so that the calculation processing of makeup simulation can be simplified more preferably.

In addition, the first embodiment and the second embodiment can be modified and implemented as described below.
In the above-described first and second embodiments, the face is the target for the makeup simulation, and the application target to which the cosmetic is applied is the skin or lips of the face. However, the present makeup simulation system 1 is not limited to this, and can be applied to areas other than the face as long as cosmetics are applied. The object to which the cosmetic is applied may be, for example, hair. In that case, the applied cosmetic is, for example, hair manicure. Also, the object to which the cosmetic is applied may be, for example, a nail. In that case, the applied cosmetic is, for example, nail polish.

The configuration in which the GPU 23 is used to execute the processing of steps S2 to S8 in the makeup simulation system 1 has been exemplified, but the present invention is not limited to this. may be executed. In this case, although the processing speed of steps S2 to S8 in the server 20 is reduced, the GPU 23 can be omitted. Further, for example, the CPU 22 may execute any one of the processes of steps S2 to S8 depending on the performance of the CPU 22. FIG.

- The configuration in which the first color signal C ₁ (λ) and the second color signal C ₂ (λ) are calculated based on the illumination spectral distribution E(λ) for each illumination environment and the spatial distribution of the illumination light source has been exemplified. However, it is not limited to this, and for example, the first color signal C ₁ (λ) and the second color signal C ₂ (λ) are obtained using only one type of illumination spectral distribution E(λ) and the spatial distribution of the illumination light source. It may be configured to calculate. _In this case, although the makeup simulation for _each lighting environment cannot be performed, the procedure for selecting the lighting environment is not required. Calculation processing is simplified. Further, for example, if a spectral distribution having a uniform intensity in the visible light region of 400 to 700 nm is used as the spectral illumination distribution E (λ), the spectral illumination distribution E ( λ) can be omitted.

・At step S5, a second spectral reflectance S ₂ (λ) is calculated based on the first spectral reflectance S ₁ (λ) and a second conversion function T(λ) corresponding to the color of the object to be coated. The configuration is illustrated. However, it is not limited to this, and the spectral reflectance of the cosmetic is actually measured using a measuring instrument such as a spectrophotometer, and the measured value of the spectral reflectance of the cosmetic is used as the second spectral reflectance S ₂ (λ). may In this case, the reproduction of the second color signal C ₂ is better than when the second spectral reflectance S ₂ (λ) is calculated based on the first spectral reflectance S ₁ (λ) and the second conversion function T(λ). Although the accuracy is slightly reduced, it is possible to easily perform makeup simulations for various cosmetics. For example, when a new cosmetic is developed, the makeup simulation system 1 can perform makeup simulation by actually measuring the second spectral reflectance and the second reflection characteristic parameter of the new cosmetic. In this case, the process of step S5 is omitted.

- The configuration for calculating the first spectral reflectance S ₁ (λ) based on the first conversion function M corresponding to the color of the application target is exemplified. However, the present invention is not limited to this. For example, the first spectral reflectance S ₁ ( λ) may be calculated. In this case, although the difference between the second three-dimensional object 30b as a simulation result and the actual appearance after makeup is slightly increased, the first conversion function M or the procedure for the user 2 to select a color close to the object to be coated from the color sample. Therefore, the process of calculating the first spectral reflectance S ₁ (λ) from the first RGB values can be simplified.

- Although the configuration for acquiring the face image of the user 2 using the imaging unit 14 provided in the user terminal 10 is illustrated, it is not limited to this. A three-dimensional object 30a may be generated.

- The configuration in which the object generation unit 23a executes the process of generating the first three-dimensional object 30a from the face image of the user 2 without makeup has been exemplified. However, the present invention is not limited to this, and for example, a three-dimensional object of a non-makeup face generated by other software may be imported and used as the first three-dimensional object 30a. In this case, the RGB values forming the imported three-dimensional object are the first RGB values.

- In the present embodiment, the GPU 23 included in the server 20 is used to execute the processing of steps S2 to S8 in the makeup simulation system 1 as an example. However, without being limited to this, for example, a GPU is provided in the control unit 11 of the user terminal 10, a makeup simulation program is installed in the user terminal 10, and the processing of steps S2 to S8 is executed by the GPU provided in the user terminal 10. may

The makeup simulation system 1 may be configured to be able to execute a process of purchasing cosmetics applied to the second three-dimensional object 30b as a simulation result, for example, using a three-dimensional object display program. In this case, the server 20 is provided with a payment unit that executes payment processing. As a result, for example, after searching for cosmetics that meet one's wishes in the makeup simulation system 1, the cosmetics can be purchased.

DESCRIPTION OF SYMBOLS 1... Makeup simulation system 10... User terminal 11... Control part 12... Memory 13... Touch panel 14... Imaging part 20... Server 21... Control part 23... GPU
23a... Object generator 23b... First spectral reflectance calculator 23c... First color signal calculator 23d... Second spectral reflectance calculator 23e... Second color signal calculator 23f... Third color signal calculator 23g... Third 2RGB value calculator 24 Memory 24a Spectral conversion database 24b Lighting environment database 24c Cosmetic database 30 Three-dimensional object 30a First three-dimensional object 30b Second three-dimensional object 31 Makeup area 32 Glossy area 40 … app image

Claims (11)

a first spectral reflectance calculation unit that calculates a first spectral reflectance of the application target before makeup from first RGB values before makeup of the application target on which the cosmetic is applied;
Using a first three-dimensional reflection characteristic model represented by the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristic of the application target before the makeup, and the spectral distribution and geometric conditions of the illumination light a first color signal calculation unit that calculates a first color signal in the application target before the makeup;
using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristic of the cosmetic, the spectral distribution of the illumination light, and the geometric condition; a second color signal calculation unit for calculating a second color signal of the cosmetics;
a third color signal calculation unit that calculates a third color signal of the application object in a makeup state in which the cosmetic is applied to the application object from the first color signal and the second color signal;
a second RGB value calculation unit that calculates second RGB values of the application target in the makeup state from the third color signal;
a makeup simulation system comprising: an object generation unit that generates a three-dimensional object including the application target in the makeup state from the second RGB values. a spectral conversion database storing a first conversion function for converting the first RGB values into the first spectral reflectance, the first conversion function corresponding to the color of the object to be applied before the makeup; prepared,
The makeup simulation system according to claim 1, wherein the first spectral reflectance calculator calculates the first spectral reflectance using the first conversion function according to the color of the application target before the makeup. a second conversion function for calculating the second spectral reflectance from the first spectral reflectance, the cosmetic database storing the second conversion function according to the color of the application target before the makeup; When,
a second spectral reflectance calculator that calculates the second spectral reflectance using the first spectral reflectance and the second conversion function corresponding to the color of the application target before the makeup. ,
3. The second color signal calculation unit calculates the second color signal using the second three-dimensional reflection characteristic model including the second spectral reflectance calculated by the second spectral reflectance calculation unit. The makeup simulation system described in . The cosmetic database stores the second conversion function and the second reflection characteristic parameter for each cosmetic product,
The second spectral reflectance calculation unit calculates the second spectral reflectance using the second conversion function for each cosmetic product,
The second color signal calculator calculates the second three-dimensional reflection including the second spectral reflectance calculated by the second spectral reflectance calculator and the second reflection characteristic parameter for each cosmetic product. The makeup simulation system according to claim 3, wherein the second color signal is calculated using a characteristic model. The cosmetic database includes spectral reflectance of a glittering agent contained in the cosmetic, a reflection characteristic parameter of the glittering agent that defines the reflection characteristics of the glittering agent, and shape information that defines the size of the glittering agent. , and dispersion information that defines the dispersion state of the glitter agent, and
The second color signal calculation unit calculates the second color signal of the cosmetic using the second three-dimensional reflection characteristic model, and further estimates the second color signal from the shape information and the dispersion information of the brightening agent. A third three-dimensional reflection characteristic model expressed by the spectral reflectance of the glitter agent, the reflection characteristic parameter of the glitter agent, the spectral distribution of the illumination light, and the geometric condition in the region where the glitter agent is scattered Calculate the color signal of the brightening agent using
The third color signal calculation unit applies the cosmetic containing the glitter to the application target before the makeup from the first color signal, the second color signal, and the color signal of the glitter. The makeup simulation system according to claim 4, wherein the third color signal for the application target in the makeup state is calculated. 3. The makeup simulation system according to claim 1, wherein at least one of said second spectral reflectance and said second reflection characteristic parameter is a measured value actually measured for said cosmetic. A lighting environment that stores, for each lighting environment, the spectral distribution of the illumination light and the spatial distribution of an illumination light source that generates the illumination light, the spatial distribution in all directions centering on the three-dimensional object. further equipped with a database,
The first color signal calculation unit calculates the first color signal using the first three-dimensional reflection characteristic model including the spectral distribution of the illumination light for each lighting environment,
7. The second color signal calculator calculates the second color signal using the second three-dimensional reflection characteristic model including the spectral distribution of the illumination light for each illumination environment. 1. The makeup simulation system according to claim 1. at least one of the first spectral reflectance calculator, the first color signal calculator, the second color signal calculator, the third color signal calculator, the second RGB value calculator, and the object generator; 8. The makeup simulation system according to any one of claims 1 to 7, further comprising a GPU that functions as one. a step of calculating a first spectral reflectance of the application object before makeup from a first RGB value before makeup of the application object to which the cosmetic is applied;
Using a first three-dimensional reflection characteristic model represented by the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristic of the application object before the makeup, and the spectral distribution and geometric conditions of the illumination light a step of calculating a first color signal in the application target before the makeup;
using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter that defines the reflection characteristic of the cosmetic, the spectral distribution of the illumination light, and the geometric condition; calculating a second color signal of the cosmetic with
a step of calculating a third color signal for the application object in a makeup state in which the cosmetic is applied to the application object before makeup from the first color signal and the second color signal;
calculating second RGB values of the application target in the makeup state from the third color signal;
and generating a three-dimensional object including the application target in the makeup state from the second RGB values. to the computer,
a process of calculating the first spectral reflectance of the application target before makeup from the first RGB values before makeup of the application target on which the cosmetic is applied;
Using a first three-dimensional reflection characteristic model represented by the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristic of the application object before the makeup, and the spectral distribution and geometric conditions of the illumination light a process of calculating a first color signal in the application target before the makeup;
using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter that defines the reflection characteristic of the cosmetic, the spectral distribution of the illumination light, and the geometric condition; a process of calculating a second color signal of the cosmetic by
a process of calculating, from the first color signal and the second color signal, a third color signal of the application object in a makeup state in which the cosmetic is applied to the application object before the makeup;
a process of calculating second RGB values of the application target in the makeup state from the third color signal;
A makeup simulation program for executing a process of generating a three-dimensional object including the application target in the makeup state from the second RGB values. a first spectral reflectance calculation unit that calculates a first spectral reflectance of the application target before makeup from first RGB values before makeup of the application target on which the cosmetic is applied;
Using a first three-dimensional reflection characteristic model represented by the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristic of the application object before the makeup, and the spectral distribution and geometric conditions of the illumination light a first color signal calculation unit that calculates a first color signal in the application target before the makeup;
using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter that defines the reflection characteristic of the cosmetic, the spectral distribution of the illumination light, and the geometric condition; a second color signal calculation unit for calculating a second color signal of the cosmetics;
a third color signal calculation unit that calculates a third color signal of the application object in a makeup state in which the cosmetic is applied to the application object from the first color signal and the second color signal;
a second RGB value calculation unit that calculates second RGB values of the application target in the makeup state from the third color signal;
A makeup simulation apparatus, comprising: an object generation unit that generates a three-dimensional object including the application target in the makeup state from the second RGB values.

Images