JP2023143446A - Operation method of information processing device, information processing device, and program - Google Patents

Abstract

To make it possible to convert images so as not to damage attributes of original face images while expressing various makeup methods.SOLUTION: An operation method of an information processing device includes: a first step of converting a first face image to a second face image by applying makeup information to the first face image by a prescribed procedure; a second step of converting the first face image to a third face image with the second face image as a reference image by an image conversion model generated by machine learning processing for converting a conversion object face image by applying makeup information included in a reference face image; and a third step of adjusting the image conversion model by reducing a loss of the first face image in the third face image.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to an operating method of an information processing device, an information processing device, and a program.

2. Description of the Related Art There is a known technique for converting the style of a target image using an image conversion model obtained by machine learning processing for applying the style of a reference image to another image (for example, Patent Document 1). Such technology is applied in the beauty field and the like. For example, by using an image conversion model that adds make-up information from a face image with make-up applied to a face image without make-up, that is, a reference image, the user's face image can be transformed into a face image with make-up applied. A makeup conversion method has been proposed in which a face image is converted and makeup is virtually tried out.

JP2021-135822

Image transformation models that machine learn various styles of reference images using methods such as GAN (Generative Adversarial Network) do not have stable learning, and when converting the style of the target image, There is a problem that the attributes of the image are lost. In the case of makeup conversion, the texture of the original facial image may be lost. In addition, makeup techniques depend on the makeup artist's personal skills and change from day to day, so variations are likely to occur. In order to use machine learning to generate an image conversion model that can express any makeup, endless learning is required. It has to be done and is not practical.

In view of the above, hereinafter, a method of operating an information processing apparatus and the like will be disclosed that enables image conversion that does not impair the attributes of the original facial image while expressing various makeup techniques.

In order to solve the above problems, an operating method of an information processing apparatus according to the present disclosure includes a first step of adding makeup information to a first face image according to a predetermined procedure and converting it into a second face image; The second face image is used as a reference image by an image conversion model generated by machine learning to transform the face image to be converted by adding makeup information included in the face image for conversion. a second step of converting a first facial image into a third facial image; and a third step of adjusting the image transformation model by reducing loss of the first facial image in the third facial image. process.

In addition, the information processing device according to the present disclosure is generated by machine learning processing for converting a face image to be converted by adding makeup information included in a reference face image, and is generated by applying makeup information to a first face image. Loss of the first face image in the second face image when the first face image is converted into a second face image using the face image to which makeup information has been added as a reference image according to the procedure of a storage unit that stores an image transformation model adjusted to reduce
and a control unit that converts an input facial image into an output facial image using the image conversion model after executing the predetermined procedure.

Furthermore, a program in the present disclosure is a program executed by an information processing device, wherein the information processing device converts a face image to be converted by adding makeup information included in a reference face image. When the first face image is converted into a second face image using a face image generated by machine learning processing and with makeup information added to the first face image according to a predetermined procedure as a reference image. an image transformation model adjusted to reduce a loss of the first facial image in the second facial image, and performing the predetermined procedure on an input facial image; and a second step of converting the face image on which the first step has been performed into a face image for output using the image conversion model.

According to the operating method of the information processing device according to the present disclosure, it is possible to perform image conversion that expresses various makeup techniques while not impairing the attributes of the original facial image.

FIG. 1 is a diagram illustrating a configuration example of an information processing system. FIG. 3 is a flowchart diagram illustrating an example of an operation procedure of the server device. FIG. 3 is a flowchart diagram illustrating an example of an operation procedure of the server device. FIG. 3 is a diagram illustrating a face image used for image conversion. FIG. 2 is a sequence diagram showing an example of an operation procedure of a server device and a terminal device.

Embodiments of the present invention will be described below.

[System configuration]
FIG. 1 is a diagram showing a configuration example of an embodiment of the present invention. The information processing system 1 includes a server device 10 and a terminal device 12 that are connected to each other via a network 11 so as to be able to communicate information. In the information processing system 1, the server device 10 performs machine learning using various information sent from the terminal device 12. The terminal device 12 is, for example, one or more personal computers. The personal computer may include a tablet terminal device, a smartphone, and the like. The server device 10 is, for example, one or more server computers. When the server device 10 is a single server computer, the server device 10 may be a plurality of server computers that coordinately execute the operations in this embodiment and provide cloud services. The network 11 is, for example, a LAN (Local Area Network), the Internet, an ad hoc network, a MAN (Metropolitan Area Network), a mobile communication network, or another network, or a combination of any of these.

The server device 10 acquires a facial image obtained by capturing a person's face from the terminal device 12, performs machine learning using the facial image, and generates an image conversion model 108. The terminal device 12 is, for example, a device owned by a user, a device installed in a physical store and used by a salesperson, or the like. The image conversion model 108 is an image conversion model that adds make-up information of a face image with make-up applied, that is, a reference image, to a face image without make-up (an image including the entire face of a person when viewed from the front). It is. Further, the server device 10 converts the original face image using the image conversion model 108 using an image in which makeup information is added to a face image without makeup (hereinafter referred to as the original face image) using a predetermined procedure. The image conversion model 108 is adjusted so that the attributes of the original face image are not impaired during the process.

Specifically, the server device 10 executes a first step (hereinafter referred to as an algorithm processing step) of adding makeup information to the original facial image according to a predetermined procedure and converting it into a primary facial image. Further, the server device 10 converts the primary facial image into a primary facial image using an image conversion model 108 that is generated by machine learning processing for converting a facial image to be converted by adding makeup information included in a reference facial image. A second step (hereinafter referred to as makeup conversion step) of converting the original facial image into a secondary facial image as a reference facial image is executed. Then, the server device 10 executes a third step (hereinafter referred to as an adjustment step) of adjusting the image conversion model 108 by reducing the loss of the original face image in the secondary face image. Here, the server device 10 corresponds to an "information processing device."

According to the present embodiment, the server device 10 simulates the makeup artist's technique without going through machine learning by executing the algorithm processing step according to a procedure for reproducing the makeup artist's technique. becomes possible. Furthermore, by separating the procedure for adding makeup information as an algorithm processing step, the amount of information processing to be performed by machine learning can be reduced, and the stability of machine learning is improved. Furthermore, the server device 10 adjusts the image conversion model 108 by converting the original face image into a secondary face image using the image conversion model 108 and reducing the loss of the original face image in the secondary face image. When converting an image into a secondary facial image through an algorithm processing step and a makeup conversion step, it is possible to prevent the attributes of the original facial image from being impaired. In other words, it is possible to perform image conversion that reproduces various makeup techniques without impairing the attributes of the original facial image, especially the so-called textures such as gloss, matte, sheer, and foggy.

Next, the configurations of the server device 10 and the terminal device 12 will be explained.

The server device 10 includes a communication section 101, a storage section 102, a control section 103, an input section 105, and an output section 106. When the server device 10 is composed of two or more server computers, these configurations are appropriately arranged in the two or more server computers.

Communication unit 101 includes one or more communication interfaces. The communication interface is, for example, a LAN interface. The communication unit 101 receives information used for the operation of the server device 10 and transmits information obtained by the operation of the server device 10. The server device 10 is connected to a network 11 by a communication unit 101 and performs information communication with a terminal device 12 via the network 11.

The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these, which function as a main storage device, an auxiliary storage device, or a cache memory. The semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The RAM is, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). The ROM is, for example, an EEPROM (Electrically Erasable Programmable ROM). The storage unit 102 stores information used for the operation of the control unit 103 and information obtained by the operation of the control unit 103. The storage unit 102 stores an image conversion model 108 generated by the control unit 103 based on information sent from the terminal device 12.

Control unit 103 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit), or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 103 executes information processing related to the operation of the server device 10 while controlling each part of the server device 10 .

The functions of the server device 10 are realized by a processor included in the control unit 103 executing a control program. The control program is a program for causing the processor to function as the control unit 103. Furthermore, some or all of the functions of the server device 10 may be realized by a dedicated circuit included in the control unit 103. Further, the control program may be stored in a non-transitory recording/storage medium that can be read by the control unit 103, and may be read from the medium by the control unit 103.

The input unit 105 includes one or more input interfaces. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen provided integrally with the display, or a microphone that accepts voice input. The input unit 105 accepts an operation to input information used for the operation of the server device 10 and sends the input information to the control unit 103.

The output unit 106 includes one or more output interfaces. The output interface is, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 106 outputs information obtained by the operation of the server device 10.

The terminal device 12 includes a communication section 121, a storage section 122, a control section 123, an input section 125, and an output section 126.

The communication unit 121 includes a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as LTE, 4G, and 5G, and the like. The terminal device 12 is connected to the network 11 by a communication unit 121 via a nearby router device or a mobile communication base station, and performs information communication with the server device 10 and the like via the network 11.

The storage unit 122 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. The semiconductor memory is, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, an EEPROM. The storage unit 122 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 122 stores information used for the operation of the control unit 123 and information obtained by the operation of the control unit 123.

The control unit 123 includes, for example, one or more general-purpose processors such as a CPU or an MPU (Micro Processing Unit), or one or more dedicated processors such as a GPU specialized for specific processing. Alternatively, the control unit 123 may include one or more dedicated circuits such as FPGA or ASIC. The control unit 123 comprehensively controls the operation of the terminal device 12 by operating according to a control/processing program or operating according to an operation procedure implemented as a circuit. The control unit 123 transmits and receives various information to and from the server device 10 and the like via the communication unit 121, and executes operations according to the present embodiment.

The functions of the terminal device 12 are realized by a processor included in the control unit 123 executing a control program. The control program is a program for causing the processor to function as the control unit 123. Further, some or all of the functions of the terminal device 12 may be realized by a dedicated circuit included in the control unit 123. Further, the control program may be stored in a non-transitory recording/storage medium that can be read by the control unit 123, and may be read by the control unit 123 from the medium.

The input unit 125 includes one or more input interfaces. Input interfaces include, for example, physical keys, capacitive keys, pointing devices, and touch screens integrated with the display. The input interface also includes a microphone that accepts audio input and a camera that captures captured images. Furthermore, the input interface may include a scanner or camera for scanning image codes, and an IC card reader. The input unit 125 accepts an operation to input information used for the operation of the control unit 123 and sends the input information to the control unit 123. Further, the input unit 125 sends an image captured by the camera to the control unit 123.

The output unit 126 includes one or more output interfaces. The output interface includes, for example, a display and a speaker. The display is, for example, an LCD or an organic EL display. The output unit 126 outputs information obtained by the operation of the control unit 123.

[Generation of image conversion model]
FIG. 2 is a flowchart for explaining an example of the operation of the server device 10 related to generation of the image conversion model 108. Each step is executed by the control unit 103.

In step S20, the control unit 103 acquires facial images necessary for machine learning to generate an image conversion model. The face image includes an original face image to be converted without makeup and a reference face image with makeup. The original face image is generated by capturing an image of a person's face without makeup. Further, a reference face image is generated by capturing an image of a person's face with makeup applied. For example, the terminal device 12 captures an image of a person's face. For example, the control unit 103 receives a plurality of original face images and a plurality of reference face images sent from the terminal device 12 via the communication unit 101, and stores them in the storage unit 102. The control unit 103 may acquire a plurality of original facial images and a plurality of reference facial images from open data.

In step S22, the control unit 103 executes machine learning. The control unit 103 executes deep learning using GAN, for example. The control unit 103 includes a module corresponding to a generator that adds makeup information of a reference facial image to the original facial image, and a module corresponding to a discriminator that identifies the facial image generated by the generator from the original facial image. has. The control unit 103 generates the image transformation model 108 by causing the generator and the classifier to learn adversarially. The control unit 103 stores the generated image conversion model 108 in the storage unit 102.

[Adjust image conversion model]
FIG. 3 is a flowchart for explaining an example of the operation of the server device 10 related to adjustment of the image conversion model 108. Each step is executed by the control unit 103. FIG. 4 is a diagram for explaining a face image used for adjusting the image conversion model 108. The procedure in FIG. 3 will be explained with reference to FIG. 4.

In step S30, the control unit 103 acquires the original face image Isrc for executing the algorithm processing step. The original face image Isrc is a face image without makeup, and is generated by, for example, the terminal device 12 capturing an image of a person's face without makeup. For example, the control unit 103 receives the original face image Isrc sent from the terminal device 12 via the communication unit 101 and stores it in the storage unit 102. The control unit 103 may acquire a plurality of original facial images Isrc from open data.

In step S32, the control unit 103 executes an algorithm processing step. The control unit 103 adds predetermined makeup information MU to the original facial image Isrc, and converts the original facial image Isrc into a primary facial image Isyn. The makeup information MU includes one or more of hue, brightness, and saturation given to a region including the eyes, bridge of the nose, cheeks, and lips (hereinafter referred to as makeup region) in the original facial image Isrc. The makeup information MU and the procedure for adding makeup information are image processing procedures that are arbitrarily set in advance. The hue, brightness, and saturation applied to the makeup area may be arbitrarily determined quantitatively, or information on colors already applied to the reference facial image may be extracted and applied. The reference face image can be arbitrarily selected by the user, for example, according to his/her own image. The reference images include, for example, facial images with various types of makeup such as elegant, cool, and trendy, and facial images of real people. The makeup information MU may be input by an operator at the terminal device 12 and sent from the terminal device 12 to the server device 10. The control unit 103 performs image processing to apply eyeliner, nose shadow, cheek color, lip color, etc. of a color arbitrarily set in the makeup information MU to the eyes, bridge of the nose, cheeks, or lips in the face image Isrc, respectively. Execute.

In step S34, the control unit 103 executes a makeup conversion process. The control unit 103 inputs the original face image Isrc as a face image to be converted and the primary face image Isyn as a reference face image to the image conversion model 108. The image conversion model 108 uses an encoder to extract feature information such as face shape, three-dimensional effect, and surface condition from the original facial image Isrc, and makeup information from the primary facial image Isyn, and converts the primary facial image Isyn into the original facial image Isrc. Make-up information is added and converted into a secondary facial image Ig by a decoder.

In step S36, the control unit 103 executes an adjustment process. The control unit 103 adjusts the parameters of the image conversion model 108 so as to minimize the value of the loss function L when the secondary face image Ig is used as a parameter of the loss function L. The loss function L includes, for example, one or more of Adversarial loss, Makeup loss, Perceptual loss, MGE (Mean Gradient Error) loss, and Color loss.

Adversarial loss is a loss function for training the generator to fool the classifier in the GAN. The control unit 103 causes the discriminator to identify whether the secondary facial image Ig is the result of the makeup conversion process or the original facial image Isrc, and uses the result to learn the generator so as to fool the discriminator. let By doing so, the parameters of the image conversion model 108 are adjusted so that the loss of the original face image Isrc in the secondary face image Ig is reduced. The classifier may include a global classifier and a local classifier. The global classifier identifies whether the secondary facial image Ig is the result of the makeup conversion process or the original facial image Isrc, targeting the entire facial image. The local classifier identifies whether the secondary facial image Ig is the result of the makeup conversion process or the original facial image Isrc, targeting the makeup area. By using a global classifier and a local classifier together, it is possible to improve the learning accuracy of the generator.

Makeup loss is a loss function that trains the generator to fool the classifier regarding the color distribution. The control unit 103 generates a pseudo facial image having the same color distribution as the primary facial image Isyn by histogram matching based on the primary facial image Isyn. Histogram matching may be performed on the color distribution of the entire face image, or may be performed on the color distribution of the makeup area. Then, the control unit 103 causes the discriminator to identify whether the pseudo face image or the secondary face image Ig is the result of the histogram matching or makeup conversion process, or the original face image Isrc, and uses the result. train the generator to fool the discriminator. By doing so, the parameters of the image conversion model 108 are adjusted so that the loss of the original face image Isrc in the secondary face image Ig is reduced.

Perceptual loss is a loss function for training the generator to fool the classifier regarding the outline of the face image. The control unit 103 acquires a face image in the middle of conversion from an intermediate layer when the primary face image Isyn is converted into a secondary face image Ig by the generator, and extracts the edges of each part such as the eyes, nose, and mouth in the image. Generate an extracted edge image. Then, the control unit 103 causes the classifier to identify whether the edge of the edge image is an edge of the edge image obtained by the makeup conversion process or an edge of the original face image Isrc, and uses the result to fool the classifier. Train the generator. In deep learning, the edge information of the image to be converted is extracted as a feature quantity in the middle layer, and by performing learning using the edge images in the middle layer, the edge information of the original face image Isrc in the secondary face image Ig is The parameters of the image transformation model 108 are adjusted so that the loss is small. That is, the image conversion model 108 is adjusted so that the outline of the face in the secondary face image Ig matches the outline of the face in the original face image Isrc.

MGE loss is a loss function regarding the contour of a face image. The control unit 103 applies a differential filter to each of the original face image Isrc and the secondary face image Ig to increase the edge resolution. Then, the control unit 103 adjusts the parameters of the image conversion model 108 so that the edges in the high-resolution secondary face image Ig match the edges in the high-resolution original face image Isrc. By doing so, the image conversion model 108 is adjusted so that the loss related to the edges of the original face image Isrc in the secondary face image Ig is reduced.

Color loss is a loss function for reinforcing Makeup loss regarding color distribution. The control unit 103 calculates the differences between the average value and variance value of the color defined by the makeup information MU given to the original facial image Isrc and the average value and variance value of the color in the primary facial image Isyn of the makeup area. Derive. Then, the control unit 103 adjusts the parameters of the image conversion model 108 so as to reduce each difference. By doing so, the parameters of the image conversion model 108 are adjusted so that the loss of the original face image Isrc in the secondary face image Ig is reduced.

The image conversion model 108 is adjusted through the procedure described above.

The adjusted image transformation model 108 is used to transform any original facial image. Specifically, the control unit 103 generates a primary facial image by performing algorithmic processing on an arbitrary original facial image, and converts the primary facial image into a secondary facial image using the image conversion model 108. Here, the verification results of the operation of the image conversion model 108 will be shown.

[Verification 1]
As algorithm processing by the server device 10, a process was performed in which the lip color included in the reference face image was applied to the original face image by histogram matching. As the original facial images, 150 arbitrary facial images from open data were used. Each original facial image was converted into a primary facial image by algorithm processing, and the primary facial image was converted into a secondary facial image by the image conversion model 108. Here, the image conversion model 108 employs U-net architecture. Additionally, makeup information was added to each original face image using related technology. BeautyGAN, PSGAN (Pose and Expression Robust Spatial-Aware GAN), and CPM (Color-Pattern Makeup Transfer) were adopted as related technologies. For the secondary face image and the face image with makeup information added using each related technology, scores indicating impressions regarding the following four items were obtained from 12 subjects (females aged 18 to 23). Heard.
<Item 1> Is the lip color of the reference face image reflected? <Item 2> Is the skin texture of the original face image maintained? <Item 3> Is the lip texture of the original face image maintained? < Item 4> Overall evaluation for <1> to <3>

Table 1 below shows the results of aggregation of scores for each item for the secondary face image in this embodiment and the face images obtained by each related technology. The higher the score, the better the impression. As shown in Table 1, in <Item 2> to <Item 4>, the secondary face image according to this embodiment gave a better impression than the face image according to the related technology.

[Verification 2]
As algorithm processing by the server device 10, a process was performed in which the lip color included in the reference face image was applied to the original face image by histogram matching. An arbitrary face image from open data was used as the original face image. Furthermore, four lip colors were used: crimson, red bean, vermilion, and tan. During algorithm processing, the texture information of the original facial image was deleted and the texture was lost. When the original facial image is converted into a primary facial image by algorithm processing, the primary facial image is converted into a secondary facial image by four image conversion models 108 adjusted by combinations of different loss functions in the server device 10. It was done. Here, the image conversion model 108 employs U-net architecture. The four combination patterns of loss functions used to adjust the image transformation model 108 are as follows.
<Pattern 1> Adversarial loss, Makeup loss, Perceptual loss, and Color loss
<Pattern 2> Adversarial loss, Makeup loss, MGE loss, and Color loss
<Pattern 3> Adversarial loss, Makeup loss, Perceptual loss, and MGE loss
<Pattern 4> Adversarial loss, Makeup loss, Perceptual loss, MGE loss, and Color loss

An arbitrary original facial image was converted into a total of 16 secondary facial images, with four patterns for each of the four lip colors. Then, for each color of lip color, a qualitative evaluation was made of the secondary facial image according to each pattern.The qualitative evaluation for each pattern was as follows.
<Pattern 1> The texture of the original face image cannot be reproduced. <Pattern 2> The outline stands out. <Pattern 3> Learning becomes unstable. <Pattern 4> All of the above are resolved.

[Verification 3]
As algorithm processing by the server device 10, a process was performed in which the lip color included in the reference face image was applied to the original face image by histogram matching. An arbitrary face image from open data was used as the original face image. Furthermore, after the algorithm processing, lip colors were clustered in the color space to identify glossy areas, and gloss was suppressed (filling the glossy areas with the color of the non-glossy areas). The original facial image was converted into a primary facial image by algorithm processing, and the primary facial image was converted into a secondary facial image by the image conversion model 108. Here, the image conversion model 108 employs U-net architecture. As a result, a qualitative evaluation was obtained that in the secondary facial image, the three-dimensional effect of the original facial image was reproduced in the area where gloss was suppressed.

[Example]
FIG. 5 is a sequence diagram for explaining an example of the operation of the information processing system 1 in the embodiment. The procedure in FIG. 5 relates to the cooperative operation of the server device 10 and the terminal device 12, which have the image conversion model 108 adjusted according to the procedure of this embodiment. The terminal device 12 is used, for example, by a user who attempts to apply makeup using a captured image of his or her own face.

In step S50, the terminal device 12 captures an image of the user. The control unit 123 of the terminal device 12 captures an image using a camera included in the input unit 125 in response to a user's operation input to the input unit 125. Thereby, the terminal device 12 acquires the original face image.

In step S51, the terminal device 12 receives input for selecting algorithm processing. The control unit 123 of the terminal device 12 executes, for example, an application program that provides virtual makeup. The control unit 123 displays a selection menu on a display included in the output unit 126, for example. Then, the control unit 123 selects the type of algorithm processing and the color to be applied to the face image by the algorithm processing in accordance with the user's operation input to the input unit 125. The selected algorithmic process is the application of eyeliner, nose shadow, cheek color, lip color, or the like. The control unit 123 may display a reference face image including makeup information as a sample, and the user may select the algorithm process by selecting the sample.

In step S52, the terminal device 12 sends the original facial image and an image conversion request to the server device 10. The image conversion request includes information identifying the selected algorithmic process. The control unit 123 uses the communication unit 121 to send the original face image and an image conversion request. In the server device 10 , the control unit 103 receives information sent from the terminal device 12 through the communication unit 101 .

In step S53, the server device 10 executes algorithm processing on the original facial image. The control unit 103 executes the specified algorithm processing on the original face image. As a result, the original facial image is converted into a primary facial image.

In step S55, the control unit 103 of the server device 10 converts the primary facial image into a secondary facial image using the image conversion model 108.

In step S56, the server device 10 sends the secondary facial image for output to the terminal device 12. The control unit 103 sends the secondary facial image via the communication unit 101. In the terminal device 12 , the control unit 123 receives information sent from the server device 10 through the communication unit 121 .

In step S57, the terminal device 12 displays the secondary facial image. For example, the control unit 123 causes a display included in the output unit 126 to display the secondary facial image.

According to the above-described procedure, the adjusted image conversion model 108 makes it possible to output a natural face image that reproduces the texture of the original face image even after makeup has been applied. The user can check the image of his or her face with the specified makeup applied.

As described above, according to the present embodiment, it is possible to perform image conversion that expresses various makeup techniques without impairing the attributes of the original facial image.

In the above description, the server device 10 corresponds to the "information processing device." However, the server device 10 and the terminal device 12 may cooperate to form an "information processing device," or the terminal device 12 may correspond to the "information processing device."

In the embodiment described above, the processing/control program that defines the operation of the terminal device 12 is stored in the storage unit 102 of the server device 10 or the storage device of another server device, and is downloaded to the terminal device 12 via the network 11. Alternatively, the information may be stored in a computer-readable non-transitory recording/storage medium, and the terminal device 12 may read the information from the medium.

Although the embodiments have been described above based on the drawings and examples, it should be noted that those skilled in the art can easily make various changes and modifications based on the present disclosure. It should therefore be noted that these variations and modifications are included within the scope of this disclosure. For example, the functions included in each means, each step, etc. can be rearranged so as not to be logically contradictory, and it is possible to combine multiple means, steps, etc. into one, or to divide them. .

10: Server device 11: Network 12: Terminal device 101, 121: Communication section 102, 122: Storage section 103, 123: Control section 105, 125: Input section 106, 126: Output section 108: Image conversion model Isrc: Original face Image Isyn: Primary face image Ig: Secondary face image L: Loss function

Claims (9)

A method for operating an information processing device, the method comprising:
a first step of adding makeup information to the first face image according to a predetermined procedure and converting it into a second face image;
The second face image is used as a reference image using an image conversion model generated by machine learning processing of converting a face image to be converted by adding makeup information included in the reference face image. a second step of converting the first facial image into a third facial image;
a third step of adjusting the image transformation model by reducing loss of the first face image in the third face image;
How it works, including: In claim 1,
The predetermined procedure is a procedure for adding makeup information of one or more of hue, brightness, and saturation to a region of the first face image including the eyes, bridge of the nose, cheeks, and lips.
How it works. In claim 2,
the predetermined procedure does not include machine learning;
How it works. In claim 2,
The loss is a loss of one or more of hue, brightness, and saturation of the part in the first facial image, or a loss of an edge of the part,
How it works. In claim 4,
The loss is represented by any one or more of Adversarial loss, Makeup loss, and Perceptual loss,
How it works. In claim 5,
In the third step, the loss is further adjusted using MGE loss or color loss.
How it works. In claim 1,
The image conversion model includes an encoder that extracts makeup information from the reference face image, and a decoder that converts the face image to be converted.
How it works. Generated by machine learning processing of converting the target face image by adding makeup information included in the reference face image, and makeup information is added to the first face image according to a predetermined procedure. An image adjusted to reduce loss of the first face image in the second face image when the first face image is converted into a second face image using the face image as a reference image. a storage unit that stores the conversion model;
a control unit that converts an input facial image into an output facial image using the image conversion model after executing the predetermined procedure;
An information processing device having: A program executed by an information processing device,
The information processing device is generated by machine learning processing for converting a face image to be converted by adding makeup information included in a reference face image, and applies makeup to the first face image according to a predetermined procedure. To reduce the loss of the first face image in the second face image when the first face image is converted into a second face image using a face image to which close-up information is attached as a reference image. It is possible to use an image transformation model adjusted to
a first step of performing the predetermined procedure on the input facial image;
a second step of converting the face image on which the first step has been performed into a face image for output using the image conversion model;
programs, including.

Images