JP7448879B2 - Image generation method, system, and computer program - Google Patents

Description

The present specification relates to an image data generation technique including style conversion processing using a machine learning model.

2. Description of the Related Art There is a known technique for converting the style of an image using an image generation model using a neural network or the like. For example, the image forming apparatus described in Patent Document 1 inputs image data indicating a conversion source image and image data indicating a style reference image, and then outputs image data indicating the converted image. Output. The converted image is an image in which the style of the style reference image is applied to the content of the conversion source image.

JP2018-132855A Japanese Patent Application Publication No. 2011-197995 Japanese Patent Application Publication No. 2004-213598

However, with the above technique, only one style of the style reference image is applied to one conversion source image, so there is a possibility that flexible style conversion cannot be performed.

This specification discloses a technique that can realize flexible style conversion.

The technology disclosed in this specification can be implemented as the following application examples.

[Application example 1] An image acquisition step of acquiring input image data indicating an input image, and using the input image data, a first input partial image that is a part of the input image, and a first input partial image that is a part of the input image. a second input partial image located at a different position from the first input partial image; and a machine learning model for the first partial image data indicating the first input partial image. a first conversion step of executing a first style conversion process using a method to generate first converted data representing a first converted partial image; Then, the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, is executed to generate second converted data indicating the second converted partial image. a second conversion step of generating, and an output image generation step of generating output image data representing an output image based on the input image using the first converted data and the second converted data, the output The image includes a first output partial image corresponding to the first input partial image and a second output partial image corresponding to the second input partial image, the first output partial image being the first converted partial image. and the second output partial image is an image based on the second converted partial image.

According to the above configuration, the first converted data generated by performing the first style conversion process on the first partial image data representing the first input partial image, and the second portion representing the second input partial image. Output image data representing an output image based on the input image is generated using the second converted data generated by performing the second style conversion process on the image data. The output image includes a first output partial image based on the first converted partial image indicated by the first converted data, and a second output partial image based on the second converted partial image indicated by the second converted data. including. In this way, since output image data is generated by applying the first style conversion process and the second style conversion process to one piece of input image data, flexible style conversion can be realized.
[Application example 2]
The image generation method described in Application Example 1, comprising:
The first style conversion process is performed using first style image data indicating a first style image,
The second style conversion process is performed using second style image data indicating a second style image,
The first converted partial image is an image in which the style of the first style image is applied to the first input partial image,
The second converted partial image is an image in which the style of the second style image is applied to the second input partial image.
[Application example 3]
The image generation method according to Application Example 1 or 2,
The output image generation step includes:
a first step of generating intermediate image data representing an intermediate image including the first converted partial image and the second converted partial image using the first converted data and the second converted data;
a second step of performing specific post-processing on the intermediate image data to generate the output image data;
An image generation method, including:
[Application example 4]
The image generation method described in Application Example 3,
The specific post-processing includes, in the intermediate image, a difference in pixel values between the first converted partial image and a portion adjacent to the first converted partial image, and a difference between the second converted partial image and the pixel value. An image generation method comprising: reducing a difference in pixel values between a second converted partial image and an adjacent portion.
[Application example 5]
The image generation method according to Application Example 3 or 4,
The image generation method, wherein the specific post-processing includes the third style conversion process that is a third style conversion process using a machine learning model and is different from the first style conversion process and the second style conversion process.
[Application example 6]
The image generation method described in Application Example 5,
The third style conversion process is performed using the input image data as style image data.
[Application example 7]
The image generation method described in Application Example 6,
The input image includes an image showing a person's face,
The specific post-processing includes processing for correcting the skin color of the person's face on the input image data to generate the corrected input image data,
The third style conversion process is performed using the corrected input image data as style image data.
[Application example 8]
The image generation method according to any one of Application Examples 3 to 7,
The specific post-processing includes the fourth style conversion process that is a fourth style conversion process using a machine learning model and is different from the first style conversion process and the second style conversion process,
The input image includes an image showing a person's face,
In the image generation method, the fourth style conversion process is a process of changing the facial expression of the person.
[Application example 9]
The image generation method according to any one of Application Examples 1 to 8,
The input image includes an image showing a person's face,
The first input partial image is an image showing a first part of the person's face,
In the image generation method, the second input partial image is an image showing the second part of the person's face and located at a different position from the first part.
[Application example 10]
The image generation method according to any one of Application Examples 1 to 9, further comprising:
comprising a type identifying step of identifying the type of the input image;
When the input image is a first type input image,
In the first conversion step, a first type of first style conversion process is performed on the first partial image data,
In the second conversion step, a first type of second style conversion process is performed on the second partial image data,
When the input image is a second type input image,
In the first conversion step, a second type of first style conversion process is performed on the first partial image data,
In the second conversion step, a second type of second style conversion process is performed on the second partial image data.
[Application example 11]
The image generation method according to Application Example 10,
The input image includes an image showing a person's face,
The type of the input image is a type related to at least part of the person's gender, race, facial expression, and facial angle.
[Application example 12]
The image generation method according to any one of Application Examples 1 to 11,
The first style conversion process is performed using a first parameter that specifies the degree of difference between the first input partial image and the first converted partial image to be generated,
The second style conversion process is performed using a second parameter that specifies the degree of difference between the second input partial image and the second converted partial image to be generated,
The first parameter and the second parameter are adjusted independently.
[Application example 13]
The image generation method according to any one of Application Examples 1 to 12, further comprising:
a process selection step of selecting a process to be performed on the first partial image data indicating the first input partial image;
a color conversion step of performing a color conversion process on the first partial image data to convert at least part of the color of the first input partial image without using a machine learning model;
Equipped with
When the first style conversion process is selected in the process selection step, the first conversion process is executed without executing the color conversion process,
An image generation method, wherein when the color conversion process is selected in the process selection step, the color conversion process is executed without executing the first conversion process.
[Application example 14]
The image generation method according to Application Example 13,
The input image includes an image showing a person's face,
The first input partial image is an image showing the eyes of the person,
The image generation method is characterized in that the color conversion process is a process of converting a value of a pixel corresponding to the white part of the eye in the image showing the eye into a specific value showing white.
[Application example 15]
The image generation method according to any one of Application Examples 1 to 14, further comprising:
Obtain first input information for the first style conversion process based on a first input by the user, and obtain second input information for the second style conversion process based on a second input by the user. Equipped with an information acquisition process to acquire
In the first conversion step, the first style conversion process is performed using the first input information,
In the image generation method, in the second conversion step, the second style conversion process is performed using the second input information.
[Application example 16]
The image generation method according to Application Example 15,
The first input information includes data indicating an image corresponding to the first input partial image and having a style different from that of the first input partial image,
The second input information includes data indicating an image corresponding to the second input partial image and having a style different from the second input partial image.
[Application example 17]
The image generation method according to any one of Application Examples 1 to 16,
The input image includes an image showing a person's face,
The second input partial image is an image showing the mouth of the person,
The second style conversion process is a process of correcting tooth alignment in the image showing the mouth.

Note that the technology disclosed in this specification can be realized in various forms, such as a system, an image generation device, a computer program for realizing the functions of these methods, devices, and the system, and a computer program for implementing the computer program. This can be realized in the form of a recorded recording medium, etc.

FIG. 1 is a block diagram showing the configuration of a system 1000 of this embodiment. An explanatory diagram of the configuration of the generation network group GNG. 5 is a flowchart of processing executed by the terminal device 200 of the first embodiment. The figure which shows an example of the input image Iin and the output image Iout. The figure which shows an example of a selection screen. 5 is a flowchart of processing executed by the server 100 of the first embodiment. 5 is a flowchart of processing executed by the terminal device 200. FIG. 7 is a diagram showing a selection screen UD in the second embodiment. 7 is a flowchart of processing executed by the server 100 of the second embodiment.

A. First Example A-1. Configuration of System 1000 Next, an embodiment will be described based on an example. FIG. 1 is a block diagram showing the configuration of a system 1000 of this embodiment. The system 1000 includes a server 100 and a terminal device 200. The system 1000 of the first embodiment is an image generation system for generating output image data representing an output image using input image data. The sewing machine 300 indicated by a broken line in FIG. 1 is a component included in the system of the second embodiment, and is not a component included in the system of the first embodiment, so it will be described in the second embodiment.

Server 100 is a computer connected to Internet IT. The server 100 includes a CPU 110 as a controller of the server 100, a volatile storage device 120 such as a RAM, a non-volatile storage device 130 such as a hard disk drive or flash memory, and a communication interface (IF) 160. The communication interface 160 is a wired or wireless interface for connecting to the Internet IT.

The volatile storage device 120 provides a buffer area that temporarily stores various intermediate data generated when the CPU 110 performs processing. The nonvolatile storage device 130 stores a computer program PGs, a style image data group SDG (described later), and a skin color data group SKG (described later).

The computer program PGs, the style image data group SDG, and the skin color data group SKG are provided by, for example, an operator of the server 100 and uploaded to the server 100. By executing the computer program PGs, the CPU 110 cooperates with the terminal device 200 to execute processing for generating an output image, which will be described later.

The computer program PGs includes, as a module, a computer program that causes the CPU 110 to realize a generation network group GNG including a plurality of generation networks GN, which will be described later.

The terminal device 200 is, for example, a portable terminal device such as a smartphone. The terminal device 200 includes a CPU 210 which is a processor serving as a controller of the terminal device 200, a non-volatile storage device 220 such as a hard disk drive or a flash memory, a volatile storage device 230 such as a RAM, and a touch panel etc. that receives user operations. It includes an operation unit 240, a display device 250 such as a liquid crystal display superimposed on a touch panel, and a wireless communication interface 260 for communicating with external devices. The terminal device 200 is communicably connected to the server 100 via the wireless network NW and the Internet IT.

A computer program PGt is stored in the nonvolatile storage device 220 of the terminal device 200. The computer program PGt is provided by the operator of the server 100 described above, and is provided, for example, in the form of being downloaded from a predetermined server connected to the terminal device 200 via the Internet IT. By executing the computer program PGt, the CPU 210 cooperates with the server 100 to execute processing for generating an output image, which will be described later.

A-2. Configuration of Generation Network Group FIG. 2 is an explanatory diagram of the configuration of the generation network group GNG. The generation network group GNG includes four generation networks GN1 to GN4, as shown in the block diagram of FIG. 2(A). Note that the two generation networks GN4 and GN5 indicated by broken lines are provided in the second embodiment, so they will be explained in the second embodiment.

Each of the four generation networks GN1 to GN4 has a configuration shown as generation network GN in FIG. 2(B). The generative network GN is a machine learning model that performs style conversion. In this embodiment, the generative network GN is a machine learning model disclosed in the paper "Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017."

A data pair consisting of content image data CD and style image data SD is input to the generation network GN. Content image data CD is image data indicating a content image. For example, in the generation network GN1 for eyes, the content image is an image showing a person's eyes (described later). Style image data SD is image data indicating a style image. For example, in the eye generation network GN1, the style image is an image showing a person's eyes, and is an image having a different style (for example, eye color tone and makeup characteristics) from the content image.

When a data pair is input, the generation network GN executes an operation using a plurality of parameters on the data pair, generates converted image data TD, and outputs the converted image data TD. The converted image data TD is data indicating a converted image obtained by applying the style of the style image to the content image. For example, the converted image is an image that has the style of the style image while maintaining the shape of the content image (eg, the shape of an eye).

In this embodiment, the content image data CD, style image data SD, and converted image data TD are bitmap data indicating an image including a plurality of pixels, and specifically, each pixel is divided by RGB values. This is RGB image data representing colors. The RGB value is a color value of the RGB color system including gradation values of three color components (hereinafter also referred to as component values), that is, an R value, a G value, and a B value. The sizes of the images indicated by these image data CD, SD, and TD are equal to each other, and are, for example, 256 pixels vertically by 256 pixels horizontally.

As shown in FIG. 2(B), the generation network GN includes an encoder EC, a feature combination section CC, a strength adjustment section SA, and a decoder DC.

Content image data CD and style image data SD are input to the encoder EC. The encoder EC performs dimension reduction processing on the input image data to generate feature data indicating the characteristics of the input image data. The encoder EC is, for example, a neural network (Convolutional Neural Network) having a plurality of layers including a convolution layer that performs convolution processing. In this embodiment, the portion from the input layer to the RElu4_1 layer of the neural network called VGG19 is used for the encoder EC. VGG19 is a learned neural network trained using image data registered in an image database called ImageNet, and its learned calculation parameters are publicly available. In this embodiment, published learned calculation parameters are used as the calculation parameters of the encoder EC.

The feature combination unit CC is the "AdaIN layer" disclosed in the above paper. The feature combination unit CC uses feature data f(c) obtained by inputting the content image data CD to the encoder EC and feature data f(s) obtained by inputting the style image data SD to the encoder EC. Then, converted feature data t is generated.

The strength adjustment unit SA adjusts the strength of style transformation using a parameter α indicating the strength of style transformation. Specifically, the intensity adjustment unit SA generates the intensity-adjusted converted feature data t_ad using the parameter α, the feature data f(c) of the content image data CD, and the converted feature data t. The conversion feature data t_ad is expressed by the following equation (1).
t_ad=(1-α)f(c)+αt…(1)

The parameter α takes a value in the range of 0<α≦1. The closer the parameter α is to 1, the stronger the style transformation becomes. In other words, the closer the parameter α is to 1, the closer the converted image indicated by the converted image data TD is to the style image, and the greater the difference from the content image. For this reason, it can be said that the parameter α is a parameter that specifies the degree of difference between the content image and the transformed image. The parameter α is specified by the user, as will be described later. The parameter α is set to 1 when training the decoder DC.

The intensity-adjusted conversion feature data t_ad is input to the decoder DC. The decoder DC performs a dimension restoration process on the transformed feature data t_ad using a plurality of calculation parameters, which is opposite to that of the encoder EC, and generates the above-mentioned transformed image data TD. The decoder DC is a neural network having multiple layers including a transposed convolution layer that performs transposed convolution.

A plurality of calculation parameters of the decoder DC are adjusted by the following training. A predetermined number (for example, tens of thousands) of data pairs consisting of learning content image data CD and style image data SD are prepared. One adjustment process is performed using a predetermined batch size of data pairs selected from these data pairs.

In one adjustment process, a plurality of calculation parameters are adjusted according to a predetermined algorithm so that the loss function L calculated using data pairs corresponding to the batch size becomes small. As the predetermined algorithm, for example, an algorithm (adam in this embodiment) using error backpropagation and gradient descent is used.

The loss function L is expressed by the following equation (2) using content loss Lc, style loss Ls, and weight λ.
L=Lc+λLs…(2)

In this embodiment, the content loss Lc is a loss (also called an error) between the feature data f(g(t)) of the converted image data TD and the converted feature data t. The feature data f(g(t)) of the converted image data TD is calculated by further inputting the converted image data TD obtained by inputting the data pair to be used into the generation network GN to the encoder EC. . As described above, the converted feature data t is calculated by inputting the feature data f(c) and f(s) obtained by inputting the data pair to be used into the encoder EC to the feature combination unit CC.

Style loss Lc is a data group output from multiple layers of encoder EC when converted image data TD is input to encoder EC, and a data group output from multiple layers of encoder EC when style image data SD is input to encoder EC. This is the loss between the data groups output from each layer.

The above adjustment process is repeated multiple times. With this, when content image data CD and style image data SD are input, converted image data TD indicating a converted image obtained by applying the style of the style image to the content image can be output. , the generative network GN is trained.

The basic configuration of the generation networks GN1 to GN4 is the configuration shown in network GN in FIG. 2(B), but the data pairs used for training the generation networks GN1 to GN4 are different from each other. For example, the generation network GN1 for eyes is trained using data pairs representing human eyes. The generative network GN2 for the nose is trained using data pairs representing a person's nose. The mouth generation network GN3 is trained using data pairs representing a person's mouth. The face generation network GN4 is trained using data pairs representing the entire face of a person. For this reason, the values of a plurality of calculation parameters are different from each other in the trained generation networks GN1 to GN4.

A-3. System Operation FIG. 3 is a flowchart of processing executed by the terminal device 200 of the first embodiment. This process is a process of performing style conversion on input image data using a style conversion service provided by the server 100 to obtain output image data obtained. This process is started based on a user's start instruction, for example, with the computer program PGt of the terminal device 200 being executed.

In S105 of FIG. 3, the CPU 210 of the terminal device 200 obtains input image data indicating the input image Iin. For example, the CPU 210 obtains image data specified by the user from among a plurality of pieces of image data stored in the nonvolatile storage device 130 as input image data. Alternatively, the CPU 210 causes a digital camera (not shown) included in the terminal device 200 to execute photography in response to a user's photography instruction, and obtains image data generated by the photography as input image data. The input image data is, for example, RGB image data.

FIG. 4 is a diagram showing an example of an input image Iin and an output image Iout. As shown in FIG. 4A, the input image Iin of this embodiment is an image showing a photograph including the entire face FC of a person.

In S110 of FIG. 3, the CPU 210 uses the input image data to display the selection screen UDa including the input image Iin on the display device 250. FIG. 5 is a diagram showing an example of a selection screen. The selection screen UDa in FIG. 5A includes an input image Iin, and pull-down menus PM1 and PM2 for inputting selection instructions regarding the type of input image Iin (specifically, selection instructions for gender and race). It includes buttons BT1 and BT2 for inputting a selection screen switching instruction.

In S115 of FIG. 3, the CPU 210 transmits the input image data to the server 100. In this embodiment, data is transmitted from the terminal device 200 to the server 100 as an HTTP request in accordance with HTTP (Hypertext Transfer Protocol).

When the server 100 receives input image data transmitted from the terminal device 200, the CPU 110 of the server 100 starts processing to provide a style conversion service. FIG. 6 is a flowchart of processing executed by the server 100 of the first embodiment. The process shown in FIG. 3 by the terminal device 200 and the process shown in FIG. 6 by the server 100 are executed in parallel while exchanging data.

In S205 of FIG. 6, the CPU 110 of the server 100 receives input image data transmitted from the terminal device 200. In S210 of FIG. 6, the CPU 110 performs a predetermined region specifying process on the input image data to specify regions of a plurality of body parts included in the face FC of the input image Iin. Specifically, as shown in FIG. 4(A), right eye, left eye, nose, and mouth regions Per, Pel, Pn, and Pm are specified. A known image recognition method is used for the area identification process.

For example, an image recognition algorithm called yolo (You only look once) uses a convolutional neural network to simultaneously recognize the location and type of an object in an image. In this example, we use a YOLO convolutional neural network trained to recognize the positions and types of four types of objects: right eye, left eye, nose, and mouth. Pel, Pn, and Pm are specified. yolo can be used, for example, in the paper “J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once:Unified, real-time object detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 779-788.

In S212 of FIG. 6, the CPU 110 generates region information indicating the regions Per, Pel, Pn, and Pm of the plurality of identified body parts, for example, region information indicating the positions and sizes of these regions within the input image Iin. , is transmitted to the terminal device 200.

In S120 of FIG. 3, the CPU 210 of the terminal device 200 receives the area information transmitted from the server 100, and uses the area information to display the identification results of the areas Per, Pel, Pn, and Pm of a plurality of body parts. Displayed on device 250. For example, as shown in FIG. 5(A), on the input image Iin of the selection screen UDa, a plurality of rectangular frames Ser, Sel, Sn, Sm indicating regions Per, Pel, Pn, Pm of a plurality of body parts are displayed. Display. Although omitted in the flowchart, when the user inputs an instruction to modify the position or size of the rectangular frames Ser, Sel, Sn, or Sm, the CPU 210 adjusts the area Per of the corresponding part in accordance with the input. , Pel, Pn, and Pm are corrected. The revised area information is sent to the server 100.

In S125 of FIG. 3, the CPU 210 transmits information on the gender and race selected by the user to the server 100. For example, the pull-down menu PM1 in FIG. 5A includes an option indicating male and an option indicating female. The pull-down menu PM2 includes a plurality of choices indicating races registered in advance. The user operates the pull-down menus PM1 and PM2, selects one option from the plurality of options, and presses the button BT2. When the button BT2 is pressed, the CPU 210 transmits to the server 100 the gender and race information corresponding to the options selected in the pull-down menus PM1 and PM2.

In S215 of FIG. 6, the CPU 110 of the server 100 receives the gender and race information transmitted from the terminal device 200. In S220, CPU 110 transmits style image data SD and skin color data according to the gender and race indicated by the received information to terminal device 200. For example, the style image data group SDG (FIG. 1) stored in the nonvolatile storage device 130 of the server 100 includes a plurality of style image data SD for each combination of gender and race. A plurality of style image data SD corresponding to one combination of gender and race are a plurality of style image data SD each indicating a facial region for each facial region (in this example, eyes, mouth, and nose). Contains. For example, a plurality of style image data SD corresponding to the gender and race indicated by the received information are transmitted to the terminal device 200. The skin color data group SKG (FIG. 1) stored in the nonvolatile storage device 130 of the server 100 includes a plurality of pieces of skin color data (for example, RGB values indicating skin color) for each combination of gender and race. For example, a plurality of pieces of skin color data corresponding to the gender and race indicated by the received information are transmitted to the terminal device 200.

In S127 of FIG. 3, the CPU 210 of the terminal device 200 receives the style image data SD and skin color data transmitted from the server 100.

In S130 of FIG. 3, the CPU 210 selects a region of interest from the regions of facial parts (eyes, nose, mouth) specified in the input image Iin.

In S135 of FIG. 3, the CPU 210 displays a selection screen for the region of interest on the display device 250. The selection screen UDb in FIG. 5(B) is a selection screen for the eye area. The selection screen UDb includes an input image Iin, a selection window SWb for inputting an eye style image selection instruction, a slide bar SBb for inputting the intensity of eye style conversion, and buttons BT1 and BT2. Contains. The selection window SWb includes, as options, a plurality of style images SIe1 and SIe2 indicated by the plurality of style image data SD of eyes received in S127. The selection screen UDc in FIG. 5C is a selection screen for the nose region. The selection screen UDc includes an intermediate image Ima to be described later, a selection window SWc for inputting an instruction to select a nose style image, a slide bar SBc for inputting the intensity of nose style conversion, and buttons BT1 and BT2. , contains. The selection window SWc includes, as options, a plurality of style images SIn1 and SIn2 indicated by the plurality of style image data SD of the nose received in S127. Illustration of the selection screen for the mouth area is omitted.

In S140 of FIG. 3, the CPU 210 transmits information indicating the style image and intensity selected by the user to the server 100. For example, when the attention area is the eye area, the user selects one style image to be used from among the multiple style images SIe1 and SIe2 displayed in the selection window SWb of FIG. 5(B). select. The user operates the knob of the slide bar SBc to move it to a position corresponding to the strength to be used. After that, the user presses button BT2. When the button BT2 is pressed, the CPU 210 displays information indicating the style image selected in the selection window SWb (e.g. image ID) and information indicating the intensity corresponding to the position of the knob of the slide bar SBb (e.g. , and the above-mentioned parameter α) are transmitted to the server 100.

In S225 of FIG. 6, the CPU 110 of the server 100 receives information indicating the style image and intensity selected for the region of interest from the terminal device 200.

In S227 of FIG. 6, the CPU 110 acquires the style image data SD to be used.
For example, when the attention area is the eye area Per or Pel, the CPU 110 selects the style image data group SDG (FIG. 1) stored in the nonvolatile storage device 130 based on the information received in S225. , obtain the eye style image data SD to be used.

In S230 of FIG. 6, the CPU 110 executes style conversion processing for the region of interest. The CPU 110 extracts two pieces of partial image data representing two partial images PIer and PIel (FIG. 4(A)) corresponding to the eye regions Per and Pel from the input image data. The CPU 110 executes reduction processing or enlargement processing on each of the two partial image data to obtain content image data of two eyes of a predetermined size (in this embodiment, 256 pixels vertically x 256 pixels horizontally). Generate a CD. The CPU 110 inputs the data pair of the right eye content image data CD and the style image data SD acquired in S227 to the eye generation network GN1 to generate right eye converted image data TD. Similarly, the CPU 110 inputs the data pair of left-eye content image data CD and style image data SD to the eye generation network GN1 to generate left-eye converted image data TD. The CPU 110 executes an enlargement process or a reduction process on the two generated converted image data TD, and adjusts the size of the image indicated by the converted image data TD to be the same size as the original partial image. In the following, the converted image data TD whose size has been adjusted will be referred to as converted data. When the region of interest is the nose region Pn or the mouth region Pm, converted data of the nose and mouth is generated by style conversion processing using the nose generation network GN2 and the mouth generation network GN3. .

In S232 of FIG. 6, the CPU 110 generates intermediate image data representing an intermediate image by replacing partial image data corresponding to the region of interest in the input image data with converted data. FIG. 4B shows an intermediate image Ima after the partial image data corresponding to the eye regions Per and Pel have been replaced. In the face FCa of the intermediate image Ima, the eye partial images PIer and PIel of the input image Iin in FIG. 4A are replaced with converted partial images TIer and TIel indicated by the converted data. In the intermediate image Ima, a streak BL located at the boundary between the converted partial images TIer, TIel and other parts appears. This is because the pixel values do not change smoothly at the boundaries between the converted partial images TIer and TIel and other parts, and the difference in pixel values becomes large.

In S235 of FIG. 6, CPU 110 transmits intermediate image data to terminal device 200.

In S240 of FIG. 6, the CPU 110 determines whether all regions of the face have been processed. If there is an unprocessed area (S240: NO), the process returns to S225. If the regions of all parts have been processed (S240: YES), the process proceeds to S245.

In S145 of FIG. 3, the CPU 210 of the terminal device 200 receives intermediate image data transmitted from the server 100. In S147, CPU 210 updates the selection screen displayed on display device 250 using the intermediate image data. For example, on the selection screen UDc in FIG. 5(C), an intermediate image Ima (FIG. 4(B)) indicated by intermediate image data is displayed instead of the input image Iin. The user can view the intermediate image Ima displayed on the display device 250 and check the result of style conversion of the region of interest. Although omitted in the flowchart, if the user is not satisfied with the result of the style conversion of the attention area, by pressing button BT1, the user can perform steps S135 to S147 of FIG. 3 again for the already processed attention area, and S225 to S235 in FIG. 6 can be repeated.

In S150 of FIG. 3, the CPU 210 determines whether all regions of the face have been processed. If there is an unprocessed area (S150: NO), the process returns to S130. If the regions of all parts have been processed (S150: YES), the process proceeds to S155.

When the process proceeds to S155, intermediate image data representing the intermediate image Imb of FIG. 4(C) is generated in the server 100 and transmitted to the terminal device 200. In the face FCb of the intermediate image Imb, the partial images PIer, PIel, PIn, and PIm of each part of the input image Iin in FIG. has been replaced. In the intermediate image Imb, the above-described streaks BL appear at the boundaries between the converted partial images TIer, TIel, TIn, and TIm and other parts.

In S155 of FIG. 3, the CPU 210 of the terminal device 200 displays the skin color selection screen UDd of FIG. 5(D) on the display device 250. The selection screen UDd in FIG. 5(D) includes an intermediate image Imb (figure), a selection window SWd for inputting a skin color selection instruction, and buttons BT1 and BT2. The selection window SWd includes, as options, rectangular images CP1 and CP2 having the skin color indicated by the plurality of skin color data received in S127.

In S160 of FIG. 3, CPU 210 transmits information indicating the skin color selected by the user to server 100. For example, the user selects one image from among the plurality of rectangular images CP1 and CP2 displayed in the selection window SWd of FIG. 5(D) and presses the button BT2. When the button BT2 is pressed, the CPU 210 transmits information (for example, an ID such as a color number) indicating the skin color of the rectangular image selected in the selection window SWd to the server 100.

At 245 in FIG. 6, the CPU 110 of the server 100 receives information indicating the selected skin color from the terminal device 200. In S250 of FIG. 6, the CPU 110 performs skin color correction on the input image data acquired in S205 to generate corrected input image data. A known correction process is used for the skin color correction process. For example, the CPU 110 executes recognition processing using a known face recognition algorithm on the input image data, and specifies the area of the person's face FC in the input image Iin. The face recognition algorithm uses, for example, the above-mentioned YOLO convolutional neural network that is trained to recognize the region of a person's face. The CPU 110 specifies a skin color pixel having an RGB value within a predetermined range indicating skin color among the plurality of pixels in the area of the person's face FC, and calculates the average RGB value of the specified plurality of skin color pixels. do. The CPU 110 determines the amount of correction for each RGB component based on the difference between the average RGB value of the skin color pixels and the RGB value indicating the skin color selected by the user. The CPU 110 determines a tone curve for each RGB component according to the correction amount, and uses the tone curve to correct the RGB values of the specified plurality of skin color pixels. FIG. 4(D) shows a corrected image Ic represented by corrected input image data. The person's face FCc in the corrected image Ic has a skin color selected by the user.

In S255, the entire face style conversion process is performed on the intermediate image data to generate output image data. For example, the CPU 110 performs reduction processing or Execute enlargement processing. As a result, the sizes of the intermediate image Imb and the corrected image Ic are adjusted to a predetermined size (in this embodiment, 256 pixels vertically by 256 pixels horizontally). The CPU 110 inputs the size-adjusted intermediate image data as content image data CD and the size-adjusted corrected input image data as style image data SD to the face generation network GN4. Thus, converted image data TD of the entire face is generated. The CPU 110 executes an enlargement process or a reduction process on the generated converted image data TD of the entire face, and adjusts the size of the image represented by the converted image data TD to the same size as the original input image Iin. . The converted image data TD after the size adjustment is output image data indicating the final output image Iout. In the face generation network GN4, the parameter α indicating the strength is set to a smaller value than the parameter α in the above-described style conversion process for each part of the face (S230 in FIG. 6). This is to prevent the features of each part of the face appearing in the intermediate image Imb through the style conversion process for each part from being lost due to the style conversion process for the entire face. Even when the value of the parameter α is relatively small, the overall characteristics such as the skin color of the face are reflected in the output image Iout.

FIG. 4(E) shows an example of the output image Iout. The human face FCo in the output image Iout has the characteristics of the facial region in the intermediate image Imb, and the skin color of the face FCo is close to the skin color of the face FCc in the corrected image Ic. Furthermore, in the human face FCo of the output image Iout, the streaks BL are less noticeable than in the intermediate image Imb. That is, in the output image Iout, the difference in pixel values at the boundaries forming the streaks BL is reduced. This is because the face FCc of the corrected image Ic used as a style image does not include streaks BL, so when the style of the corrected image Ic is applied to the intermediate image Imb through style conversion processing, the streaks BL are reduced. be.

In S260, the CPU 110 transmits the generated output image data to the terminal device 200 and ends the process.

In S165 of FIG. 3, the CPU 210 of the terminal device 200 receives the output image data transmitted from the terminal device 200. In S170, CPU 210 outputs output image data. The output mode of the output image data includes, for example, display, storage, and printing. For example, the CPU 210 displays the output image Iout indicated by the output image data on the display device 250. For example, the CPU 210 stores a file containing output image data in the volatile storage device 120 and the nonvolatile storage device 130 based on a user's instruction. For example, the CPU 210 uses the output image data to generate print data representing the output image Iout, and sends it to a printer (not shown).

In the first embodiment described above, the CPU 110 of the server 100 acquires input image data (S205 in FIG. 6), and uses the input image data to generate a first input partial image that is a part of the input image Iin (for example, , partial images PIer, PIel corresponding to the eye regions Per, Pel), and a second input partial image that is part of the input image and located at a different position from the first input partial image (for example, a nose region Pn). The corresponding partial image PIn) is specified (S210 in FIG. 6). The CPU 110 uses a machine learning model (for example, eye generation network GN1) on first partial image data indicating the first input partial image (for example, partial image data indicating eye partial images PIer, PIel). The first style conversion process is executed to perform first converted data (for example, the converted partial images TIer, TIel of the eyes) indicating the first converted partial images (for example, the converted partial images TIer, TIel of the eyes). (converted data) is generated (S230 in FIG. 6). The CPU 110 generates a second partial image data representing the second input partial image (for example, partial image data representing the nose partial image PIn) using a machine learning model (for example, the nose generation network GN2). Execute the style conversion process to generate second converted data (for example, converted data indicating the converted nose partial image TIn) indicating the second converted partial image (for example, the converted partial image TIn of the nose) (S230 in FIG. 6). The CPU 110 uses the first converted data and the second converted data to generate output image data indicating the output image Iout based on the input image Iin (S232, S250, and S255 in FIG. 6). The output image Iout in FIG. 4(D) includes a first output partial image (for example, eye partial images OIer, OIel) corresponding to the first input partial image and a second output partial image corresponding to the second input partial image. (nose partial image OIn). The first output partial image (for example, the eye partial image OIer, OIel) is an image based on the first converted partial image (for example, the eye converted partial image TIer, TIel). The second output partial image (eg, eye partial image OIn) is an image based on the second converted partial image (eg, nose converted partial image TIn). According to the first embodiment, the output image data is generated by applying the first style conversion process and the second style conversion process to one piece of input image data, so the style can be flexible. Conversion can be realized.

Furthermore, in the above embodiment, the first style conversion process (for example, the style conversion process for the eye areas Per and Pel) uses the style image data SD indicating the first style image (for example, the eye style image SIe1). The second style conversion process (for example, the style conversion process for the nose region Pn) is executed using the style image data SD indicating the second style image (for example, the nose style image SIn1) (FIG. 2 (B) etc.). The first converted partial images (for example, the converted eye partial images TIer, TIel) are such that the style of the first style image (for example, the eye style image SIe1) is the same as that of the first input partial image (for example, the eye partial image TIer, TIel). PIer, PIel), and the second converted partial image (for example, the converted nose partial image TIn) is an image in which the style of the second style image (for example, the nose style image SIn1) is the second converted partial image (for example, the nose style image SIn1). This is an image applied to an input partial image (for example, nose partial image PIn). As a result, output image data indicating an output image to which the style of the first style image and the style of the second style image are applied can be generated, so that more flexible style conversion can be realized.

Further, the CPU 110 uses the first converted data and the second converted data to convert the first converted partial images (for example, the converted partial images TIer and TIel of the eyes) and the second converted partial images (for example, Intermediate image data indicating an intermediate image (for example, intermediate image Imb) including the converted partial image TIn of the nose is generated (S232 in FIG. 6, FIG. 4C). The CPU 110 performs specific post-processing (S255 in FIG. 6) on the intermediate image data to generate output image data. As a result, appropriate output image data can be generated by performing specific post-processing.

Specifically, as specific post-processing in this embodiment, style conversion processing for the entire face (S255 in FIG. 6) is performed. Through this processing, as described above, in the intermediate image Ima, there is a gap between the converted partial images (for example, the converted partial images TIer, TIel, TIn of eyes and nose) and the portion adjacent to the one converted partial image. The difference in pixel values at is reduced. As a result, for example, in the output image Iout, the streaks BL appearing in the intermediate image Ima are not noticeable. In this way, output image data can be generated so that the output image Iout has a natural appearance.

Furthermore, the overall style conversion process for the face (S255 in FIG. 6) of this embodiment is a third style conversion process using a machine learning model (for example, the face generation network GN4). As a result, more flexible style conversion can be achieved by performing style conversion processing on a partial image, style conversion on the entire image, and third style conversion processing.

Further, the third style conversion process (the style conversion process for the whole face in S255 in FIG. 6) of this embodiment is executed using the input image data as the style image data SD. As a result, for example, it is possible to easily generate output image data showing an output image having a natural appearance in which the above-described streaks BL are not noticeable.

Furthermore, the specific post-processing of this embodiment is a process of executing a process of correcting the skin color of a person's face FC on input image data to generate corrected input image data (S250 in FIG. 6). including. Then, the third style conversion process (the overall style conversion process of the face in S255 in FIG. 6) is executed using the corrected input image data as the style image data SD. As a result, the skin color of the person's face in the corrected input image (corrected image Ic in FIG. 4(D)) is applied to the output image Iout as a style. Therefore, output image data representing an output image Iout having an arbitrary skin color can be easily generated.

Furthermore, in this embodiment, as described above, the input image Iin includes an image showing the person's face FC (FIG. 4(A)), and the first input partial images (for example, partial images PIer, PIel) are The image is an image showing a first part (e.g., eyes) constituting the person's face FC, and the second input partial image (e.g., partial image PIn) is an image showing the second part (e.g., This is an image showing the nose. As a result, flexible style conversion can be realized for the first part and the second part that make up a person's face. For example, if you select the eye image of person A as the eye style image and the nose image of person B as the nose style image, the eyes of the person's face FC in the input image Iin will be brought closer to the eyes of person A. , the style can be converted so that the nose of the face FC approaches the nose of the person B.

Further, in this embodiment, the type of input image Iin (for example, the gender and race of the person) is specified by receiving information from the terminal device 200 (S215 in FIG. 6). Then, candidates for style image data SD to be used in the style conversion process of S230 are changed depending on the type of input image Iin (S220 in FIG. 6). That is, in S230, different style conversion processes are performed depending on the type of input image Iin. In other words, when the input image Iin is a first type input image (for example, an input image of a woman's face), the first type style conversion process is performed on partial image data of each part of the face. , when the input image Iin is a second type input image (for example, an input image of a man's face), the second type style conversion process is performed on partial image data of each part of the face. As a result, flexible style conversion can be realized depending on the type of input image Iin. For example, it is considered that the style conversion preferred by the user may differ depending on the gender, race, etc. of the person in the input image Iin, so according to this embodiment, style conversion that meets the user's needs can be realized.

Furthermore, according to this embodiment, the user can set the parameter α indicating the strength of style conversion for each part of the face by operating the slide bars SBb and SBc on the selection screens UDb and SDc (Fig. 5(B) ), (C), S140 in FIG. 3, S225 in FIG. 6). In other words, the first style conversion process (e.g., eye style conversion process) is performed using the first parameter α1, and the second style conversion process (e.g., nose style conversion) is performed using the first parameter α1. is performed using an independently adjusted second parameter α2. As a result, more flexible style conversion can be achieved. For example, it is possible to easily generate output image data that shows an output image Iout in which the eyes have a large difference from the input image Iin, and the nose has a small difference from the input image Iin. As a result, for example, even if the number of prepared style image data SD is relatively small, flexible and various style conversions can be realized.

Further, according to the present embodiment, the CPU 110 obtains the eye style image data SD based on the user's instruction to select the eye style image (FIG. 5(B)), and the user's selection of the nose style image. Based on the instruction (FIG. 5(C)), a nose style image is acquired (S227 in FIG. 6). The eye and nose style conversion process is performed using the acquired eye and nose style image data SD (S230 in FIG. 6). As a result, flexible style conversion can be realized in response to a user's instruction to select a style image. For example, by inputting a selection instruction, the user can cause the server 100 to generate output image data indicating the output image Iout to which a style similar to the eyes and nose is applied, or a style that is significantly different for the eyes and nose. It is also possible to cause the server 100 to generate output image data indicating the output image Iout to which the Iout has been applied.

As can be seen from the above description, the instruction to select the eye style image is an example of the first input, and the instruction to select the nose style image is an example of the second input. Further, the eye style image data SD obtained based on the selection instruction of the eye style image is an example of the first input information, and the nose style image data obtained based on the selection instruction of the nose style image. SD is an example of second input information.

B. Second Example B-1. Configuration of System 1000 The basic configuration of the system 1000 of the second embodiment is the configuration shown in FIG. 1 as in the first embodiment. I will explain about it.

In addition to the configuration of the first embodiment, a system 1000 according to the second embodiment includes a sewing machine 300 that is communicably connected to a terminal device 200. The sewing machine 300 sews an embroidery pattern onto the cloth by sewing threads of multiple colors onto the cloth based on the embroidery data.

The terminal device 200 of the second embodiment is a stationary terminal device such as a personal computer. The computer program PGt stored in the volatile storage device 230 of the terminal device 200 of the second embodiment is a driver program that controls the sewing machine 300. The computer program PGt is provided by the manufacturer of the sewing machine 300, and is provided in the form of being downloaded from a server connected to the terminal device 200 via the Internet IT. Alternatively, the computer program PGt may be provided in a form stored on a CD-ROM, DVD-ROM, or the like. By executing the computer program PGt, the CPU 210 cooperates with the server 100 to perform a process of generating embroidery data, which will be described later, and supplying it to the sewing machine 300.

The computer program PGs stored in the nonvolatile storage device 130 of the server 100 of the second embodiment is provided by the manufacturer of the sewing machine 300 and uploaded to the server 100. By executing the computer program PGs, the CPU 110 cooperates with the terminal device 200 to perform a process of generating embroidery data, which will be described later, and supplying it to the sewing machine 300.

B-2. Configuration of Generation Network Group In the second embodiment, the input image Iin is an image showing a photograph including the entire face FC of a person, as in the first embodiment. When generating embroidery data from image data such as a photograph, it is common to perform preprocessing on the image data and generate the embroidery data using the preprocessed image data. The number of colors of thread used to sew the embroidery pattern (e.g., several dozen colors) is smaller than the number of colors represented in the photograph (e.g., about 10 million colors), and the outlines are clear. This is because it is preferable. Such preprocessing is generally performed by an experienced operator using an image processing program (also called photo retouching software). In the second embodiment, style conversion processing is used to generate output image data representing the preprocessed output image Iout using input image data.

The generation network group GNG of the second embodiment includes generation networks GN1 to GN4, as in the first embodiment. In the second embodiment, style conversion of each part of the face is performed so that the output image Iout becomes an image suitable for generating embroidery data. For this reason, the style image represented by the style image data SD used in training the generation networks GN1 to GN4 and in generating embroidery data, which will be described later, is a preprocessed image suitable for generating embroidery data. There are many preprocessing methods, such as methods for clarifying outlines, adding shading, and adjusting colors, and these methods vary depending on the operator. For this purpose, a plurality of images that have been preprocessed using various methods are used as style images.

For example, the generation network GN1 for eyes uses a large number of images obtained by preprocessing photos of various eyes using various methods as style image data SD for training. In addition, when generating embroidery data, the style image data SD that can be selected via the selection screen UDb in FIG. A plurality of style image data SD representing a plurality of style images are used.

The generation network group GNG of the second embodiment further includes a generation network GN5 for facial expressions and a generation network GN6 for dentition.

The generative network GN5 for facial expressions is a machine learning model, and is a generative network that constitutes generative adversarial networks (GANs) called StarGAN. The facial expression generation network GN5 executes style conversion processing to change facial expressions. Specifically, when image data indicating a person's face and label data indicating the type of facial expression are input to the facial expression generation network GN5, the facial expression generation network GN5 outputs converted image data. . The converted image shown by the converted image data is the face of the person shown by the input image data, and shows the face having the expression shown by the label data. In this embodiment, the facial expression generation network GN5 is configured to be able to convert facial expressions such as a neutral expression, a smile that does not show teeth, a grin that shows teeth, and a serious face. Has been trained. StarGAN is disclosed in the paper "Yunjey Choi et al., "StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation", arXiv preprint arXiv:1711.09020, 2017."

The generation network GN6 for the dentition is a machine learning model similar to the generation networks GN1 to GN4 described above. The generation network GN6 for dentition receives image data showing the face of a person with an expression with exposed teeth as content image data CD, and generates a generation network GN6 of a person with an expression with exposed teeth and whose dentition has been corrected. Image data showing a face is input as style image data SD. The image represented by the converted image data TD output by the generation network GN6 is the face of the person represented by the content image data CD, and is the face of a person whose teeth have been corrected.

B-3. System Operation FIG. 7 is a flowchart of processing executed by the terminal device 200. This process uses a preprocessing service using style conversion provided by the server 100 to perform preprocessing on input image data, obtains output image data, and uses the output image data to perform embroidery. This is the process of generating data. This process is started based on a user's start instruction, for example, with the computer program PGt of the terminal device 200 being executed.

In S305 of FIG. 7, the CPU 210 of the terminal device 200 obtains input image data indicating the input image Iin including the face FC of the person shown in FIG. 4(A). Note that in the first and second embodiments, although the images that are expected to be used (for example, input images, style images, and output images) are not the same, images showing similar human faces and body parts are used. Therefore, for convenience of explanation, the same drawings and the same reference numerals will be used for explanation. For example, the CPU 210 obtains image data specified by the user from among a plurality of pieces of image data stored in the nonvolatile storage device 130 as input image data.

In S310, CPU 210 displays selection screen UD including input image Iin on display device 250. FIG. 8 is a diagram showing the selection screen UD of the second embodiment. The selection screen UD in FIG. 8 includes an input image Iin, pull-down menus PM1 to PM3, selection windows SWa to SWd, slide bars SBa to SBc, check boxes CBa and CBb, and buttons BT3 and BT4. There is.

The pull-down menus PM1 and PM2 are menus for inputting selection instructions regarding the type of input image Iin (specifically, selection instructions for gender and race), and are similar to the pull-down menus in FIG. 5(A) of the first embodiment. This menu is similar to menus PM1 and PM2. The pull-down menu PM3 is a menu for inputting selection instructions for whether or not to change the facial expression using the above-mentioned facial expression generation network GN5, and the type of facial expression after the change when changing the facial expression. be.

The selection windows SWb and SWc are selection windows for inputting selection instructions for eye and nose style images, and have the same menus as the selection windows SWb and SWc in FIGS. 5(B) and 5(C) of the first embodiment. It is. In the selection window SWa, a plurality of style images Sm1 and Sm2 indicated by a plurality of style image data SD of the mouth are displayed as options. Note that the style images in each selection window are not displayed at this point, but will be displayed in S335, which will be described later.

Slide bars SBa to SBc are slide bars for inputting the strength of style conversion of the mouth, eyes, and nose, similar to slide bars SBb and SBc in FIGS. 5(B) and 5(C).

The check box CBa is a check box for specifying whether or not to perform white eye processing, which will be described later. The check box CBb is a check box for specifying whether or not to correct the dentition using the generation network GN6 for the dentition.

In S315 of FIG. 7, the CPU 210 transmits the input image data to the server 100, similar to S115 of FIG.

When the server 100 receives input image data transmitted from the terminal device 200, the CPU 210 of the server 100 starts processing to provide a preprocessing service using style conversion processing. FIG. 9 is a flowchart of processing executed by the server 100 of the second embodiment. As shown in S405 of FIG. 9, the CPU 110 of the server 100 executes the processes of S205 to S220 of FIG. 6 while exchanging data with the terminal device 200, as in the first embodiment.

In S205 of FIG. 6, the CPU 110 of the server 100 receives input image data transmitted from the terminal device 200. In S210, the CPU 110 executes a predetermined area specifying process on the input image data to identify a plurality of parts included in the face FC of the input image Iin, that is, areas Per and Pel of the right eye, left eye, nose, and mouth. , Pn, and Pm. In S212, CPU 110 transmits area information indicating areas Per, Pel, Pn, and Pm of a plurality of body parts to terminal device 200.

In S320 of FIG. 7, similarly to 120 of FIG. The identification results of Pn and Pm are displayed on the display device 250. In S325 of FIG. 7, similarly to S125 of FIG. 3, the CPU 210 transmits information on the gender and race selected by the user to the server 100.

In S215 of FIG. 6, the CPU 110 of the server 100 receives the gender and race information transmitted from the terminal device 200. In S220, CPU 110 transmits style image data SD and skin color data according to the gender and race indicated by the received information to terminal device 200.

In S330 of FIG. 7, the CPU 210 of the terminal device 200 receives the style image data SD and skin color data transmitted from the server 100. In S335 of FIG. 7, the style images SIm1, SIm2, SIe1, SIe2, SIn1, and SIn2 of the mouth, eyes, and nose indicated by the received style image data SD are displayed in the corresponding selection windows SWa, SWb, and SWc ( Figure 8).

In S340 of FIG. 7, the CPU 210 transmits information for the conversion process selected on the selection screen UD to the server 100. The user selects the style image to be used for each part of the face, the strength of style conversion for each part, and the skin color that the face of the output image should have, via the selection windows SWa to SWd and slide bars SBa to SBc in FIG. Enter instructions. The user inputs selection instructions as to whether or not to perform white eye treatment and whether to perform orthodontic correction via check boxes CBa and CBb. The user inputs selection instructions for whether or not to change the facial expression and the type of facial expression after the change if the facial expression is to be changed, via the pull-down menu PM3. However, if a selection instruction to perform the white eye processing is input, the eye style image selection window SWb is invalidated. That is, only one of the instruction to select to perform the white eye processing and the instruction to select the eye style image is valid. This is because, as will be described later, only one of the white eye processing and the eye style conversion processing can be executed in the server 100. Thereafter, with the selection instruction input, the user presses the button BT3 for instructing execution of preprocessing. CPU 210 transmits to server 100 information corresponding to the selection instruction input at the time button BT2 was pressed.

In S410 of FIG. 9, the CPU 110 of the server 100 receives information for conversion processing from the terminal device 200.

In S415 of FIG. 9, the CPU 110 determines whether execution of the white eye process has been selected based on the information received in S410. If executing the white eye processing is selected (S415: YES), in S420, the CPU 110 executes the white eye processing on the input image data. The white eye processing is a process of filling the white part of the image showing the eyes with a specific white color that looks good in the eye regions Per and Pel. For example, the CPU 110 converts the value of a pixel corresponding to the white part of the eye to a specific value indicating white (for example, an RGB value of (255, 255, 255)). For example, pixels having values in a predetermined range indicating white and a color close to white are specified as pixels corresponding to the white part of the eye. As a result, for example, the cloudiness of the whites of the eyes in the input image Iin is reduced, and the appearance of the eyes of the person's face expressed by the embroidery pattern is improved. The white eye processing can be said to be a process of converting the color of at least a portion of the eye partial images PIer and PIel without using a machine learning model.

In S425 of FIG. 9, the CPU 110 excludes the eye areas Per and Pel from the style conversion target area. This is because if the style conversion process is performed after the white of the eyes process is executed, the white of the eyes may become cloudy in the image after the style conversion process, which reduces the effectiveness of the white of the eyes process.

If executing the white eye processing is not selected (S415: NO), the CPU 110 skips S420 and S425 and advances the process to S430.

In S430 of FIG. 9, the CPU 110 selects a region of interest from target regions to be subjected to style conversion processing among regions of facial parts (eyes, nose, mouth) specified in the input image Iin. When the eye area is excluded from the target area, the target area is the mouth and nose area Pn, Pm, and when the eye area is not excluded from the target area, the target area is the eye area. and the mouth and nose areas Per, Pel, Pn, and Pm.

In S435 of FIG. 9, the CPU 110 selects the style image data group SDG (FIG. 1) stored in the non-volatile storage device 130 based on the information received in S410, and determines which image data should be used in the style conversion process of the region of interest. Acquire style image data SD.

In S440 of FIG. 9, the CPU 110 executes style conversion processing for the region of interest, similar to S230 of FIG. In S442, similarly to S232 in FIG. 6, the CPU 110 generates intermediate image data representing an intermediate image by replacing partial image data corresponding to the region of interest in the input image data with converted data.

In S445 of FIG. 9, the CPU 110 determines whether all target areas have been processed. If there is an unprocessed area (S445: NO), the process returns to S430. If all target areas have been processed (S445: YES), the process proceeds to S450.

In S450 of FIG. 9, CPU 110 performs skin color correction on the input image data to generate corrected input image data, similar to S250 of FIG. In S455 of FIG. 9, the CPU 110 performs overall face style conversion processing on the intermediate image data to generate output image data, similar to S255 of FIG.

In S460 of FIG. 9, CPU 110 determines whether or not it has been selected to change the facial expression based on the information received in S410. If it is selected to change the facial expression (S460: YES), in S420, the CPU 110 further executes a style conversion process for changing the facial expression on the output image data. For example, the CPU 110 determines the type of facial expression after the change (e.g., a smile that does not show teeth, a straight face) based on the information received in S410, and generates label data indicating the type of facial expression after the change. do. The CPU 110 inputs the output image data and label data to the facial expression generation network GN5, thereby generating output image data indicating an output image (not shown) including a person's face whose facial expression has been changed.

If it is not selected to change the facial expression (S460: NO), the CPU 110 skips S465 and proceeds to S470.

In S470 of FIG. 9, CPU 110 determines whether or not it has been selected to perform orthodontic correction based on the information received in S410. If it is selected to change the facial expression (S470: YES), in S475 of FIG. 9, the CPU 110 executes a style conversion process for correcting the tooth alignment. For example, the CPU 110 inputs the output image data as the content image data CD and the image data prepared in advance showing the face of a person whose dentition has been corrected as the style image data SD to the generation network GN6 for the dentition. This generates output image data representing an output image (not shown) including the face of a person whose dentition has been corrected.

If it is not selected to perform tooth alignment correction (S470: NO), the CPU 110 skips S475 and proceeds to S480.

If neither facial expression change nor tooth alignment correction is performed, the output image data generated in S455 is the final output image data. If the facial expression is changed and the tooth alignment is not corrected, the output image data generated in S465 is the final output image data. When correction of tooth alignment is performed, the output image data generated in S475 is the final output image data.

In S480 of FIG. 9, the CPU 110 transmits the final output image data to the terminal device 200, and ends the process.

In S345 of FIG. 7, the CPU 210 of the terminal device 200 receives the output image data transmitted from the terminal device 200. In S350, CPU 210 displays the output image on display device 250 using the output image data. Specifically, the output image is displayed in place of the input image Iin on the selection screen UD in FIG. The user checks the output image on the selection screen UD, and if satisfied with the output image, presses the output button BT4. When the user wants to generate the output image again, the user changes the input content of the selection instruction as appropriate on the selection screen UD and presses the preprocessing button BT3.

In S355 of FIG. 9, the CPU 210 determines whether the output button BT4 or the preprocessing button BT3 has been pressed. If the output button BT4 is pressed (S355: YES), the CPU 210 advances the process to S360. If the preprocessing button BT3 is pressed (S355: NO), the CPU 210 returns to S340.

In S360, the CPU 210 converts the output image data into embroidery data. Embroidery data is data that represents an embroidery pattern, and for example, data that indicates, for each stitch, the coordinates of the needle drop point, the sewing order, and the color of thread to be used to form the stitches of the embroidery pattern. It is. The process of converting the output image data into embroidery data uses a known process, for example, the process disclosed in Japanese Patent Application Publication No. 2019-41834.

In S365, CPU 210 transmits the embroidery data to sewing machine 300. Upon receiving the embroidery data, the sewing machine 300 sews an embroidery pattern onto cloth using the embroidery data.

According to the second embodiment described above, when generating output image data, flexible style conversion processing can be realized as in the first embodiment. As a result, it is possible to generate output image data that has undergone flexible preprocessing according to the user's preferences, for example. Therefore, for example, even if the user does not have the skills to perform preprocessing using a general image processing program, various embroidery patterns according to the user's preferences can be printed on cloth.

For example, according to the second embodiment, a style conversion process (S465 in FIG. 9) for changing the expression of a person's face is executed as a specific post-process. As a result, flexible style conversion including changing the expression of a person's face can be realized. For example, by simply preparing one piece of input image data, the user can have the system 1000 generate output image data showing faces with various expressions, and can even create embroidery patterns of faces with various expressions on the sewing machine. 300 can be sewn.

Further, according to the second embodiment, the CPU 110 selects the process to be performed on the partial image data representing the eye partial images PIer and PIel from the white of the eye process and the style conversion process (S415 in FIG. 9). . When the style conversion process is selected, the CPU 210 executes the style conversion process without executing the pewter process, and when the pewter process is selected, the CPU 210 executes the pewter process without executing the style conversion process. do. As a result, style conversion processing using a machine learning model and white eye processing not using a machine learning model can be used as processing for eye partial image data, thereby improving processing flexibility. For example, a user may prefer simple white eye processing to style conversion processing as processing for eyes, and the present embodiment can also meet the needs of such users.

Furthermore, according to the second embodiment, style conversion processing for correcting the tooth alignment in the image showing the mouth is executed (S475 in FIG. 9). As a result, output image data showing an image with corrected tooth alignment can be easily generated.

B. Variant:
(1) In each of the above embodiments, different style image data SD are used depending on the race and gender of the person included in the input image Iin. For example, different style image data SD may be used depending on the facial expression (e.g., angry, laughing, straight face) or the angle of the face (e.g., front, side, diagonal) included in the input image Iin. It's okay. Further, in the above embodiment, the type of these input images Iin is specified based on the user's selection instruction, but it may also be specified using an image recognition process, for example, the above-mentioned image recognition algorithm called yolo. It's okay.

(2) In each of the above embodiments, the target parts of the style conversion process for each part (S230 in FIG. 6, S440 in FIG. 9) are the eyes, nose, and mouth. The target site is not limited to this, and may be other sites such as the head (hair), ears, cheeks, and chin.

(3) In each of the above embodiments, the input image Iin is not limited to an image including a person's face FC, but may be another image. For example, the input image Iin may be an image that includes landscapes, animals, and buildings, but does not include people. Regardless of which image is used as the input image, the first partial image that is part of the image and the second partial image located at a different position from the first partial image undergo different style conversion processing. is preferably performed.

(4) The generative network (machine learning model) used in each of the above embodiments is an example, and is not limited to this. For example, a common generation network may be used for eyes, nose, and mouth. Furthermore, for example, a generation network may be used that is capable of converting style images used during training into only one type of style. In this case, for example, generation networks for the number of selectable style images may be prepared for style conversion of one part (e.g., nose), and used depending on the selected style image. .

(5) In each of the above embodiments, the style image data SD is selected from the style image data group SDG stored in the server 100. Alternatively, the style image data SD may be image data prepared by the user. In this case, the user inputs the prepared style image data SD into the terminal device 200. The input style image data SD is transmitted from the terminal device 200 to the server 100, and is used for style conversion processing in the server 100.

(6) In each of the above embodiments, the CPU 110 acquires the style image data SD selected by the user (for example, S227 in FIG. 6), inputs the style image data SD to the generation network, and executes the style conversion process. (For example, S230 in FIG. 6). Alternatively, a plurality of style image data SD may be input in advance to the encoder EC of the generation network GN to generate a plurality of feature data. In this case, feature data corresponding to the style image data SD selected by the user may be acquired, and the style conversion process may be performed using the feature data.

(7) In each of the above embodiments, the streaks BL appearing in the intermediate image Imb of FIG. 4(C) are removed by executing style conversion processing for the entire face (for example, S255 in FIG. 6) as specific post-processing. It is being reduced. Instead of this, other processing, for example, smoothing processing using a filter, may be performed on the pixels of the streak BL portion. Generally, processing is performed to reduce the difference in pixel values between a portion constituting the streak BL, for example, a converted partial image TIer in FIG. 4(C) and a portion adjacent to the converted partial image TIer. is preferably performed.

(8) The processing in each of the above embodiments is merely an example, and may be omitted, added, or otherwise modified as appropriate. For example, all or part of the white eye processing in S420, the style conversion process in S460, and the style conversion process in S475 in FIG. 9 may be omitted. Further, these processes may be executed as appropriate during the process shown in FIG. 6 of the first embodiment. In FIG. 6 or 9, the entire face style conversion process (S255 in FIG. 6, S455 in FIG. 9) may be omitted. Further, the style conversion strength parameter α may be a fixed value, or a common value may be used in the style conversion of each region.

(9) All or part of the processing executed by the server 100 in each of the above embodiments may be executed by the terminal device 200. For example, the identification of the region of the face in S210 of FIG. 6 may be performed by the CPU 210 of the terminal device 200. Further, the converted data corresponding to the region of each part generated in S230 of FIG. 6 is transmitted to the terminal device 200, and the terminal device 200 uses the input image data and the converted data to generate intermediate image data, Alternatively, final output image data may be generated.

(10) The hardware configurations of the server 100 and the terminal device 200 in FIG. 1 are merely examples, and are not limited thereto. For example, the processor of the server 100 or terminal device 200 that performs the processing of each embodiment is not limited to a CPU, but may also be a GPU (Graphics Processing Unit), an ASIC (application specific integrated circuit), or a combination of these and a CPU. Also good. Further, the server 100 may be a plurality of computers (for example, a so-called cloud server) that can communicate with each other via a network.

(11) In each of the above embodiments, part of the configuration realized by hardware may be replaced with software, or conversely, part or all of the configuration realized by software may be replaced by hardware. You can do it like this. For example, the generation networks GN1 to GN6 may be realized by a hardware circuit such as an ASIC (Application Specific Integrated Circuit) instead of a program module.

Although the present invention has been described above based on examples and modifications, the embodiments of the invention described above are for facilitating understanding of the present invention, and are not intended to limit the present invention. The present invention may be modified and improved without departing from the spirit and scope of the claims, and the present invention includes equivalents thereof.

100...Server, 1000...System, 110...CPU, 120...Volatile storage device, 130...Nonvolatile storage device, 160...Communication interface, 200...Terminal device, 210...CPU, 220...Nonvolatile storage device, 230...Volatile 240...operation unit, 250...display device, 260...communication interface, 300...sewing machine, CC...characteristic combination unit, CD...content image data, DC...decoder, EC...encoder, GN1 to GN6...generation network, GNG...Generation network group, IT...Internet, Ic...Corrected image, Iin...Input image, Ima, Imb...Intermediate image, Iout...Output image, NW...Wireless network, PGs, PGt...Computer program, SD...Style image data ,SDG...style image data group,TD...converted image data

Claims (24)

an image acquisition step of acquiring input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification step for identifying ,
A first style conversion process using a machine learning model is performed on the first partial image data representing the first input partial image to generate first converted data representing the first converted partial image. 1 conversion step;
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion step of generating second converted data indicating the second converted partial image;
a first step of generating intermediate image data representing an intermediate image including the first converted partial image and the second converted partial image using the first converted data and the second converted data; , a second step of performing specific post-processing on the intermediate image data to generate output image data representing an output image based on the input image, the output image generating step comprising: includes a first output partial image corresponding to the first input partial image and a second output partial image corresponding to the second input partial image, the first output partial image being based on the first transformed partial image. the output image generating step, wherein the second output partial image is an image based on the second converted partial image;
Equipped with
The specific post-processing includes the third style conversion process that is a third style conversion process using a machine learning model and is different from the first style conversion process and the second style conversion process,
The third style conversion process is performed using the input image data as style image data . The image generation method according to claim 1 ,
The input image includes an image showing a person's face,
The specific post-processing includes processing for correcting the skin color of the person's face on the input image data to generate the corrected input image data,
The third style conversion process is performed using the corrected input image data as style image data. an image acquisition step of acquiring input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification step for identifying ,
A first style conversion process using a machine learning model is performed on the first partial image data representing the first input partial image to generate first converted data representing the first converted partial image. 1 conversion step;
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion step of generating second converted data indicating the second converted partial image;
a first step of generating intermediate image data representing an intermediate image including the first converted partial image and the second converted partial image using the first converted data and the second converted data; , a second step of performing specific post-processing on the intermediate image data to generate output image data representing an output image based on the input image, the output image generating step comprising: includes a first output partial image corresponding to the first input partial image and a second output partial image corresponding to the second input partial image, the first output partial image being based on the first transformed partial image. the output image generating step, wherein the second output partial image is an image based on the second converted partial image;
Equipped with
The specific post-processing includes the fourth style conversion process that is a fourth style conversion process using a machine learning model and is different from the first style conversion process and the second style conversion process,
The input image includes an image showing a person's face,
In the image generation method, the fourth style conversion process is a process of changing the facial expression of the person . The image generation method according to any one of claims 1 to 3,
The specific post-processing includes determining, in the intermediate image, the difference in pixel values between the first converted partial image and a portion adjacent to the first converted partial image, and the difference between the second converted partial image and the portion adjacent to the first converted partial image. An image generation method comprising: reducing a difference in pixel values between a second converted partial image and an adjacent portion. an image acquisition step of acquiring input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification step for identifying ,
A first style conversion process using a machine learning model is performed on the first partial image data representing the first input partial image to generate first converted data representing the first converted partial image . 1 conversion step, the first style conversion process uses a first parameter that specifies the degree of difference between the first input partial image and the first converted partial image to be generated. the first conversion step being performed ;
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion step of generating second converted data indicating a second converted partial image, wherein the second style conversion process includes the second input partial image and the second converted portion to be generated. the second transformation step is performed using a second parameter specifying the degree of difference between the images, the second parameter being adjusted independently of the first parameter;
an output image generation step of generating output image data representing an output image based on the input image using the first converted data and the second converted data, the output image being the first input partial image; a first output partial image corresponding to the first output partial image and a second output partial image corresponding to the second input partial image, the first output partial image being an image based on the first converted partial image; the output image generation step, wherein the output partial image is an image based on the second converted partial image;
An image generation method comprising: an image acquisition step of acquiring input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification step for identifying ,
a process selection step of selecting a process to be performed on first partial image data indicating the first input partial image;
a first conversion step of performing a first style conversion process using a machine learning model on the first partial image data to generate first converted data indicating the first converted partial image;
a color conversion step of performing a color conversion process on the first partial image data to convert at least a part of the color of the first input partial image without using a machine learning model;
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion step of generating second converted data indicating the second converted partial image;
an output image generation step of generating output image data representing an output image based on the input image using the first converted data and the second converted data, the output image being the first input partial image; a first output partial image corresponding to the first output partial image and a second output partial image corresponding to the second input partial image, the first output partial image being an image based on the first converted partial image; the output image generation step, wherein the output partial image is an image based on the second converted partial image;
Equipped with
When the first style conversion process is selected in the process selection step, the first conversion process is executed without executing the color conversion process,
An image generation method , wherein when the color conversion process is selected in the process selection step, the color conversion process is executed without executing the first conversion process . The image generation method according to claim 6 ,
The input image includes an image showing a person's face,
The first input partial image is an image showing the eyes of the person,
In the image generation method, the color conversion process is a process of converting a value of a pixel corresponding to the white part of the eye in the image showing the eye to a specific value indicating white. an image acquisition step of acquiring input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification step for identifying ,
Based on the first input by the user, first input information for the first style conversion process using the machine learning model is acquired, and based on the second input by the user, the second style conversion process using the machine learning model is acquired. an information acquisition step of acquiring second input information for the second style conversion process that is a style conversion process and is different from the first style conversion process;
Execute the first style conversion process on the first partial image data indicating the first input partial image using the first input information to generate first converted data indicating the first converted partial image. a first conversion step of generating;
Execute the second style conversion process on the second partial image data indicating the second input partial image using the second input information to generate second converted data indicating the second converted partial image. a second conversion step to generate;
an output image generation step of generating output image data representing an output image based on the input image using the first converted data and the second converted data, the output image being the first input partial image; a first output partial image corresponding to the first output partial image and a second output partial image corresponding to the second input partial image, the first output partial image being an image based on the first converted partial image; the output image generation step, wherein the output partial image is an image based on the second converted partial image;
An image generation method comprising: The image generation method according to claim 8 ,
The first input information includes data indicating an image corresponding to the first input partial image and having a style different from the first input partial image,
The second input information includes data indicating an image corresponding to the second input partial image and having a style different from the second input partial image. The image generation method according to any one of claims 1 to 9 ,
The first style conversion process is performed using first style image data indicating a first style image,
The second style conversion process is performed using second style image data indicating a second style image,
The first converted partial image is an image in which the style of the first style image is applied to the first input partial image,
The second converted partial image is an image in which the style of the second style image is applied to the second input partial image. The image generation method according to any one of claims 1 to 10 ,
The input image includes an image showing a person's face,
The first input partial image is an image showing a first part of the person's face,
In the image generation method, the second input partial image is an image showing the second part of the person's face and located at a different position from the first part. The image generation method according to any one of claims 1 to 11 , further comprising:
comprising a type identifying step of identifying the type of the input image;
When the input image is a first type input image,
In the first conversion step, a first type of first style conversion process is performed on the first partial image data,
In the second conversion step, a first type of second style conversion process is performed on the second partial image data,
When the input image is a second type input image,
In the first conversion step, a second type of first style conversion process is performed on the first partial image data,
In the image generation method, in the second conversion step, a second type of second style conversion process is performed on the second partial image data. The image generation method according to claim 12 ,
The input image includes an image showing a person's face,
The type of the input image is a type related to at least part of the person's gender, race, facial expression, and facial angle. The image generation method according to any one of claims 1 to 13 ,
The input image includes an image showing a person's face,
The second input partial image is an image showing the mouth of the person,
The second style conversion process is a process of correcting the alignment of teeth in the image showing the mouth. an image acquisition unit that acquires input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification unit that identifies ,
A first style conversion process using a machine learning model is performed on the first partial image data representing the first input partial image to generate first converted data representing the first converted partial image. 1 conversion section;
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion unit that generates second converted data indicating the second converted partial image;
a first part that uses the first converted data and the second converted data to generate intermediate image data representing an intermediate image including the first converted partial image and the second converted partial image; , a second part that performs specific post-processing on the intermediate image data to generate output image data representing an output image based on the input image, the output image generation unit comprising: includes a first output partial image corresponding to the first input partial image and a second output partial image corresponding to the second input partial image, the first output partial image being based on the first transformed partial image. an image, and the second output partial image is an image based on the second converted partial image;
Equipped with
The specific post-processing includes the third style conversion process that is a third style conversion process using a machine learning model and is different from the first style conversion process and the second style conversion process,
The system wherein the third style conversion process is executed using the input image data as style image data . an image acquisition unit that acquires input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification unit that identifies ,
A first style conversion process using a machine learning model is performed on the first partial image data representing the first input partial image to generate first converted data representing the first converted partial image. 1 conversion section;
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion unit that generates second converted data indicating the second converted partial image;
a first part that uses the first converted data and the second converted data to generate intermediate image data representing an intermediate image including the first converted partial image and the second converted partial image; , a second part that performs specific post-processing on the intermediate image data to generate output image data representing an output image based on the input image, the output image generation unit comprising: includes a first output partial image corresponding to the first input partial image and a second output partial image corresponding to the second input partial image, the first output partial image being based on the first transformed partial image. an image, and the second output partial image is an image based on the second converted partial image;
Equipped with
The specific post-processing includes the fourth style conversion process that is a fourth style conversion process using a machine learning model and is different from the first style conversion process and the second style conversion process,
The input image includes an image showing a person's face,
The system wherein the fourth style conversion process is a process of changing the facial expression of the person . an image acquisition unit that acquires input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification unit that identifies ,
A first style conversion process using a machine learning model is performed on the first partial image data representing the first input partial image to generate first converted data representing the first converted partial image . 1 conversion unit, wherein the first style conversion process uses a first parameter that specifies the degree of difference between the first input partial image and the first converted partial image to be generated. The first conversion unit is executed ;
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion unit that generates second converted data indicating a second converted partial image, wherein the second style conversion process is performed on the second input partial image and the second converted portion to be generated. and the second transformation unit is performed using a second parameter that specifies the degree of difference between the images, and the second parameter is adjusted independently of the first parameter ;
an output image generation unit that uses the first converted data and the second converted data to generate output image data representing an output image based on the input image, the output image being the first input partial image; a first output partial image corresponding to the first output partial image and a second output partial image corresponding to the second input partial image, the first output partial image being an image based on the first converted partial image; the output image generation unit, wherein the output partial image is an image based on the second converted partial image;
A system equipped with an image acquisition unit that acquires input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification unit that identifies ,
a process selection unit that selects a process to be performed on first partial image data indicating the first input partial image;
a first conversion unit that executes a first style conversion process using a machine learning model on the first partial image data to generate first converted data indicating the first converted partial image;
a color conversion unit that performs a color conversion process on the first partial image data to convert at least part of the color of the first input partial image without using a machine learning model;
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion unit that generates second converted data indicating the second converted partial image;
an output image generation unit that uses the first converted data and the second converted data to generate output image data representing an output image based on the input image, the output image being the first input partial image; a first output partial image corresponding to the first output partial image and a second output partial image corresponding to the second input partial image, the first output partial image being an image based on the first converted partial image; the output image generation unit, wherein the output partial image is an image based on the second converted partial image;
Equipped with
When the first style conversion process is selected by the process selection unit, the color conversion unit does not execute the color conversion process, and the first conversion unit executes the first style conversion process,
When the color conversion process is selected by the process selection unit, the color conversion unit executes the color conversion process, and the first conversion unit does not execute the first style conversion process. an image acquisition unit that acquires input image data indicating the input image;
Using the input image data, a first input partial image that is a part of the input image, and a second input partial image that is a part of the input image and located at a different position from the first input partial image. a partial image identification unit that identifies ,
Based on the first input by the user, first input information for the first style conversion process using the machine learning model is acquired, and based on the second input by the user, the second style conversion process using the machine learning model is acquired. an information acquisition unit that acquires second input information for the second style conversion process that is a style conversion process and is different from the first style conversion process;
Execute the first style conversion process on the first partial image data indicating the first input partial image using the first input information to generate first converted data indicating the first converted partial image. a first conversion unit that generates;
Execute the second style conversion process on the second partial image data indicating the second input partial image using the second input information to generate second converted data indicating the second converted partial image. a second conversion unit that generates;
an output image generation unit that uses the first converted data and the second converted data to generate output image data representing an output image based on the input image, the output image being the first input partial image; a first output partial image corresponding to the first output partial image and a second output partial image corresponding to the second input partial image, the first output partial image being an image based on the first converted partial image; the output image generation unit, wherein the output partial image is an image based on the second converted partial image;
A system equipped with first partial image data indicating a first input partial image that is a part of the input image; and second partial image data indicating a second input partial image that is a part of the input image and is located at a different position from the first input partial image. a partial image acquisition function that acquires two partial image data;
a first conversion function that executes a first style conversion process using a machine learning model on the first partial image data to generate first converted data indicating the first converted partial image;
The second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, is performed on the second partial image data to obtain a second converted part. a second conversion function that generates second converted data representing the image;
a first function of generating intermediate image data representing an intermediate image including the first converted partial image and the second converted partial image using the first converted data and the second converted data; , a second function for performing specific post-processing on the intermediate image data to generate output image data representing an output image based on the input image, the output image generation function comprising: includes a first output partial image corresponding to the first input partial image and a second output partial image corresponding to the second input partial image, the first output partial image being based on the first transformed partial image. the output image generation function, wherein the second output partial image is an image based on the second converted partial image;
to be realized by a computer,
The specific post-processing includes the third style conversion process that is a third style conversion process using a machine learning model and is different from the first style conversion process and the second style conversion process,
The third style conversion process is a computer program that is executed using input image data indicating the input image as style image data . first partial image data indicating a first input partial image that is a part of the input image; and second partial image data indicating a second input partial image that is a part of the input image and is located at a different position from the first input partial image. a partial image acquisition function that acquires two partial image data;
a first conversion function that executes a first style conversion process using a machine learning model on the first partial image data to generate first converted data indicating the first converted partial image;
The second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, is performed on the second partial image data to obtain a second converted part. a second conversion function that generates second converted data representing the image;
a first function of generating intermediate image data representing an intermediate image including the first converted partial image and the second converted partial image using the first converted data and the second converted data; , a second function for performing specific post-processing on the intermediate image data to generate output image data representing an output image based on the input image, the output image generation function comprising: includes a first output partial image corresponding to the first input partial image and a second output partial image corresponding to the second input partial image, the first output partial image being based on the first transformed partial image. the output image generation function, wherein the second output partial image is an image based on the second converted partial image;
Let the computer realize it,
The specific post-processing includes the fourth style conversion process that is a fourth style conversion process using a machine learning model and is different from the first style conversion process and the second style conversion process,
The input image includes an image showing a person's face,
The computer program is a computer program , wherein the fourth style conversion process is a process of changing the facial expression of the person . first partial image data indicating a first input partial image that is a part of the input image; and second partial image data indicating a second input partial image that is a part of the input image and is located at a different position from the first input partial image. a partial image acquisition function that acquires two partial image data;
A first conversion function that executes a first style conversion process using a machine learning model on the first partial image data to generate first converted data indicating a first converted partial image, The first style conversion process is performed using a first parameter that specifies the degree of difference between the first input partial image and the first converted partial image to be generated. conversion function and
performing the second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, on second partial image data indicating the second input partial image; , a second conversion function that generates second converted data indicating a second converted partial image, wherein the second style conversion process includes the second input partial image and the second converted portion to be generated. the second transformation function is performed using a second parameter specifying the degree of difference between the images, the second parameter being adjusted independently of the first parameter;
Let the computer realize it,
The first transformed data and the second transformed data are used to generate an output image representing an output image based on the input image, the output image corresponding to the first input partial image. a first output partial image corresponding to the second input partial image, the first output partial image is an image based on the first converted partial image, and the second output partial image is an image based on the first converted partial image; The computer program product, wherein the image is an image based on the second converted partial image. first partial image data indicating a first input partial image that is a part of the input image; and second partial image data indicating a second input partial image that is a part of the input image and is located at a different position from the first input partial image. a partial image acquisition function that acquires two partial image data;
a process selection function that selects a process to be performed on the first partial image data;
a first conversion function that executes a first style conversion process using a machine learning model on the first partial image data to generate first converted data indicating the first converted partial image;
a color conversion function that performs a color conversion process on the first partial image data to convert at least part of the color of the first input partial image without using a machine learning model;
The second style conversion process, which is a second style conversion process using a machine learning model and which is different from the first style conversion process, is performed on the second partial image data to obtain a second converted part. a second conversion function that generates second converted data representing the image;
Let the computer realize it,
The first transformed data and the second transformed data are used to generate an output image representing an output image based on the input image, the output image corresponding to the first input partial image. a first output partial image corresponding to the second input partial image, the first output partial image is an image based on the first converted partial image, and the second output partial image is an image based on the first converted partial image; The image is an image based on the second converted partial image,
When the first style conversion process is selected by the process selection function, the color conversion function does not execute the color conversion process, and the first conversion function executes the first style conversion process,
When the color conversion process is selected by the process selection function, the color conversion function executes the color conversion process, and the first conversion function does not execute the first style conversion process. first partial image data representing a first input partial image that is part of the input image; and second partial image data representing a second input partial image that is part of the input image and located at a different position from the first input partial image. a partial image acquisition function that acquires two partial image data;
Based on the first input by the user, first input information for the first style conversion process using the machine learning model is acquired, and based on the second input by the user, the second style conversion process using the machine learning model is acquired. an information acquisition function that acquires second input information for the second style conversion process that is a style conversion process and is different from the first style conversion process;
a first conversion function that executes the first style conversion process on the first partial image data using the first input information to generate first converted data indicating the first converted partial image; ,
a second conversion function that executes the second style conversion process on the second partial image data using the second input information to generate second converted data indicating the second converted partial image; and,
Let the computer realize it,
The first transformed data and the second transformed data are used to generate an output image representing an output image based on the input image, the output image corresponding to the first input partial image. a first output partial image corresponding to the second input partial image, the first output partial image is an image based on the first converted partial image, and the second output partial image is an image based on the first converted partial image; The computer program product, wherein the image is an image based on the second converted partial image.

Images