JP2021168078A - Image generation system and machine learning method for image generation system - Google Patents

Abstract

To provide an image generation system that can produce original designs.SOLUTION: An image generation system 1 includes: an input section 2 in which an input image of a natural object is input; a recording section 3 which is designed to record a learning model that has been learned to output an image that is a crossover of the input image and an artificial object when the input image of the natural object is input; and an output section 4 which outputs the output image that is the crossover of the input image and the artificial object based on the input image input to the input section 2 and the learning model recorded in the recording section 3.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image generation system in product design.

Various artificial objects are designed with natural objects as motifs so that they are not directly related to them. Traditionally, these tasks have been done by product designers based on their own experience.

Japanese Patent Application Laid-Open No. 2001-0767177

However, the designs created by product designers based on their own experience cannot escape from their own past ideas, and it was difficult to come up with unusual designs.

Therefore, the present invention provides an image generation system capable of conceiving an unusual design, a method of generating a design support learning model used in the image generation system, and a method of manufacturing an image providing device equipped with this design support learning model. ..

According to one aspect of the invention
Input section for inputting images of natural objects and
A recording unit in which a learning model learned to output an image obtained by multiplying the input image and an artificial object when an input image of a natural object is input is recorded.
An image generation system including an output unit that outputs an output image obtained by multiplying the input image and the artificial object based on the input image input to the input unit and the learning model recorded in the recording unit. Is provided.

In the above image generation system
The learning model is a Cycle-Consistent Generative Adversarial Networks (Cycle-Consistent Generative Adversarial Networks) that outputs an output image obtained by multiplying the input image and the artificial object when the input image is input.
The image generation system includes a plurality of trained models in which a plurality of different numerical values are set as a coefficient λ1 indicating the degree to which the consistency of the image is lost when the input image is converted to the output image in the CycleGAN. It may have a recorded recording unit.

In the above image generation system
When the input image is input, the output unit is configured to output each of the output images output from each of the plurality of trained models having different coefficients λ1 in a comparable manner. May be good.

According to one aspect of the invention
It is a method of generating a design support learning model that generates an output image of an artificial object when an input image of a natural object is given.
The learning model is provided by a method for generating a design support learning model, which uses Cycle-Consistent Generative Adversarial Networks (Cycle GAN) that outputs an output image obtained by multiplying the input image and an artificial object when the input image is input. Will be done.

In the above design support learning model generation method,
In the CycleGAN, the coefficient λ1 indicating the degree to which the consistency of the image is lost when the input image is converted to the output image is lost when the output image is converted to the input image. It may be set smaller than the coefficient λ2 indicating the degree of the image.

According to one aspect of the invention
A method of manufacturing an image providing device including a chip that executes a computer-readable instruction and a memory that records the computer-readable instruction processed by the chip.
A method for manufacturing an image providing device is provided, in which the learning model obtained by the method for generating the design support learning model according to claim 1 is written in the memory.

According to the present invention, there is provided an image generation system capable of conceiving an unusual design, a method of generating a design support learning model used in the image generation system, and a method of manufacturing an image providing device equipped with the design support learning model. Will be done.

It is a block diagram of the image generation system which concerns on embodiment of this invention. It is a figure which shows the model of CycleGAN. It is a figure explaining the mapping of the 1st generator and the 2nd generator in CycleGAN. It is a figure explaining the consistency loss. It is a figure which shows the input image IMG_IN and the output image IMG_OUT of an image generation system. It is a figure which shows the result of the user evaluation. 7 (a) to 7 (g) are diagrams for explaining the change in the consistency loss when the coefficient λ1 is changed. It is a figure which shows an example of the display of a display.

Hereinafter, the present invention will be described with reference to the drawings based on the preferred embodiments. The same or equivalent components, members, and processes shown in the drawings shall be designated by the same reference numerals, and redundant description will be omitted as appropriate. Further, the embodiment is not limited to the invention, but is an example, and all the features and combinations thereof described in the embodiment are not necessarily essential to the invention.

<Image generation system>
The image generation system 1 according to the embodiment of the present invention is a system that generates an output image of an artificial object when an input image of a natural object is given. This system 1 is used when a user who is considering the design of an artificial object such as a product designer thinks about the design of an artificial object with a natural object as a motif. When the input image of a natural object is given, the system 1 generates an output image of an artificial object design with the natural object as a motif. More specifically, when the input image of a natural object is given, the system 1 generates an image in which the input image and the artificial object to be obtained are multiplied.

FIG. 1 is a block diagram of the image generation system 1 according to the embodiment. The image generation system 1 includes an input unit 2, a recording unit 3, and an output unit 4.
The input image IMG_IN of a natural object is input to the input unit 2 from the user's terminal 5. The input unit 2 may be directly connected to the user's terminal 5 or may be connected via a network. When the user's terminal 5 is directly connected to the image generation system 1, the image generation system 1 can be realized by a single personal computer or the like. In this case, the user's terminal 5 is an input device such as a keyboard or a mouse. When the user's terminal 5 is connected to the image generation system 1 via a network, the image generation system 1 is realized by a server operating on the network. In this case, the user's terminal 5 is an information processing device such as a personal computer, a tablet terminal, or a mobile phone. The input unit 2 transmits the acquired input image to the output unit 4.

The recording unit 3 records a learning model learned to output an output image IMG_OUT obtained by multiplying an input image IMG_IN and an artificial object when an input image of a natural object is input. As a learning model, an algorithm called GAN (Generative Adversarial Networks) can be used. Cycle-Consistent GAN (Cycle-Consistent GAN) and DCGAN (Deep Convolutional Generative GAN) can be used, and it is particularly preferable to use Cycle-GAN. The details of this learning model will be explained later.

The output unit 4 (image generator) outputs an output image IMG_OUT obtained by multiplying the input image IMG_IN and an artificial object based on the input image IMG_IN input to the input unit 2 and the learning model recorded in the recording unit 3. .. The output unit 4 is a computer having a processor 11, a memory 12, and the like in terms of hardware, and various arithmetic processes can be executed by executing a software program. The output unit 4 may be directly connected to the user's terminal 5 or may be connected via a network.
When the image generation system 1 is started, the processor 11 of the output unit 4 reads out the learned model recorded in the recording unit 3 and expands it on the memory 12. Further, when the input image IMG_IN is input from the input unit 2, the processor 11 inputs the input image IMG_IN to the trained model and generates the output image IMG_OUT. The output unit 4 outputs the generated output image IMG_OUT to the user's terminal 5.

<How to generate a learning model>
Next, a learning method of CycleGAN, which is a learned model recorded in the recording unit 3, will be described. FIG. 2 is a diagram showing a learning model of CycleGAN. The CycleGAN includes two Generators 21 and 22, and two Discliminators 23 and 24.

The set X of images of natural objects and its elements are expressed as x, and the set Y of images of artificial objects and its elements are expressed as y. The first generator 21 converts an image of a natural object (an image of a cat's eye) x into a fake image of an artificial object (an image of a headlamp) y'.

On the contrary, the second generator 22 converts an artificial object image (headlamp image) y into a natural object fake image (eye image) x.

FIG. 3 is a diagram illustrating mapping of the first generator 21 and the second generator 22 in the CycleGAN 20. The conversion by the first generator 21 is referred to as a map G.
y'= G (x)
Similarly, the conversion by the first generator 21 is referred to as a map F.
x'= F (y)

Return to FIG. The first classifier 23 determines whether or not the fake image y'= G (x) generated by the first generator 21 belongs to the artificial object Y (that is, whether or not it is an image of a real headlamp). do. The second classifier 24 determines whether the fake image x'= F (y) generated by the second generator 22 belongs to the natural object X (that is, whether it is a real eye image).

In the machine learning of CycleGAN20, a plurality of real images x of natural objects and a plurality of real images y of artificial objects are used as training data. A set of a plurality of images x is, for example, a set of images of an actual animal eye, and a set of a plurality of images y is, for example, a set of images of an actual headlamp. Note that CycleGAN is unsupervised learning and does not require pairing (data labeling) of real images x and y with different domains.

In machine learning in CycleGAN 20, the parameters of the first generator 21 are optimized to generate a fake image that is close enough to deceive the first classifier 23. Similarly, the parameters of the second generator 22 are optimized to generate a fake image that is close enough to deceive the second classifier 24. The machine learning may be executed in the output unit 4, but may be executed in another computer.

On the contrary, the parameters of the first classifier 23 are optimized so that the fake image y'generated by the first generator 21 can be distinguished from the real image y. Similarly, the parameters of the second classifier 24 are optimized so that the fake image x'generated by the second generator 22 can be distinguished from the real image x.

Further, the fake image y'= G (x) obtained by the first generator 21 transforming a certain image x is inversely transformed by the second generator 22, and the image x ^ = F (y') is the original. It is a constraint condition to return to the image x of. Similarly, the image y ^ = G (x') obtained by inversely transforming the fake image x'= F (y) obtained by the second generator 22 by converting a certain image y by the first generator 21 is the original. It is a constraint condition to return to the image data y of. These are called cycle consistency conditions. Note that x ^ and y ^ are images obtained when the switch of FIG. 2 is connected in a state opposite to that shown in the drawing.

The first generator 21 and the first classifier 23 are in a hostile relationship, and the second generator 22 and the second classifier 24 are also in a hostile relationship. By competing with each other, learning of the CycleGAN 20 is promoted. ..

_{The loss function L GAN} used in the machine learning process of Cycle GAN can be expanded into the first term, the second term, and the third term as represented by the equation (1).

X: Set of variables x (that is, images) of natural objects x ∈ X
Y: A set of variables y (that is, images) of an artificial object y ∈ Y
G: mapping F to y from x According to a first generator 21: mapping D _Y to x from y by the second generator 22: first determination by the discriminator 23 Results D _X: a determination result by the second classifiers 24

This loss function, D _Y to maximize paragraph, G is optimized to minimize the paragraph. Further, D _X to maximize paragraph, F is being optimized to minimize the second term. G and F are also optimized to minimize the third term.

The first and second terms are referred to as Adversarial Loss and are expressed by the following equation (2).

_DY (y): Judgment result of the first classifier 23 for the real image y _DY (G (x)): Judgment result of the first classifier 23 for the fake image G (x) generated by the first generator 21 D _X (x): real for the image x of the second discriminator 24 determination result D _X (F (y)): the determination result of the second classifier 24 to the second fake image F of generator 22 has generated (y) E [] represents the mini-batch average.

The third term on the right-hand side of equation (1) is Consistency Loss, which is introduced to meet the above-mentioned consistency requirements. Specifically, this loss term is expressed by the equation (3), and the term of the consistency loss of the natural object weighted by the first coefficient λ1 and the artificial object weighted by the second coefficient λ2 different from the first coefficient λ1. Including the term Consistency Loss.

FIG. 4 is a diagram illustrating a consistency loss. The loss term of the natural object X is such that the image x ^ = F (G (x)) when the original image x is mapped by the generator G and inversely mapped by the generator F is closer to the original image x. It is defined using the distance (norm) of two points x and x ^. The same applies to the loss term of the artificial object Y, and the image y ^ = G (F (y)) when the original image y is mapped by the generator F and inversely mapped by the generator G is the original image y. Is defined using the distance (norm) of the two points y and y ^ so that

The term of consistency loss is a loss for suppressing excessive conversion and preventing image corruption. λ1 is a coefficient indicating the degree to which image consistency is lost when converting an input image to an output image. λ2 is a coefficient indicating the degree to which image consistency is lost when converting an output image to an input image. The term of consistency loss in conventional CycleGAN is defined with the same weighting for two domains, so it was common for λ1 = λ2. On the other hand, in this embodiment, λ1 <λ2. This aims to generate a dramatically changed output image from the input image.

Based on the above concept, a learning model based on CycleGAN is generated using an information processing device for machine learning. Specifically, this machine learning is described as a software program and executed by an information processing device equipped with a CPU and a memory. This machine learning is performed by preparing a model equipped with a generator and a discriminator, preparing training data of natural objects and man-made objects, setting appropriate λ1 and λ2, and executing them. The specific processing of machine learning is the same as that of the known technique, and is executed with a specified number of epochs and batch size. Specifically, for the combination of training data, the parameters of the first generator 21, the second generator 22, the first classifier 23, and the second classifier 24 are optimized while calculating the loss function.

The above is the explanation of machine learning of CycleGAN. Of the learning models obtained by machine learning, the part of the first generator 21 is mainly mounted on the output unit 4 of FIG.
An image providing device is manufactured by writing a learning model obtained by the above-described design support learning model generation method to a memory and mounting the memory and a chip that executes a computer-readable instruction recorded in the memory. be able to.

According to this embodiment
Input unit 2 for inputting an input image of a natural object and
A recording unit 3 in which a learning model trained to output an image obtained by multiplying an input image and an artificial object when an input image of a natural object is input is recorded.
An image generation system 1 is provided, which includes an output unit 4 that outputs an image obtained by multiplying an input image and an artificial object based on an input image input to the input unit 2 and a learning model recorded in the recording unit 3. NS.

Since the image generation system 1 of the present embodiment outputs an image obtained by multiplying an input image and an artificial object based on an artificially learned learning model, it is a novel idea that is not bound by the product designer's own past ideas. You can come up with a design.

Further, according to the present embodiment,
It is a method of generating a design support learning model that generates an output image of an artificial object when an input image of a natural object is given.
As the learning model, a method for generating a design support learning model is provided, which uses Cycle-Consistent Generative Adversarial Networks (Cycle GAN) that outputs an output image obtained by multiplying an input image and an artificial object when an input image is input.

According to the design support learning model generation method of the present embodiment, since CycleGAN is used, an image obtained by multiplying the input image and the artificial object is output based on the artificially learned learning model. By using this learning model, it is possible to realize an image providing device that can conceive a strange design that is not bound by the product designer's own past ideas.

It is also known that CycleGAN produces images that are completely different from those of two domains that seem to work, such as the conversion of dogs and cats. Therefore, it is recognized by those skilled in the art that when the combination of animal eyes and headlamps is selected as the combination of natural objects and artificial objects, an intermediate image between the animal eyes and the headlamps can be successfully generated. should not be done.

Further, according to the present embodiment,
A method for manufacturing an image providing device (output unit 4) having a chip for executing a computer-readable instruction and a memory for recording a computer-readable instruction processed by the chip.
Provided is a method for manufacturing an image providing device, which writes a learning model obtained by the above-described design support learning model generation method in a memory.

According to the manufacturing method of the image providing device of the present embodiment, the learning model using CycleGAN, which outputs an image obtained by multiplying the input image and the artificial object, is written in the memory, so that the product designer's own past It is possible to realize an image providing device that can conceive a strange design that is not bound by the idea of.

In the above-described embodiment, setting λ1 to a value smaller than λ2 has been described. This will be described in detail below.
The set of values of λ1 and λ2 set in the training model is called a parameter set. FIG. 5 is a verification of how the output images are different when the parameter sets are different. FIG. 5 compares and shows what kind of output image is obtained when the same input image is given to each of the six trained models in which different parameter sets are set. Here, a trained model was generated in which λ2 was fixed at 10 and λ1 was set at 30.0, 10.0, 0.2, 0.1, 0.05, and 0.01. When learning the learning model, an image of a cat's eyes was given as an input image, and an image of a vehicle headlight was given as an output image.

The conditions of the actual learning are shown below.
・ Training data Natural object image (cat's eye) 14456 sheets Artificial object image (head lamp) 14606 sheets ・ Test data Natural object image (cat's eye) 1000 sheets Artificial object image (head lamp) 1000 sheets ・ Epoch number 100
・ Batch size 8
・ Learning rate 0.0002

When the three input images IMG_IN shown at the left end of FIG. 5 are input to the image generation system 1 using the generated trained model, the output image IMG_OUT is generated as shown in FIG. As shown in FIG. 5, it can be seen that the larger the λ1 is, the closer the output image is to the cat's eyes, and the smaller the λ1 is, the closer the output image is to the vehicle headlight. As a result, it can be seen that as λ1 becomes smaller than λ2, a dramatically changed image of the vehicle headlight can be obtained from the input image of the cat's eyes.

By the way, among the multiple trained models with different parameter sets in this way, we investigated which trained model output the image is useful for the product designer. Forty-five input images were input to each of the six trained models generated as described above, and 45 sets of image groups were generated. Each image group includes an input image and six output images, as shown in FIG. The subjects were asked to select one of the most preferable images for design reference for each of the 45 sets of image groups. In FIG. 5, since three input images were input, a total of 18 image groups of 3 images vertically and 6 images horizontally were input, but the images shown to the subject were a total of 45 images vertically and 6 images horizontally. It is a group of 270 images.

FIG. 6 is a diagram showing the evaluation results of the subjects. For one input image, we were asked to select which of the six output images generated by the six trained models was better, or the input image was better. Fifteen subjects voted for each of the 45 sets of images, and the results shown in FIG. 6 were obtained. The total number of votes obtained is 15 x 45 = 675.

From this result, it can be seen that λ1 = 0.2 (λ2 = 10) has the highest number of votes and is effective as a design motif. This is a higher vote than the original cat's eye image, and a higher vote than the case of λ1 = λ2 = 10. That is, it is suggested that by setting λ1 <λ2 to generate an image closer to the headlamp, information useful for the design can be provided to the designer.

The original cat's eye image still has a high vote. This may imply that some designers have a rejection of AI. Therefore, the advantage of λ1 <λ2 can be more pronounced when targeting only designers who are not rejected or prejudiced against AI and who can actively accept computer-based design support. ..

By setting λ1 <λ2 in this way, an image more useful for design is provided as compared with the case of λ1 = λ2. This should not be found to be easily accomplished by those skilled in the art. 7 (a) to 7 (g) are diagrams for explaining the change in consistency loss when λ1 is changed. In each figure, the left image is the input x to the first generator 21, the center image is the output y'= G (x) of the first generator 21, and the right image is the center image by the second generator 22. The inversely transformed image x ^ = F (y'). In the case of λ1 = λ2 = 10, the left image x and the right image x ^ are in good agreement, and it can be seen that the consistency loss is small. On the other hand, when λ1 ≦ 0.2 when λ2 is kept constant at 10.0, the divergence between the left image x and the right image x ^ is large, that is, the consistency loss is large. In general CycleGAN machine learning, the objective function is designed so that the consistency loss is reduced as much as possible, but in this embodiment, by allowing the consistency loss to remain, an image that is more helpful to the designer can be obtained. It is provided, and it can be said that these are fundamentally different positions.

In the image generation system 1 described above, a plurality of trained models trained by the recording unit 3 with different parameter sets are recorded, and the output unit 4 has a plurality of trained models having different λ1 when the input image is input. Each output image output from each of the trained models may be configured to be output in a comparable manner.

For example, as shown in FIG. 5, the output unit 4 outputs the input image and the output images generated from a plurality of trained models trained in different λ1s in a horizontal row so that each can be compared. You may. Alternatively, as shown in FIG. 8, the input image is displayed in the upper row as a motif image, and the output images generated from a plurality of trained models trained in different λ1s are arranged in a horizontal row in the lower row. May be output in a comparable manner. By displaying a plurality of output images at the same time in this way, the product designer can obtain more hints, ideas, and ideas.

The present invention has been described above based on the embodiments. This embodiment is an example, and it will be understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. .. Hereinafter, such a modification will be described.

In the embodiment, the image of the cat's eyes is used as a motif, but the motif is not limited to that, and the eyes of insects such as mammals such as humans and dogs, birds, reptiles, amphibians and fish, grasshoppers, crickets, and dragonflies may be used as motifs. In addition, flowers, seeds, leaves, etc. of plants may be used as motifs. Motif is a central subject that motivates the creation of an artificial object that you want to design. As the input image, not only the front view of the animal's eyes but also the front view and the side view of the whole body of the animal may be used as the input image.

In this embodiment, the design motif is the animal eye. When looking at a car from the front, the car is often likened to a face, with the left and right headlamps associated with the animal's eyes. Therefore, the animal eye is particularly useful as a headlamp motif.

An artificial object is an industrial product that you want to design. In particular, it is an industrial product in which an eye-catching design aspect is important due to the nature of the product. The man-made object is preferably a vehicle or a vehicle headlight. As an output image, it is preferable to output an image of a front view of the design surface of the artificial object. The output image is particularly preferably an image of the entire front view of the vehicle headlight mounted on the left side of the vehicle or the vehicle headlight mounted on the right side of the vehicle.

When learning the learning model, an input image in which different types of natural objects are mixed may be input. Further, an image of another kind of natural object may be given as an input image to the image generation system 1 using the trained model in which a certain kind of natural object is generated as an input image.

In the above-described embodiment, a trained model is generated for a set of N = 6 parameters, but the number of parameter sets and a specific coefficient value are not limited.

A plurality of trained models corresponding to a plurality of parameter sets may be mounted on the recording unit 3 of the image generation system 1, or λ1 and λ2 of the optimum combination in a specific motif and a specific product combination may be mounted. One trained model trained using may be implemented.

Further, in the embodiment, the design of the headlamp is taken as an example, but the present invention is not limited to this, and the design can be applied to the design of vehicle lighting equipment of other parts such as tail lamps and side markers. Furthermore, the present invention is not limited to vehicle lamps, and can be applied to support various industrial designs and product designs.

Although the present invention has been described using specific terms and phrases based on the embodiments, the embodiments show only one aspect of the principles and applications of the present invention, and the embodiments are defined in the claims. Many modifications and arrangement changes are permitted without departing from the ideas of the present invention.

1 Image generation system 2 Input unit 3 Recording unit 4 Output unit (image generator)
5 User's terminal 11 Processor 12 Memory 20 CycleGAN
21 1st generator 22 2nd generator 23 1st classifier 24 2nd classifier G 1st generator F 2nd generator IMG_IN Input image IMG_OUT Output image L _GAN loss function λ1 1st coefficient λ2 2nd coefficient

Claims (6)

Input section for inputting images of natural objects and
A recording unit in which a learning model learned to output an image obtained by multiplying the input image and an artificial object when an input image of a natural object is input is recorded.
An image generation system including an output unit that outputs an output image obtained by multiplying the input image and the artificial object based on the input image input to the input unit and the learning model recorded in the recording unit. .. The learning model is a Cycle-Consistent Generative Adversarial Networks (Cycle-Consistent Generative Adversarial Networks) that outputs an output image obtained by multiplying the input image and the artificial object when the input image is input.
The image generation system includes a plurality of trained models in which a plurality of different numerical values are set as a coefficient λ1 indicating the degree to which the consistency of the image is lost when the input image is converted to the output image in the CycleGAN. The image generation system according to claim 1, further comprising a recorded recording unit. According to claim 1, when the input image is input, the output unit outputs each of the output images output from each of the plurality of trained models having different coefficients λ1 in a comparable manner. The image generation system described. It is a method of generating a design support learning model that generates an output image of an artificial object when an input image of a natural object is given.
The learning model is a method of generating a design support learning model using Cycle-Consistent Generative Adversarial Networks (Cycle-Consistent Generative Adversarial Networks), which outputs an output image obtained by multiplying the input image and an artificial object when the input image is input. In the CycleGAN, the coefficient λ1 indicating the degree to which the consistency of the image is lost when the input image is converted to the output image is lost when the output image is converted to the input image. The method for generating a design support learning model according to claim 4, which is set to be smaller than the coefficient λ2 indicating the degree of the image. A method of manufacturing an image processing apparatus including a chip that executes a computer-readable instruction and a memory that records the computer-readable instruction processed by the chip.
A method for manufacturing an image processing apparatus, wherein the learning model obtained by the method for generating a design support learning model according to claim 4 is written in the memory.

Images