CN108564126B - Specific scene generation method fusing semantic control - Google Patents

Abstract

The invention provides a specific scene generation method integrating semantic control, which comprises the steps of selecting a plurality of object pictures and a plurality of different specific scene pictures containing the object; manufacturing different attribute labels according to the characteristics of a specific scene in a specific scene picture, and obtaining a training sample after cutting the specific scene picture; constructing a condition generating type countermeasure network consisting of a discriminator and a generator; inputting the item graph and the label into a generator as input, and generating a specific scene graph described by the label; the method comprises the steps that a specific scene graph of an article is used as a target scene graph, the specific scene graph, the target scene graph, the article graph and the label which are described by a label generated by a generator are input into a discriminator together, and the discriminator performs model training through a conditional countermeasure network; inputting the similar object images to be processed and the scene to be obtained into the trained model in a label mode to obtain the corresponding scene image.

Description

A Scenario-Specific Generation Method with Semantic Control

technical field

The invention belongs to the field of machine learning algorithms, and specifically relates to a method for generating a specific scene integrating semantic control.

Background technique

The specific scene generation with semantic control refers to the use of semantic control to let the computer generate the scene described by the language. It has always been the pursuit of human beings to be able to truly depict the world. The birth of painting stems from the need of human beings to depict the world, and the pursuit of perfection has made art. The invention of the camera made it easy for humans to record the world. After the advent of computers, humans began to let the computer describe the real world by themselves, and many generative algorithms were born. The traditional generation algorithms include gradient direction histogram, scale-invariant feature transformation, etc. These algorithms use the method of manually extracting features and combining shallow models to achieve target generation. Its solution basically follows four steps: image preprocessing → manual feature extraction → model building (classifier/regressor) → output. The idea of deep learning algorithm to solve computer vision is end-to-end (End to End), that is, from input directly to output, using neural network to automatically learn features in the middle, avoiding the tedious operation of manual feature extraction.

Deep learning is an important branch of machine learning, which has received extensive attention due to its major breakthroughs in many fields in recent years. Generative Adversarial Networks (GAN) is a generative deep learning model proposed by Goodfellow et al. in 2014. Once this model is proposed, it has become one of the hot research directions in the field of computer vision research. Due to the excellent generation ability of GAN, GAN has made remarkable achievements in the field of sample generation. Secondly, GAN has also become a huge application in the fields of image restoration and restoration, image style transfer, mutual generation of text and images, and high-quality image generation. issue of value. At the same time, many leading companies in the industry have also joined the wave of GAN development. Such as Facebook, Google, Apple and other companies. Based on the above research, GAN provides the possibility to realize fusion semantic control to generate specific scenes. However, there is currently no model that directly implements the generation of different specific scenarios through semantic control.

In order to solve the above problems, people have been looking for an ideal technical solution.

SUMMARY OF THE INVENTION

The purpose of the present invention is to aim at the deficiencies of the prior art, so as to provide a specific scene generation method integrating semantic control.

In order to achieve the above object, the technical solution adopted in the present invention is: a method for generating a specific scene integrating semantic control, comprising the following steps:

Step 1. Select several item images and several different specific scene images containing the item;

Step 2. Create different attribute labels according to the characteristics of the specific scene in the specific scene picture, and after the specific scene picture is cropped, a training sample is obtained, and the training sample includes the item map, the specific scene map and description corresponding to the item containing the item. the label of the scene;

Step 3. Construct a conditional generative adversarial network composed of a discriminator and a generator;

Step 4. The item graph and the label are used as input into the generator to generate the specific scene graph described by the label;

Step 5. The specific scene graph containing the item is used as the target scene graph, and the specific scene graph, the target scene graph, the item graph and the label described by the label generated by the generator are input into the discriminator, and the discriminator is modeled through a conditional confrontation network. train;

Step 6: Input the image of the same item to be processed and the scene to be obtained into the trained model in the form of labels to obtain the corresponding scene image.

Based on the above, the tags are semantic tags in binary form.

Based on the above, in step 1, the item image is a close-up image of an item crawled from a shopping website.

其中，y为目标场景图像域，x为原始输入图像，l为目标场景图像域标签，

为标签所描述的特定场景图；Based on the above, in step 3, the generative adversarial network is a GAN model, and the generator of the generative adversarial network is expressed as

where y is the target scene image domain, x is the original input image, l is the target scene image domain label,

the specific scene graph described by the label;

Use the cost function of the conditional GAN as the adversarial loss of the model, where the cost function is

Among them, D is the discriminator and G is the generator.

The present invention has outstanding substantive features and remarkable progress relative to the prior art, specifically:

The invention conducts model training by constructing a conditional generative confrontation network, and replaces repetitive labor by artificial intelligence technology, which can greatly improve the work efficiency of human beings, and some simple scenes can be directly generated by the system without wasting manpower for shooting and production. The specified scene is generated through semantic control. According to different situations, it is only necessary to provide some training samples required for the scene, and make domain labels for the training samples. After training, the images of the specified scene can be generated. The method of the present invention has broad application prospects, especially the images showing the details of commodities on the shopping website can be generated by the method, thereby saving labor and resources.

Description of drawings

FIG. 1 is a schematic flow chart of the algorithm of the present invention.

FIG. 2 is a schematic design diagram of a method for generating a specific scene integrating semantic control according to the present invention.

Detailed ways

The technical solutions of the present invention will be further described in detail below through specific embodiments.

As shown in Figure 1 and Figure 2, a method for generating a specific scene integrating semantic control includes the following steps:

Step 1. Crawl several item images and several different specific scene images containing the item from the shopping website;

Step 2. Make different attribute labels according to the characteristics of the specific scene in the specific scene picture, and the label is a semantic label in binary form; after the specific scene picture is cropped, a training sample is obtained, and the training sample includes an item map and corresponds to the item map. a specific scene graph containing the item and a label describing the scene;

Step 3. Construct a conditional generative adversarial network composed of a discriminator and a generator;

Step 4. The item graph and the label are used as input into the generator to generate the specific scene graph described by the label;

Step 5. The specific scene graph containing the item is used as the target scene graph, and the specific scene graph, the target scene graph, the item graph and the label described by the label generated by the generator are input into the discriminator, and the discriminator is modeled through a conditional confrontation network. train;

Step 6: Input the image of the same item to be processed and the scene to be obtained into the trained model in the form of labels to obtain the corresponding scene image.

其中，y为目标场景图像域，x为原始输入图像，l为目标场景图像域标签，

为标签所描述的特定场景图；Specifically, in step 3, the generative adversarial network is a GAN model, and the generator of the generative adversarial network is expressed as

where y is the target scene image domain, x is the original input image, l is the target scene image domain label,

the specific scene graph described by the label;

In the method of the present invention, each input item image corresponds to a paired target scene image domain y and label l, so that G can accurately learn to generate a specific scene. The discriminator learns to classify the real image and the generated image. The generator needs to learn to deceive the discriminator, and the discriminator generates a probability distribution on the input item image and label, which can specify the label and realize the generation of semantic control generator. The goal of the generator is to convert the original item images into real scene images described by labels, so the dataset of training samples is given as a set of corresponding images (x, y, l), where x is the input item image, y is the corresponding target scene image, and l is the target scene image domain label.

As the adversarial loss of the algorithm model, the cost function of the conditional GAN is used, which is a minimax two-player zero-sum game:

Among them, D is the discriminator and G is the generator.

The first term of the function shows that when a real scene image is input, the discriminator makes the objective function as large as possible and judges that it is a real image. The second term of the function means that when the generated image is input, G(x, y, l) is as small as possible, therefore, the value of the loss function is relatively large, while the generator fools the discriminator and mistakenly believes that the input is a real image. The discriminator tries to identify it as a fake image, and the two models of the function play until a Nash equilibrium is reached, allowing the generator to learn the semantic features of the label and map it to the item image.

Generative adversarial network using GAN model, the generator inputs the original image of the target domain scene, the target domain image and the label as condition variables, while generating a fake specific scene, the target domain image and the target domain label are copied as input and compared with the input Image stitching. The generator then tries to reconstruct a new scene from the input image and given the original domain labels, and tries to generate a specific scene that is indistinguishable from the real scene, making it difficult for the discriminator to distinguish. In the process of confrontation between the two, the scene generated by the generator becomes more and more realistic, and the discriminator becomes more and more difficult to distinguish the real scene image from the fake scene image, so as to achieve the purpose of training.

The overall structure of the invention is simple, the design is reasonable, and the conditional GAN is used as the model frame. In order to realize the semantic control function, the algorithm model can accept training data from multiple domains, and only use one generator to learn the mapping between all available domains, the algorithm model does not learn fixed generation (for example, only from clothes to front models ), but takes the item image and target information as input, and learns to flexibly generate the corresponding scene from the objects in the input image. By using labels to represent domain information, during the training process, a target domain label is randomly generated, and the training model converts the input image to the target domain, so as to control the domain label through semantics, and convert the input to any desired scene output during the training phase , for example, the input generates a model that is standing on the front, clutching a bag, and hanging hands, and outputs a model that meets the requirements of the input clothes.

That is, input an item graph and generate a reasonable scene containing the item. This overcomes two major hurdles, first, multi-domain generation, and second, generating non-existent and plausible scenarios in the input. For the first case, the present invention expresses the label of the training sample in the form of a vector, and corresponds to the input image and the target scene to form a map. By randomly generating a target domain label during the training process, the training model flexibly converts the input The image is converted to the target domain. By doing this, in the stage of using the model, the domain labels are controlled by semantics. For the same input image, input different labels to obtain different scenes and realize multi-domain generation. For the second case, the present invention provides a target scene image and a label describing the scene in the training phase, learns the mapping between the two through a generative adversarial network, and associates the image with the label text. During the training process , the generator learns the image representation of the text, the discriminator recognizes the real image and the generated image, and after an adversarial game, the generator generates a specific scene image that the human eye cannot distinguish between true and false.

The algorithm model of the invention has a simplified structure, convenient training, stable and reliable operation, good portability, and can be used in various specific scenarios.

Finally it should be noted that: the above embodiment is only used to illustrate the technical scheme of the present invention and not to limit it; Although the present invention has been described in detail with reference to the preferred embodiment, those of ordinary skill in the art should understand: The specific embodiment of the invention is modified or some technical features are equivalently replaced; without departing from the spirit of the technical solution of the present invention, all of them should be included in the scope of the technical solution claimed in the present invention.

Claims (1)

1. A specific scene generation method with fusion semantic control is characterized by comprising the following steps:

the method comprises the following steps of 1, selecting a plurality of item pictures and a plurality of different specific scene pictures containing the items, wherein the item pictures are close-up pictures of the items crawled from a shopping website;

step 2, making different attribute labels according to the characteristics of a specific scene in a specific scene picture, and obtaining a training sample after cutting the specific scene picture, wherein the training sample comprises an article picture, a specific scene picture which corresponds to the article picture and contains the article and a label for describing the scene;

step 3, constructing a conditional generation type countermeasure network consisting of a discriminator and a generator;

the generative countermeasure network is a GAN model, and the generator of the generative countermeasure network is represented as

Wherein, in the step (A),

is the target scene image domain, x is the original input image, l is the target scene image domain label,

a particular scene graph described for the tag;

using a cost function of conditional GAN as a antagonistic loss of the model, wherein the cost function is

Wherein D is a discriminator and G is a generator;

step 4, inputting the article graph and the label into a generator together as input to generate a specific scene graph described by the label;

step 5, taking a specific scene graph containing the article as a target scene graph, inputting the specific scene graph, the target scene graph, the article graph and the label described by the label generated by the generator into a discriminator, and performing model training by the discriminator through a conditional countermeasure network;

step 6, inputting the similar object images to be processed and the scene to be obtained into the trained model in a label mode to obtain corresponding scene images; the tags are semantic tags in binary form.

Images