RU2812413C1 - Method for image generation based on user preference analysis - Google Patents

Abstract

FIELD: generating images.

SUBSTANCE: initial training of the neural network and text marking of elements in the image is done by loading a dataset from several images of one element, whereas connections with the ontology are built; images are fed to the input of the CLIP neural network for marking, where parallel analysis of the image and text description is carried out, due to which a feature vector is created; the downloaded images are analysed by segmenting the image in several iterations, by highlighting large objects; a mask with a segmented image is formed; classification of selected objects is carried out using a neural network; in parallel with the identification of objects, image tagging is carried out; at the final stage, images are generated based on an existing library of images, and each image is equipped with a set of tags that determine the possibility of its use in a specific composition.

EFFECT: improvement of quality of image generation.

1 cl, 6 dwg

Description

TECHNICAL FIELD

This technical solution relates to the field of computer technology, in particular, to methods for generating images.

BACKGROUND OF THE ART

The solution chosen as the closest analogue is known from the prior art, US 2012231425 (A1), publ. 09.13.2012. The solution describes a method for providing recommendations for landscape design, including: obtaining a captured image of a landscape area for landscape design; creating a catalog of data relating to a landscape domain, wherein the catalog contains information about a location in the landscape domain, wherein the information is based on the natural characteristics of the location in the landscape domain; receiving information about a product from a mobile device, while the products must be built into the landscape area; matching the product information to a catalog of data relating to locations in the landscape area through the use of a processing device, wherein the matching of the product to a location in the landscape area is based at least in part on the natural characteristics of the location in the landscape area and the natural characteristics of the product; and the presentation of products and product-related metrics in a real-time video stream on a mobile device, at a recommended location within a landscape area, the recommended location being based, at least in part, on matching product information to a catalog of location-related data within the landscape area .

The proposed technical solution is aimed at eliminating the shortcomings of the current level of technology and differs from previously known ones in that the proposed solution generates images of landscape design based on a vector of user preferences in a high-quality and resource-intensive manner.

SUMMARY OF THE INVENTION

The technical problem that the claimed solution is aimed at is creating a method for generating images.

The technical result consists in realizing the purpose of the claimed method.

The claimed technical result is achieved by implementing a method for generating images, including stages at which: initial training of the neural network and text marking of elements in the image are carried out, by loading a dataset of several images of one element, while building connections with the ontology; images are input to the CLIP (Contrastive Language-Image Pre-training) neural network for marking, where parallel analysis of the image and text description is carried out, thereby creating a feature vector containing a text description of the image; the downloaded images are analyzed by segmenting the image in several iterations, by selecting large objects, and, if necessary, selecting smaller image segments; due to the previous stage, a mask with a segmented image is formed, where the pixels of one segment are marked with the same label and form a connected area; then, using a neural network, the selected objects are classified; In parallel with the identification of objects, image tagging is carried out; at the final stage, images are generated based on an existing library of images, each of the images is equipped with a set of tags that determine the possibility of its use in a certain composition, and the image generation process consists of the following steps: a factual description of the picture is compiled; a coordinate grid is formed, to the nodes of which objects are linked; a pool of options is generated, where the initial data is a set of objects that need to be placed on the grid, and the generated placement option is checked for compliance using a rule base.

DESCRIPTION OF DRAWINGS

The implementation of the invention will be described further in accordance with the accompanying drawing, which is presented to explain the essence of the invention and in no way limits the scope of the invention. The following drawing is attached to the application:

Fig. 1 illustrates the processes in the initial training block.

Fig. 2 illustrates an image analysis block.

Figure 3 illustrates a block for demonstrating and working with text data.

Fig. 4 illustrates the generation block.

Fig. Figure 5 illustrates the functional model of an intelligent decision support system with image generation based on the analysis of user preferences.

Fig. 6 illustrates a model of data flows between the main elements of the system.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description of the invention sets forth numerous implementation details designed to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention can be used with or without these implementation details. In other cases, well-known methods, procedures and components have not been described in detail so as not to unduly obscure the features of the present invention.

In addition, from the above discussion it will be clear that the invention is not limited to the above implementation. Numerous possible modifications, alterations, variations and substitutions, while retaining the spirit and form of the present invention, will be apparent to those skilled in the art.

This technical solution relates to the field of generating photorealistic images using machine learning algorithms in the field of design, in particular in the field of landscape design and landscaping. The solution can be used either independently or using a hardware complex with a camera, which ensures quick decision-making on the formation of landscape objects.

Image generation services operating on the "text-to-picture" principle, such as Midjourney, Dall-E, can create images of landscape design, but this does not take into account the user's preferences, his previous choice, or the parameters of the zone in which the object must be placed , and also does not take into account which objects should be placed. It should also be noted that the approach proposed in the solution of combining a deep analysis of user preferences, based on emotional and subconscious choice, with the generation of images based on this analysis, was used for the first time.

There are currently no solutions that generate landscape design images based on a vector of user preferences obtained from the analysis of graphical information, factual input and unstructured text.

There are several approaches to image generation.

Generative adversarial network (GAN).

"GAN" consists of two networks:

1. A generator capable of generating images from an input noise vector;

2. A discriminator that distinguishes between a real picture and a “fake” one. Generation models using "GANs" have several significant disadvantages:

- High requirements for computing power;

- Low generation speed;

- The problem of “overtraining” - a situation is possible when the generator will produce almost identical images to the discriminator.

In this case, the use of the reverse diffusion method turned out to be more preferable. This method is used by: DALLE 2, Midjourney, Imagen, etc.

Diffusion models are used to create data similar to the data they are trained on.

Training the diffusion model:

1. An image of a real or drawn object is used;

2. Gaussian noise is added to the required extent so that the final image has the required number of similarities with the original one;

3. The model is trained to perform noise reduction to create a result. The Stable Diffusion network created by StabilityAl (https://stablediffusionweb.com) was tested. Stable Diffusion builds on work on high-resolution image synthesis using latent diffusion models by Rombach et al.

The complete stable diffusion architecture consists of three models:

1. Text encoder, which accepts a text prompt and provides conversion of text into feature vectors;

2. U-Net is a diffusion model responsible for generating images. The architecture is shown in Figure 19.

3. Variational autoencoder, consisting of an encoder and decoder model.

The encoder is used to reduce the image size, which is what the UNet diffusion model works with. The decoder is then responsible for restoring the image generated by the diffusion model to its original size.

Due to the iterative nature of the propagation process, the training and generation process is more stable than in GANs. In diffusion models, transitions from pure noise to an image are made in several steps, and in GANs in one step, which is one of the sources of unstable GAN training.

The proposed technical solution makes it possible to reduce the analysis of preferences by several times, while additionally producing ready-made photorealistic images as an output, as well as additional documentation necessary for landscape design projects.

An important feature of the claimed technical solution is the ability to be used both as a stand-alone system and in conjunction with cameras. In the latter case, generation is possible immediately after photographing the land plot.

The advantage of the claimed technical solution is the digitalization of processes for determining user preferences to create unique environmental design projects, according to the preferences and needs of the user in such complex areas as various types of design (interior, landscape, design of things, architectural), where the choice of based on subconscious decisions and unstructured information. To develop the models and further test them, the direction of landscape design and landscaping was chosen, which is one of the most difficult from the point of view of creating full-fledged projects. The developed solution can be applied to various areas of design of things and environments.

The standard approach to creating landscape design, when working as a specialist designer, is to show the client images with various finished projects, photographs of gardens, flower beds, and so on, as well as conducting surveys: what plants do you like, what design elements should be there, what their location should be on the site and so on. The designer analyzes the information received, and based on the user’s visual preferences, as well as on the basis of professional knowledge regarding norms and rules, the specialist creates a project that should be unique for each user.

However, the present invention used methods for quickly and automatically assessing user preferences using various information technologies, and also proposed an original assessment model based on the analysis of displayed images.

Let us consider in more detail the main blocks with the help of which the claimed method is implemented.

1. Determining or assessing user preferences;

2. Generating images based on certain user preferences.

The model for determining user preferences consists of two complementary lines of analysis:

1. Analysis of stylistic preferences (general style of LD, styles of individual elements of landscape design (LD));

2. Analysis of preferences for object types.

The basis of the analysis is:

- Use of marked photographs of various LD styles;

- Using object libraries;

- Use of a special ontology;

- Trained convolutional neural network.

Tagged photographs of styles and individual objects allow the system to learn to “understand” which objects are of interest to the user.

The work of the model is based on the following blocks:

1. Initial training block;

2. LD image analysis unit;

3. Block for demonstrating and working with text data.

Let's take a closer look at them.

Initial training block.

In this block, the initial training of the neural network and text marking (tagging) of LD elements takes place. LD elements are understood as any objects of the classes plant, MAF, and so on. In this case, the element can be composite, for example, an object like a “flower bed” should be recognized on complex LD images, while it should be separately recognized by its constituent parts.

Element recognition process.

A dataset is loaded from several photographs of one element (at least 100, prepared datasets from iNaturalist or ImageNet are used). At the same time, a connection with ontology must be built - i.e. When loading, it is recorded which class these instances belong to. For this model, our own ontology was developed, which describes all elements and objects of landscape design, with all possible properties and attributes. For different types of objects, lists of possible attributes (features) have been generated. Objects and features are parts of the ontology.

The images are input to the neural network.

For marking, a neural network called CLIP (Contrastive Language-Image Pre-training) is used, which can significantly speed up the process of preprocessing graphic objects and their classification (tagging). The main operating principle is parallel analysis of both images and text descriptions, which allows you to create a feature vector containing a text description of the image (flower, white inflorescence, smooth stem, dark green, arrow-shaped leaves). In fact, image classification is automated and the image is tagged. In our case, the received tags are checked against the ontology and thus a specific photo is linked as an instance of the class.

LD image analysis unit.

In this block, complex complex images are analyzed using a neural network previously trained on a library of elements.

Algorithm of actions.

The LD photo is automatically loaded.

Next, the image is segmented, in several iterations - i.e. First, large objects are identified, for example, houses, MAFs, LD objects, and only then everything is segmented into smaller segments. As a result of the algorithm, we obtain a mask with a segmented image, where the pixels of one segment are labeled with the same label and form a connected area. The algorithm performs well for solving the problem of primary identification of the boundaries of objects of interest in an image (Watershed boxing). By varying the initial settings of the algorithm, including the sensitivity threshold, as well as restrictions on the size of objects, you can obtain a relevant selection of objects for further processing.

Next, using a neural network, the selected objects are classified from the initial training block. If an object is not automatically recognized, then it must be selected and sent to an analyst for manual classification and storage in the elements database.

In parallel with the definition of objects, tagging of the image is carried out - that is, automatic addition of service information to the style image.

Thus, the input of the block is an image with an LD, and the output is a vector of image features, including:

LD style (determined based on training results);

A list of found objects, with a hierarchy of inclusion (i.e. if it is a flower, then it is included in the flowerbed, and so on);

Properties of objects.

Description of the image generation model.

The following generation model is proposed, the main processes of which are illustrated in Figure 4.

A sequential approach to working with generation is used:

1. Simplified (fast) generation;

2. Fully functional generation. Fast generation.

1. If there is a sufficient set of data, the generation parameters are set:

- Style;

- Types of objects;

- Number of objects;

- Properties of objects (class, attributes);

- Location of objects.

2. Next, using a neural network, a plan for the location of objects is created taking into account the specified parameters.

Image generation is carried out based on an existing image library. Each of the images is equipped with a set of tags that determine the possibility of its use in a specific composition. The imaging process consists of the following steps:

1. A factual description of the picture is drawn up, which includes:

- style of composition;

- a set of required objects (and their number);

- a set of preferences that influence the rules for placing objects.

2. A coordinate grid is formed, to the nodes of which objects will be linked. The choice of lattice size affects, on the one hand, the performance of the algorithm, and on the other hand, the visual effects

3. A pool of options is generated. The initial data is a set of objects that need to be placed on the grid. The generated placement option is checked against a base of rules selected from the database in relation to the composition. Rules can be expressed both in declarative and in vague form, allowing for vague interpretation.

4. To speed up the process of selecting options that satisfy the rules, an implementation of a genetic algorithm is used.

During generation, rules of type C work, in particular C21 ('Coniferous trees of the same species nearby are undesirable'), specified with 80% accuracy. The parameters of the minimum and maximum distance between flowers and trees specified for the coordinate grid also determine the nature of the placement of objects

To more accurately satisfy the user's request, during generation it is possible to use a background in the form of photographs uploaded earlier.

After the user stops the demo, the images he selected, created using simple generation, are enabled for full-featured generation.

System architecture.

1. Functional model of the system.

The considered models for determining user preferences and generating images are a system of interconnected functional components. To build a conceptual vision of the system’s operation and organize a vision of information flows, a functional model of the system’s operation is formulated by analogy with the IDEF0 notation.

The control influence on all functional modules is exerted by elements of the ontology and rules of landscape design, as well as marked images with a vector of assigned features.

The system data flow model consists of the following elements:

- entities of the designed system, including technical functional modules, as well as modules that interact with the external environment;

- data storage and storage devices (databases);

- data flows between entities;

- processes;

- subjects involved in processes;

- external objects (devices).

A special conditional diffusion model is trained to generate landscape design images according to user preferences. To train the model, 102,000 images of landscape design were collected from open sources. The image data was reduced to a single size (512x512 px) and analyzed by the Detector to extract a feature vector. During its training, the inputs of the diffusion model were simultaneously supplied with information about the features of the images and the images themselves. To generate images of landscape design, stable diffusion technology (diffusers library) was selected. To speed up the generation process, the Unet family model prepares a hidden layer, from which the autoencoder then reconstructs the target image. Because the system generates according to condition, the Unet2dConditionModel is selected.

Training a diffusion model specifically for generating landscape design images allowed us to reduce image generation time from 3.5 s to 1 s on an Nvidia T4 GPU, as well as increase the variety of objects and scenes in the generated images.

The solution allows you to detect plants in the generated image and select similar images in the service database, taking into account winter hardiness, lighting and budget for their subsequent ordering in nurseries.

Plant selection algorithm.

1. Upon receiving a selection request from the main operating unit, an http request is sent to the detection service.

2. The detection service, having received the specified request, analyzes the generated image and returns a binary stream of serialized data: the class label of the detected plant, the coordinates of the bounding box, as well as the graphic content itself.

3. The plant selection service, having received data from the detection service, selects feature vectors of images (embeddings) of plants and compares them with plant vectors from the service database, calculating the cosine distance between them. In order to extract image embeddings, a machine learning model was pre-trained - an autoencoder of the DINOv2 architecture. This model was trained on images used to train the image detection service and subsequently processed by the detection service.

4. Information about the plants selected in this way is presented to the user.

The generated images are saved to S3 object storage, and information about them is entered into the Database for further use in selecting plants, generating angles, and embedding custom objects in the image.

The resulting diffusion machine learning model copes with the task of generating an image on an Nvidia P100 video accelerator in 1.2 s and occupies 1.5 GB of video memory, which is significantly lower than the standard models of the general-purpose Stable Diffusion family. The quality of the resulting images is almost identical to photographs of real objects. The service has information for connecting to a third-party S3 storage and performs pre-processing and subsequent sending of images to the storage, as well as sending a request to register information about the image in the application Database. During its operation, the service uses an object detection service to obtain information about the downloaded image, as well as the backend of the application to save this information. This service, in turn, uses the application generation service to save the results of its work.

A computing system capable of providing data processing necessary to implement the claimed solution, in the general case, contains components such as: one or more processors, at least one memory, data storage means, input/output interfaces, input means, network communication means.

When executing machine-readable instructions contained in the RAM, the processor of the device is configured to perform basic computing operations necessary for the operation of the device or the functionality of one or more of its components.

Memory, as a rule, is made in the form of RAM, into which the necessary program logic is loaded to provide the required functionality. When implementing the proposed solution, the amount of memory necessary to implement the proposed solution is allocated.

The data storage medium can be in the form of HDDs, SSDs, raid arrays, network storage, flash memory, etc. The tool allows long-term storage of various types of information, for example, the above-mentioned files with user/passenger data sets, databases containing records of time intervals measured for each user, user IDs, etc.

Interfaces are standard means for connecting and operating peripheral and other devices, for example, USB, RS232, RJ45, COM, HDMI, PS/2, Lightning, etc.

The choice of interfaces depends on the specific design of the device, which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc.

A keyboard can be used as a data input means in any embodiment of a system that implements the described method. The hardware design of the keyboard can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server or other computer device. The connection can be either wired, in which the keyboard connecting cable is connected to the PS/2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports. In addition to the keyboard, data input means can also include: joystick, display (touch display), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

The means of network interaction are selected from a device that provides network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. The tools ensure the organization of data exchange via a wired or wireless data transmission channel, for example, WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM.

The device components are interconnected via a common data bus.

In these application materials, a preferred disclosure of the implementation of the claimed technical solution was presented, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims (1)

A method for generating images, including stages at which: initial training of a neural network and text marking of elements in the image is carried out by loading a dataset of several images of one element, while building connections with the ontology; images are input to the CLIP (Contrastive Language-Image Pre-training) neural network for marking, where parallel analysis of the image and text description is carried out, thereby creating a feature vector containing a text description of the image; the downloaded images are analyzed by segmenting the image in several iterations, by selecting large objects, and, if necessary, selecting smaller image segments; due to the previous stage, a mask with a segmented image is formed, where the pixels of one segment are marked with the same label and form a connected area; then, using a neural network, the selected objects are classified; In parallel with the identification of objects, image tagging is carried out; at the final stage, images are generated based on an existing library of images, each of the images is equipped with a set of tags that determine the possibility of its use in a certain composition, and the image generation process consists of the following steps: a factual description of the picture is compiled; a coordinate grid is formed, to the nodes of which objects are linked; a pool of options is generated, where the initial data is a set of objects that need to be placed on the grid, and the generated placement option is checked for compliance using a rule base.