RU2768551C1 - Method for local generation and representation of wallpaper stream and computer implementing it - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to a method and a device for local generation of wallpaper on a computer. In the method, at least one first wallpaper of a wallpaper stream is generated on the computer, using a deep generative neural network, wherein the deep generative neural network is trained on a collection of high-quality images/videos and pre-loaded to the computer, and at least one first wallpaper is installed on the computer as wallpaper of this computer. At the same time, the method additionally contains a stage, at which current positions and colors of icons and widgets on a device screen are determined, and determined current positions and colors of icons and widgets are used as additional parameters of the deep generative neural network, when generating wallpaper.

EFFECT: generation of wallpaper taking into account current displayed data of the device.

13 cl, 3 dwg

Description

The field of technology to which the invention belongs

[0001] The present invention relates generally to the field of artificial intelligence and, in particular, to a method for generating and providing a wallpaper stream on a computing device using a deep generative neural network, as well as to a computing device that implements this method.

Description of the Prior Art

[0002] Wallpapers make up the majority of what a user sees on various devices, including smartphones, smart TVs, laptops, and the like. Currently, in order to regularly receive new aesthetically pleasing wallpapers, users can subscribe to online updates, that is, services (for example, Unsplash ) that regularly send new wallpapers to the user device via an Internet connection. The disadvantages of this method include, at the very least, the need to connect to the Internet, as well as the consumption of traffic and bandwidth.

[0003] Modern generative neural networks are capable of synthesizing highly realistic 2D images, 3D images, and videos. Therefore, such networks can be trained to generate images of realistic, aesthetically pleasing wallpapers. Once trained, many of these models (of which generative adversarial networks are the most widely used class) can generate an infinite variety of wallpapers by taking an arbitrary high-dimensional vector as input and generating an original image for that vector. Including a new vector will result in a significantly different image.

SUMMARY OF THE INVENTION

[0004] An alternative technical solution has been proposed that allows the wallpaper of a user device to be regularly updated without using the Internet. According to a first aspect of the present invention, there is provided a method for locally generating and providing a wallpaper stream to a computing device, comprising: generating at least one first wallpaper of the wallpaper stream on the computing device using a deep generative neural network, wherein the deep generative neural network is trained on a collection of high-quality images/ video and pre-loaded into the computing device, and set on the computing device at least one first wallpaper as the wallpaper of this computing device. Since acceptable wallpapers are synthesized on the computing device itself, i.e. are not downloaded from the Internet, there are no disadvantages described above, and the invention allows to reduce/eliminate the consumption of traffic and bandwidth, which were required in the known analogues for downloading.

[0005] According to a second aspect of the present invention, there is provided a computing device comprising a processor and a storage device storing a trained deep generative neural network for locally generating and providing a wallpaper stream by executing the method according to the first aspect when the trained deep generative neural network is executed by the processor.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The above and other aspects, features and advantages of the present invention will be better understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

fig. 1 is a flowchart of a method for locally generating and providing a wallpaper stream to a computing device in accordance with the embodiment of the invention disclosed herein;

fig. 2 is a flowchart of a method for locally generating and providing a wallpaper stream to a computing device in accordance with another embodiment of the invention disclosed herein;

fig. 3 is a block diagram of a computing device according to an embodiment of the invention disclosed herein.

[0007] In the following description, unless otherwise indicated, the same reference signs are used for the same elements shown in different drawings, and their parallel description may be omitted.

DETAILED DESCRIPTION

[0008] FIG. 1 shows the sequence of operations a method for locally generating and providing a wallpaper stream to a computing device according to an embodiment of the invention disclosed herein. The method is as follows: at least one first wallpaper of the wallpaper stream is generated S105 on the computing device using a deep generative neural network, wherein the deep generative neural network is trained on a collection of high-quality images/videos and pre-loaded into the computing device. Generating refers to the artificial synthesis of wallpapers by a deep generative neural network. Depending on the type of content in the collection of high-quality images/videos that was used in the training phase of the deep generative neural network, the deep generative neural network, when trained on such a collection, is configured to generate a type of wallpaper similar to this content. For example, if a collection of high-quality images contains images of landscapes, then a deep generative neural network, when trained on this collection, will be configured to generate landscape wallpapers, etc. However, the present invention is not limited to landscape wallpapers, since a collection of high-quality images/videos with any other type of content can be used in the training phase of a deep generative neural network. In addition, if a collection of high-quality videos was used at the training stage of a deep generative neural network, then the deep generative neural network, when trained on such a collection, will be configured to generate video wallpapers with a content type corresponding to the content type of the training collection of high-quality videos. A wallpaper stream refers to one or more images, which may be static, dynamic, or interactive, and/or one or more videos. When training a deep generative neural network, it can be loaded into a computing device along with weights and other parameters in advance, that is, before the use phase (inference). The deep generative neural network may be stored in a storage device such as a memory of a computing device. When generating at least one first wallpaper by a deep generative neural network, the method includes setting S110 in the computing device these at least one first wallpaper as the wallpaper of the computing device. The generated wallpaper can be for any type of user interface, for example, the generated wallpaper can be the wallpaper for the main desktop, for the lock screen, for a blank browser page, and so on without limitation. The generated wallpaper can be used as a screen saver of a computing device.

[0009] As shown in FIG. 1, the method further comprises the step of determining S115 whether the condition is met or not. This condition check is used to determine if the first wallpaper should be updated with the second wallpaper. The condition includes, without limitation, one or more of the following: (i) a user input is received on the computing device, wherein the user input represents whether the user of the computing device expresses disapproval of the wallpaper currently set as the wallpaper of the computing device, or not; (ii) a preset time period has elapsed; (iii) the computing device's GPS location has changed, wherein the computing device's GPS location can be registered by a GPS unit present in the computing device. If the user has expressed disapproval of the currently set wallpaper, then the method can be adapted to generate other wallpapers. Based on the information of such approvals/disapprovals (likes/dislikes), the system can learn to generate wallpapers that the user will like. If it is determined that this condition is satisfied (i.e., a disapproval is received at step S115), the method updates the wallpaper currently set as the wallpaper of the computing device by performing the following steps: generating S120 on the computing device at least one second wallpaper of the wallpaper stream using a deep generative neural network; setting S125 on the computing device the at least one second wallpaper as the wallpaper of the computing device. The wallpaper update described can be performed automatically in the background. For example, the user can enable the feature to generate new wallpapers every morning. The terms "first" and "second" in relation to the term "wallpaper" are used for the purpose of distinguishing them and should not be construed as terms representing any ordinal relationship between said wallpaper or method steps. At least one second wallpaper is different from at least one first wallpaper. In an alternative embodiment of the method (not illustrated), the step of determining whether a condition is met or not S115 may be performed before generating and setting each subsequent wallpaper, including the case where the step of determining S115 is performed before steps S105 and S110 described above. Therefore, it should be understood that the reference numbers are used for purposes of illustration only and should not be construed as representing any ordinal relationship between the steps of the method.

[0010] FIG. 2 is a flow diagram of a method for locally generating and providing a wallpaper stream to a computing device according to another embodiment of the invention disclosed herein. The embodiment of the method shown in FIG. 2 differs from the embodiment of the method shown in FIG. 1 in that it further comprises the steps of customizing the deep generative neural network S95 for a computing device user by using arbitrary input as a parameter of the deep generative neural network ensuring that the deep generative neural network is configured to generate a unique wallpaper for a given computing device user, and personalizing S100 a deep generative neural network for the user of the computing device, ensuring that the wallpapers generated by this deep generative neural network are personalized for this user. Steps S105-S125 shown in FIG. 2 may be similar to steps S105-S125 shown in FIG. 1, except that at least one first wallpaper and at least one second wallpaper generated and installed in the embodiment of FIG. 2, customized and personalized. Thus, each user gets their own unique wallpaper every time.

[0011] The S100 personalization may be based on, but not limited to, one or more of the following personalization parameters: one or more user preferences, one or more user inputs, one or more computing device settings, current time of day, current time of year, current the computing device's GPS location, the content of the user's gallery currently stored on the computing device, the contents of the browser history currently stored on the computing device, the current weather and weather forecast, the position and colors of icons and widgets on the device's screen. The current position and colors of icons and widgets on a device screen can be determined and used in this method to synthesize wallpapers that do not merge with the icons and widgets. In this case, the current position and colors of icons and widgets on the screen of the device can be entered into the deep generative neural network before generating/synthesizing the wallpaper as the relevant parameters to have a corresponding effect on the output of the deep generative neural network. The method may further include (not shown) the steps of parsing the contents of the user's gallery, inferring that the user is into photography of mountains, and adapting the deep generative neural network to generate more images/videos/interactive wallpapers of the mountains. Although in FIG. 2 shows that steps S95 and S100 are performed before steps S105 and S110, this should not be considered as limiting, since these steps can be performed in a different order if necessary, for example, one or both of steps S95 and S100 can be performed before generating and installing each subsequent wallpaper, including the case where one or both of steps S95 and S100 are performed before steps S120 and S125 described above. Thus, it should be understood that the reference numbers are used for purposes of illustration only and should not be construed as representing any ordinal relationship between the steps of the method.

[0012] A deep generative neural network is trained using an adversarial learning process along with one or more discriminant networks. Training is performed on a high performance computer or computing cluster on a large set of image and/or video data having wallpaper quality. A deep generative neural network may have one or more of the following variables: vector variables, latent variables in the form of a two-dimensional matrix, or a set of two-dimensional matrices. In one embodiment, latent variables can be obtained from unit normal distributions (unit normal distributions). Personalization can be performed by a separate encoder network trained to map personalization parameters to parameters of spatial distributions of latent variables, such as the mean and the covariance of the Gaussian normal distribution,from which latent variables are extracted for a deep generative neural network. When the user's sensitive information, such as one or more user preferences, one or more user inputs, one or more computing device settings, the computing device's current GPS location, the content of the user gallery currently stored on the computing device, the contents of the browser history stored in at the moment on the computing device is processed as a personalization parameter(s), it should be clear that such confidential user information is not compromised, since all processing of the specified confidential user information is performed by the processor of the user's computing device, as well as a separate encoder network and a deep generative neural network stored in the storage device of the user's computing device. In other words, it is preferable that the user's confidential information does not leave the computing device for processing.

[0013] In an alternative embodiment of the method (not illustrated), the step of generating at least one wallpaper S105, S120 further comprises the steps of image synthesis and image modification, and the step of setting S110, S125 of at least one wallpaper as the wallpaper of the computing device further comprises the step of setting modified image as at least one first wallpaper. In another embodiment (not illustrated) of the method, the step of generating at least one wallpaper S105, S120 further comprises the steps of image synthesis and image animation, and the step of setting S110, S125 of at least one wallpaper as the wallpaper of the computing device further comprises the step of setting an animated image as at least one first wallpaper. Due to the lack of bandwidth restrictions and essentially free content, it is possible to generate free dynamic wallpapers (high resolution videos) using the corresponding model. In addition to dynamic wallpapers, you can generate interactive wallpapers. The wallpaper on a smartphone can change appearance in response to a user's swipe, phone tilt, or certain interface events. For example, swiping to a different tab on the Android screen may make some change to the image (for example, move the clouds in the image in the direction of the swipe). According to another alternative implementation of the proposed method, a deep generative neural network can be adapted to generate not only realistic and believable images, but also hyperrealistic images, which may, for example, have exaggerated features, such as extremely saturated sunset colors, exaggerated geometric proportions of objects (trees, buildings), etc. Some users may desire and prefer such hyper-realistic wallpapers. Most generative models (such as adversarially trained neural networks) can allow each user to set their own preferred trade-off between realism and hyperrealism. In another embodiment of the method, the step of generating at least one wallpaper S105, S120 further comprises the step of applying ultra-high resolution to the synthesized image, and the step of setting S110, S125 of at least one wallpaper as wallpaper of the computing device further comprises the step of setting an image having super high resolution. resolution, as at least one first wallpaper. Specific methods for modifying an image, applying ultra high resolution or hyper resolution to an image, are known in the art.

[0014] FIG. 3 shows a block diagram of a computing device 200 according to one embodiment of the invention disclosed herein. Computing device 200 includes a processor 205 and a storage device 210. The processor 205 is configured to perform processing and computational tasks associated with operation of the computing device and operations in accordance with the proposed method. The storage device stores the trained deep generative neural network 210.1 for locally generating and providing a wallpaper stream by executing the proposed method when the trained deep generative neural network 210.1 is executed by the processor 205. The storage device 210 may also store processor-executable instructions that cause the processor to execute one or more of the above-described method steps. Processor 205 and storage device 210 may be operatively coupled. Processor 205 and storage device 210 may also be coupled to other components (not shown) of the computing device. Another component may include, without limitation, one or more displays, touch screen, keyboard, keypad, communication unit, speaker, microphone, camera, Bluetooth unit, NFC (near field communication) unit, RF (radio frequency) unit, unit GPS, input/output facilities, as well as the necessary electrical wires and connections, etc. The processor 205 may be implemented as, but is not limited to, a general purpose processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a system on a chip (SoC). The storage device 210 may include, without limitation, RAM, ROM, and so on. Thus, the computing device may be, without limitation, a user's computing device such as a smartphone, tablet, laptop, laptop, smart TV, in-car infotainment system, and so on.

Other implementation details

[0015] Any part of the following information should not be construed as a limitation of the present invention. Rather, the following information is provided to enable those skilled in the art to practice the embodiments described and to prove the sufficiency of this disclosure. Any specific values for any of the parameters below should not be construed as limiting.

[0016] Model architecture . Model architecture can be based on StyleGAN. The model produces images with a resolution of 256x256 (or 512x512) and has four sets of latent variables:

, который кодирует цвета и общий план сцены;- vector

, which encodes colors and the general plan of the scene;

^, который кодирует глобальное освещение (например, время суток);- vector

^,which the encodes global illumination (e.g. time of day);

квадратных матриц

,

, которые кодируют формы и детали статических объектов при N=7 различных разрешений между 4×4 and 256×256 (N=8 для 512×512);- kit

square matrices

,

, which encode the shapes and details of static objects at N=7 different resolutions between 4x4 and 256x256 (N=8 for 512x512);

квадратных матриц

,

, которые кодируют формы и детали динамических объектов при соответствующих разрешениях.- kit

square matrices

,

, which encode the shapes and details of dynamic objects at appropriate resolutions.

и преобразует его в вектор стиля

. Сверточный генератор G имеет N=7 (или 8) блоков. Внутри каждого блока за сверткой следуют два поэлементных сложения двух тензоров, полученных из

and

с помощью обучаемого поканального масштабирования. И наконец, применяется преобразование AdaIN с использованием поканальных масштабов и смещений, полученных из W с помощью обучаемого линейного преобразования. В каждом блоке эта последовательность шагов повторяется дважды, а затем следует повышающая дискретизация и сверточные слои.[0017] The generator has two components: a multilayer perceptron M and a convolutional generator G. The perceptron M takes a concatenated vector

and convert it to style vector

. The convolutional generator G has N= 7 (or 8) blocks. Within each block, the convolution is followed by two element-wise additions of the two tensors obtained from

and

using learnable per-channel scaling. Finally, the AdaIN transform is applied using the per-channel scales and offsets obtained from W with the trainable linear transform. In each block, this sequence of steps is repeated twice, followed by upsampling and convolutional layers.

[0018] The following set of input latent variables will be mentioned next:

}

}

И наконец, набор всех пространственных случайных вводов генератора будет обозначаться какas initial inputs (or initial latent variables). As with StyleGAN, the convolutional generator can use separate W vectors at each resolution (style blending). The set of all style vectors will be denoted

Finally, the set of all spatial random generator inputs will be denoted as

.

.

и наборе данных таймлапсовых видео пейзажей

. Большой набор данных статических изображений собрать относительно легко, кроме того, авторы приложили максимальные усилия для сбора нескольких сотен видео, которые не охватывают все разнообразие пейзажей. Таким образом, оба источника данных можно использовать для построения модели с улучшенными характеристиками. Для этого предлагаемую генеративную модель (глубокую генеративную нейросеть) обучают состязательным методом с двумя разными дискриминаторами.[0019] Model training. The model is trained on two data sources: a static landscape image dataset

and a time-lapse video landscape dataset

. A large set of still image data is relatively easy to collect, and the authors have made every effort to collect several hundred videos that do not cover all the variety of landscapes. Thus, both data sources can be used to build a model with improved performance. To do this, the proposed generative model (deep generative neural network) is trained by the adversarial method with two different discriminators.

имеет такие же выборы архитектуры и дизайна как и в StyleGAN. Он рассматривает изображения из

как реальные, а фиктивные экземпляры генерируются моделью. Парный дискриминатор

рассматривает пары изображений. Он дублирует архитектуру

, за исключением первого сверточного блока, который применяется отдельно к каждому кадру. Реальная пара изображений получается путем выборки видео из

, и последующей выборки двух произвольных кадров (произвольно удаленных друг от друга) из него. Фиктивная пара получается путем выборки общих статичных латентных переменных

и

, а затем индивидуальных динамичных латентных переменных

,

, и

,

. Затем два изображения получаются как

и

. Все экземпляры берутся из единичных нормальных распределений. [0020] Static discriminator

has the same architecture and design choices as in StyleGAN. He looks at pictures from

as real, and dummy instances are generated by the model. Paired discriminator

considers pairs of images. It duplicates the architecture

, except for the first convolution block, which is applied separately to each frame. A real pair of images is obtained by sampling video from

, and then fetching two arbitrary frames (arbitrarily distant from each other) from it. A dummy pair is obtained by sampling common static latent variables

And

, and then individual dynamic latent variables

,

, And

,

. The two images are then obtained as

And

. All instances are taken from unit normal distributions .

[0021] The model is trained in a non-saturating loss GAN with R1 regularization. During each update of the generator, a batch of dummy images is sampled to which a static discriminator is applied, or a batch of image pairs is sampled to which a pairwise discriminator is applied. The proportions of the static discriminator and the paired discriminator are selected from 0.5/0.5 to 0.9/0.1, respectively, at each resolution transition phase, and then kept fixed at 0.1. This helps the generator learn to separate the static and dynamic latents for each resolution ahead of time and prevents the paired generator from overfitting on the relatively small video dataset used for training.

[0022] During training, the goal of the paired discriminator is to focus on the inconsistencies in each pair, and the goal of the static discriminator is to focus on visual quality. Also, since the pairwise discriminator only sees real frames selected from a limited number of videos, it may be prone to overfitting on this limited set and actually stop contributing to the learning process (while a static discriminator observing a more diverse set of scenes continues to improve model diversity). It turned out that both problems (focusing on image quality rather than pairwise consistency, overfitting on a limited variety of videos) can be solved with a simple method. It lies in the fact that a fictitious set of frames is supplemented with pairs of image fragments taken from the same video frame, but from different places. Since these fragments have the same visual quality as images in real frames, and since they are obtained from the same videos as images in real pairs, the pair discriminator actually stops paying attention to image quality, cannot simply retrain on the statistics of scenes in the set video data, and should focus on finding pair mismatches in dummy pairs. This sampling method can be used to greatly improve the quality of the model.

[0023]Sample video from model. The model does not try to learn the full temporal dynamics of the video, but focuses on pairwise consistency. frames that are generated by resampling dynamic latent variables. In particular, the pairwise discriminator in the model does not sample real frames sequentially. The sampling procedure for dummy pairs also does not attempt to generate contiguous frames. One of the reasons why the described training phase does not attempt to learn continuity is that the training dataset contains videos with widely varying temporal coefficients, which makes the notion of temporal proximity for a pair of frames virtually meaningless.

и

. В ходе экспериментов было обнаружено, что для генерирования привлекательных видео достаточно описанной ниже простой модели движения. В частности, для выборки видео можно выбрать один статический вектор

из единичного нормального распределения, а затем интерполировать динамический латентный вектор между двумя единичными нормально распределенными выборками

и

. Для пространственных отображений также можно выбрать

и

из единичного нормального распределения, а затем непрерывно деформировать

тензор, используя преобразование гомографии, параметризованное смещениями двух верхних углов и двух точек на горизонте. Направление гомографии выбирается произвольно, скорость выбирается в соответствии со средней скоростью облаков в наборе обучающих данных. Гомография транспонируется по вертикали для положений ниже горизонта, чтобы имитировать процесс отражения. Для получения

можно сделать композицию из

идентичных преобразований, а затем применить ее к

. По мере интерполяции/деформации латентных переменных они проходят через обученную модель для получения плавных видео. Следует отметить, что описываемая модель не требует пользовательского ввода для конкретного изображения.[0024] For this reason, the proposed generation process is independent of the motion model. The generator is forced to generate believable frames regardless of changes

And

. Through experimentation, it has been found that the simple motion model described below is sufficient to generate compelling videos. In particular, one static vector can be selected for video sampling

from a unit normal distribution and then interpolate a dynamic latent vector between two unit normally distributed samples

And

.For spatial mappings can also be selected

And

from a unit normal distribution and then deform continuously

tensor using a homography transformation parameterized by the offsets of the top two corners and the two horizon points. The direction of the homography is chosen arbitrarily, the speed is chosen according to the average speed of the clouds in the training dataset. The homography is transposed vertically for positions below the horizon to simulate the reflection process. For getting

you can make a composition

identical transformations, and then apply it to

. As the latent variables are interpolated/warped, they are passed through the trained model to produce smooth videos. It should be noted that the described model does not require user input for a particular image.

, выводится набор латентных переменных, которые генерируют такое изображение в генераторе. Отыскиваются расширенные латентные параметры

и

, так чтобы

После этого можно применить ту же процедуру, что и выше, для анимации данного изображения.[0025] Animation of real scenery images using the . stage of use. To animate a specific landscape image

, a set of latent variables is displayed that generate such an image in the generator. Searching for extended latent parameters

And

, so that

After that, you can apply the same procedure as above to animate this image.

должна принадлежать выходному многообразию

). Без такого предварительного распределения латентные переменные, которые генерируют хорошую реконструкцию, могли бы все же привести к неправдоподобной анимации (или ее отсутствию). Следовательно, вывод можно реализовать, используя следующую трехэтапную процедуру:[0026] The latent space of the generator is highly redundant, and to get good animation it is necessary to ensure that the latent variables come from approximately the same distribution as during model training (most importantly,

must belong to the output manifold

). Without this predistribution, latent variables that generate a good reconstruction might still result in implausible animation (or no animation). Therefore, the output can be implemented using the following three-step procedure:

используя сеть кодировщика

прямого распространения. Кодировщик имеет архитектуру ResNet-152 и обучен на 200000 синтетических изображениях со средней абсолютной потерей на ошибки.

предсказывается двухслойным персептроном с ReLU на основе конкатенации признаков с нескольких слоев ResNet, агрегированных глобальным усредняющим пулингом. 1. Stage 1 : a set of style vectors is predicted

using encoder network

direct distribution. The encoder has a ResNet-152 architecture and is trained on 200,000 synthetic images with an average absolute error loss.

predicted by a two-layer perceptron with ReLU based on the concatenation of features from multiple ResNet layers aggregated by global average pooling.

' и нулевого

, все латентные параметры оптимизируются для уменьшения ошибки реконструкции. Кроме того, назначаются штрафы на отклонение

от прогнозируемого

(с коэффициентом 0,01) и отклонение S от нуля (путем снижения скорости обучения). Оптимизация выполняется в процессе вплоть до 500 шагов с ADAM и большой начальной скоростью обучения (0,1), которая уменьшается вдвое каждый раз, когда потери не улучшаются в течение 20 итераций. В одном варианте метода, который оценивался отдельно, используется двоичная маска сегментации, полученная с помощью предварительно обученной сети сегментации ADE20k. Эта маска определяет динамические (небо + вода) и остальные (статические) части сцены. В данном варианте

(соответственно,

) сохраняются нулевыми для динамических (соответственно, статических) частей изображения. 2. Stage 2: beginning with

' and zero

, all latent parameters are optimized to reduce the reconstruction error. In addition, penalties are imposed for deviation

from the predicted

(with a factor of 0.01) and the deviation of S from zero (by reducing the learning rate). Optimization is done in process up to 500 steps with ADAM and a large initial learning rate (0.1), which is halved every time the loss does not improve over 20 iterations. One variant of the method, which was evaluated separately, uses a binary segmentation mask obtained from a pretrained ADE20k segmentation network. This mask defines the dynamic (sky + water) and the rest (static) parts of the scene. In this variant

(respectively,

) are kept zero for dynamic (respectively, static) parts of the image.

для дальнейшего уменьшения ошибки на реконструкцию. Этот этап необходим, так как даже после оптимизации остается расхождение между реконструкцией и введенным изображением. Во время тонкой настройки минимизируется комбинация попиксельной средней абсолютной ошибки и потери восприятия, причем последняя имеет гораздо больший вес (10х). Выполняется 500 шагов с ADAM и

=0,001. 3. Stage 3 : latent parameters are frozen and the scale is fine-tuned

to further reduce the reconstruction error . This step is necessary because even after optimization there is a discrepancy between the reconstruction and the input image. During fine tuning, the combination of per-pixel mean absolute error and perceptual loss is minimized, with the latter having a much higher weight (10x). Perform 500 steps with ADAM and

=0.001.

используется для отображения

в

Осуществляется повторная выборка

, чтобы учесть изменения освещения, погоды и т.п. и чтобы

описывала только статические атрибуты (землю, строения, форму горизонта и т.д.). Для изменения освещения в реальном изображении необходимо изменить

, а затем использовать MLP для получения новых стилей

. Однако описанная процедура вывода выдает только

и было обнаружено, что очень сложно инвертировать

и получить

[0027] Lighting Manipulation . During model training

used to display

in

Resampling in progress

to account for changes in lighting, weather, etc. and to

described only static attributes (land, buildings, horizon shape, etc.). To change the lighting in the real image, you need to change

and then use MLP to get the new styles

. However, the described inference procedure only produces

and it was found to be very difficult to invert

and get

обучают аппроксимировать локальные динамические параметры

. Пусть

и

тогда

оптимизируется следующим образом:

, где

- коэффициент интерполяции между

и

. Таким образом, с=0 соответствует

, так что

; c=1 соответствует

, так что

.To solve this problem, a separate neural network

train to approximate local dynamic parameters

. Let be

And

then

is optimized as follows:

, where

- interpolation factor between

And

. So c=0 corresponds to

, so

; c=1 matches

, so

.

и относительной потери направления

. Общий критерий оптимизации:

.

обучается с ADAM до схождения. Во время тестирования сеть

позволяет сделать выборку случайной цели

и обновить

в соответствии с ней путем увеличения коэффициента интерполяции

по мере продвижения анимации.[0028] This is implemented by the combination of loss L1

and relative loss of direction

. General optimization criterion:

.

trains with ADAM until convergence. During testing, the network

allows you to select a random target

and update

according to it by increasing the interpolation factor

as the animation progresses.

). Для этого анимации придается суперразрешение, и она смешивается с исходным изображением с помощью стандартного метода получения суперразрешения изображения. Можно использовать ESRGANx4, обученный на специальном наборе данных, который генерируется следующим образом. Чтобы получить пару (высокое разрешение, низкое разрешение), берется кадр

из набора данных видео в качестве изображения с высоким разрешением, затем кадр подвергается субдискретизации, выполняются первые два этапа логического вывода и получается изображение с низким разрешением (неидеальное). Таким образом, сеть обучается на более сложной задаче, чем суперразрешение.[0029] Super resolution (SR). Since the models are trained at a medium resolution (eg 256x256), fine details can be obtained from a given image to be animated using a separate super-resolution procedure. The main idea of the super resolution method is to borrow as much as possible from the original high resolution image (which is subsampled for animation with

). This animation is super-resised and blended with the original image using the standard image super-resizing method. You can use ESRGANx4 trained on a custom dataset that is generated as follows. To get a pair (high resolution, low resolution), a frame is taken

from the video dataset as a high-resolution image, then the frame is down-sampled, the first two steps of inference are performed, and a low-resolution (non-ideal) image is obtained. Thus, the network is trained on a more complex task than super resolution.

[0030] After receiving the super-resolution video, the dynamic parts (sky and water) are transferred from it to the final result. The static parts are obtained by running a guided filter on super-resolution frames, using an input high-resolution image as a guide. Such a procedure effectively captures high-resolution detail from the input while preserving the lighting change caused by the lighting manipulation.

[0031] At least one of a plurality of modules, nodes, components, steps, sub-steps can be implemented in an artificial intelligence (AI) model). An AI-related function can be performed using non-volatile memory, volatile memory, and a processor. The processor may include one or more processors. Here, one or more processors may be a general purpose processor such as a central processing unit (CPU), an application processor (AP) or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual information processor (VIP) and/or a dedicated AI processor such as a Neural Processor (NP). One or more processors direct the processing of input data in accordance with a predetermined operating rule or artificial intelligence (AI) model stored in non-volatile memory and non-volatile memory. A predetermined operating rule or artificial intelligence model is provided through training. In this context, provision by training means that a predetermined operating rule or AI model with a desired performance is generated by applying a learning algorithm to a set of training data. The training may be performed on the device itself in which the AI is implemented according to the embodiment and/or may be implemented via a separate server/system.

[0032] An AI model may be composed of multiple neural network layers. Each layer has a set of weights and does the work of the layer by calculating the previous layer and operating on the set of weights. Examples of neural networks include, but are not limited to, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN) , generative adversarial networks (GANs), and deep Q-nets. The learning algorithm is a method of training a predetermined target device (eg, a robot) using a plurality of training data to induce, enable, or control the target device to perform a determination or prediction. Examples of learning algorithms include, without limitation, supervised learning, unsupervised learning, partially supervised learning, or reinforcement learning.

[0033] It should be clearly understood that it is not necessary to use all the technical effects mentioned in this document, in each and every embodiment of the present method. For example, embodiments may be implemented without the user using some of these technical effects, and other embodiments may be implemented with or without other technical effects.

[0034] Modifications and improvements to the implementations of the proposed method described above may be apparent to those skilled in the art. The above description is intended to be illustrative rather than limiting. Thus, the scope of the present invention is limited only by the scope of the appended claims.

[0035] While the above embodiments have been described and shown with reference to specific steps performed in a particular order, it should be understood that these steps can be combined, subdivided, or reordered without departing from the scope of the present invention. Accordingly, the order and grouping of steps is not a limitation of the proposed method.

Claims (25)

1. A method for generating wallpapers locally on a computing device and displaying these wallpapers on this computing device, which consists in the following: generating (S105) on the computing device at least one first wallpaper of the wallpaper stream using a deep generative neural network, wherein the deep generative neural network is trained on a collection of high-quality images/videos and loaded into the computing device in advance, and setting (S110) on the computing device at least one first wallpaper as the wallpaper of this computing device, wherein the method further comprises determining the current positions and colors of icons and widgets on the device screen and using the determined current positions and colors of icons and widgets as additional parameters of the deep generative neural network when generating wallpapers. 2. The method of claim. 1, further comprising the following steps, performed when the condition (S115) is met: generating (S120) on the computing device at least one second wallpaper of the wallpaper stream using a deep generative neural network, the at least one second wallpaper being different from the at least one first wallpaper; setting (S125) on the computing device at least one second wallpaper as the wallpaper of this computing device. 3. The method of claim 2, wherein said condition includes one or more of the following conditions: a user input is received on the computing device, the user input representing whether or not the user of the computing device disapproves of the wallpaper currently set as the wallpaper of the computing device; the preset period of time has expired; the GPS location of the computing device has changed, wherein the GPS location of the computing device is registered by a GPS unit contained in the computing device. 4. The method of claim 1, further comprising one or both of the following steps: individualizing (S95) the deep generative neural network for a user of the computing device using random input as a parameter of the deep generative neural network, thereby ensuring that the deep generative neural network is configured to generate unique wallpapers for the given user of the computing device; personalizing (S100) the deep generative neural network for a user of the computing device, thereby ensuring that the wallpaper generated by the deep generative neural network is personalized for that user, the personalization being based on one or more of the following personalization parameters: one or more user preferences, one or more user inputs, one or more computing device settings, current time of day, current season, current GPS location of the computing device, user gallery content currently stored on the computing device, browser history content currently stored on the computing device, current weather and weather forecast, location and colors of icons and widgets on the device screen. 5. The method according to claim 1 or 2, wherein the wallpaper generated by the deep generative neural network is static, dynamic, or interactive. 6. The method of claim. 1, in which the deep generative neural network has one or more of the following variables: vector variables, latent variables having the form of a two-dimensional matrix or a set of two-dimensional matrices. 7. The method of claim 6 wherein the latent variables are extracted from unit normal distributions. 8. The method according to any one of paragraphs. 4 or 6, in which the personalization is performed by a separate encoder network trained to map the personalization parameters to the parameters of the spatial distributions of the latent variables, such as the mean and covariance of the Gaussian normal distribution, from which the latent variables are extracted. 9. The method according to claim 1 or 2, wherein the step of generating at least one wallpaper further includes the steps of image synthesis and image modification, and the step of setting at least one wallpaper as the wallpaper of the computing device further comprises the step of setting the modified image as at least one first wallpaper. 10. The method according to claim 1 or 2, wherein the step of generating at least one wallpaper further comprises the steps of image synthesis and image animation, and the step of setting at least one wallpaper as the wallpaper of the computing device further comprises the step of setting the animated image as the at least one first wallpaper. 11. The method of claim 6, wherein the deep generative neural network is trained using an adversarial learning process along with one or more discriminant networks. 12. A computing device (200) comprising a processor (205) and a storage device (210) storing a trained deep generative neural network (210.1) for generating wallpaper locally on the computing device, the processor being configured to implement the method of any one of claims. 1-11. 13. Computing device (200) according to claim 12, characterized in that the computing device (200) is a user computing device, which is one of the following devices: smartphone, tablet, laptop, laptop, smart TV.

Images