KR20220004525A - Electronic device and controlling method of electronic device - Google Patents

Abstract

The present disclosure refers to a method of creating content in a computing-environment based on an artificial neural network (ANN) such as a generative adversarial network (GAN). The method according to the present disclosure includes a step of receiving external inputs for creating content by a GAN, wherein the GAN is configured to operate on a plurality of target domain attributes (TDA) for a target domain. The content corresponds to media or multimedia. Thereafter, the TDAs are shortlisted from the TDAs based on at least one of the external inputs and one or more clusters associated with the TDAs. The data is interpolated within a latent space defined by the representation of the shortlisted TDA, wherein a direction of interpolation is determined based on (i) a sampling vector computed based on one of (a) the external-inputs and (b) the at least one clusters; and/or (ii) automatically learned relationships within the latent codes in the latent space to predict latent codes.

Description

ELECTRONIC DEVICE AND CONTROLLING METHOD OF ELECTRONIC DEVICE

The present disclosure relates to an electronic device and a control method of the electronic device, and more particularly, to an electronic device configured to render a change in content generated based on an interaction with a user, and a control method of the electronic device.

Aggregators that research and sell products through e-portals have become very common, eliminating the need for brick and mortar stores. Accordingly, the customer can search and select the item of their choice from the comfort of their home. However, despite all the bandwidth and resources, e-commerce websites, such as brick-and-mortar stores, end up presenting a limited number of items as part of their inventory. Accordingly, it is common for customers to be disappointed or dissatisfied with the electronic inventory available.

Even if the products or content items displayed on the screen are very diverse, the quantity is difficult to meet the qualitative expectations. At least the reason it can be attributed is the lack of user-based personalization that leads to a discrepancy between the displayed product and the customer's needs.

Also, the products displayed on the screen are classified or selected only by a fixed number of product-specific options. For example, the selection of clothes is allowed only by options such as color and texture. However, a user who wants to purchase clothes may not be a beneficiary because some parameters that are important for a user to purchase a product are not displayed on the screen. For example, temperature, clothing location, etc. are not usually provided as options displayed on the screen. In another example, the information displayed on the screen regarding available products is incomplete and does not include additional information, for example, as to whether clothes may discolor when washed. Accordingly, the consumer makes unnecessary assumptions while purchasing the item. If not satisfied, the consumer returns the product. Ultimately, this leads to monetary losses incurred during the delivery of the requested product.

At least one of the exemplary underlying phenomena that may arise in online product creation is the excessive complexity of the image creation model. A phenomenon based on exemplary content-generating artificial intelligence models (AI models) and artificial neural networks (ANNs) is a generative adversarial network (GAN) that enables the creation of an infinite variety of product images. )to be. However, as indicated above, boredom of searching endless inventories is prevalent among customers. The least possible reason is that the creation process is simply random. There is no mechanism to know if a customer will like a particular product. As a result, the process must go on indefinitely.

Likewise, in any multimedia content creation system, the content items (audio, video, etc.) created during the creation process depend more heavily on the dynamics of the underlying technology, and generally cannot take human preferences into account. As a result, the final product or media item is usually created mechanically, requiring many iterations to reveal user preferences in the final result.

Overall, current content creation systems lack the appeal of user personalization, despite the use of AI for content creation. There has been a long gap between machines and humans. Neural networks learn complex patterns, but do not know which patterns relate to humans. On the other hand, a human can “see” the pattern and select the desired pattern. However, it certainly cannot quickly create new things in mind at the speed of an electronic device or machine. That is, humans cannot electronically explain their needs.

For example, the neural networks that form part of AI learn complex patterns such as shapes, poses, textures, etc. that humans can interpret differently. However, these concepts cannot be labeled as ground truth by humans in numeric format. Consequently, machine learning techniques (eg, supervised techniques) rely on humans to extensively annotate simpler concepts while devising ground truth. This is because shapes and poses are concepts that humans understand. However, on the other hand, humans cannot generate numerical vectors based on provided labels, for example, by hot-encoding, forcing support by machines (i.e. neural networks) for that effect. should get

There is a need to create content creation models on the fly so that machines can learn using human-specific biases to generate human-related patterns.

Overall, there is a need to contribute to the human-computer interface (HCI) as an attempt to unify humans and machines.

The present disclosure is in accordance with the above-described needs, and an object of the present disclosure is to provide an electronic device configured to render a change in content generated based on an interaction with a user, and a control method of the electronic device.

This Summary is provided to introduce a selection of concepts in a simplified format that are further described in the Detailed Description. This Summary is not intended to identify key or essential inventive concepts, nor is it intended to determine the scope of the present disclosure.

The present disclosure relates to a method for generating content in a computing environment based on a generative adversarial network (GAN). The method includes receiving an external input for creation of content by a generative adversarial network (GAN), wherein the GAN operates on a plurality of target domain attributes (TDA) for a target domain. is configured to Content corresponds to media or multimedia. Thereafter, the plurality of TDAs are shortlisted from the plurality of TDAs based on at least one of the external inputs and one or more clusters associated with the plurality of TDAs. Data are interpolated within a latent space defined by the representation of the short-listed TDA, wherein the direction of interpolation is (i) (a) an out-input and (b) one of the one or more clusters. a sampling vector calculated based on at least one; and/or (ii) based on automatically learned relationships within latent codes in the latent space to predict latent codes. At least one latent code is generated based on the interpolation along a determined direction in latent space. Accordingly, at least one content item is generated by the GAN based at least on the latent code.

In another embodiment, the present disclosure provides a method of generating an image in a computing environment based on artificial intelligence (AI)-based technology. The method includes receiving an external input for image generation by an artificial neural network (ANN). The ANN is configured to operate on a plurality of target domain attributes (TDAs) for the target domain. A plurality of target domain attributes (TDAs) are short-listed from a plurality of TDAs based on at least one of the external inputs and one or more clusters associated with the plurality of TDAs. Data interpolation is performed within a latent space defined by the representation of the shortlisted TDA, wherein the direction of the interpolation is calculated based on (i) (a) an external input and (b) at least one of the one or more clusters. sampling vector; and/or (i) based on automatically learned relationships within latent codes in the latent space to predict latent codes. At least one latent code is generated based on the interpolation along a direction determined in latent space, and at least one image is generated by the ANN based on at least the latent code.

In order to make the advantages and features of the present disclosure more clear, a more specific description of the present disclosure will be given with reference to specific embodiments shown in the accompanying drawings. It is to be understood that these drawings illustrate only typical embodiments of the present disclosure and are not to be considered limiting of their scope. The present disclosure will be described and explained in further detail and detail in conjunction with the accompanying drawings.

The various features, aspects and advantages of the present disclosure will be better understood upon reading the following detailed description with reference to the accompanying drawings in which like characters indicate like parts throughout the drawings.
1A is a view for briefly explaining a method of controlling an electronic device according to an embodiment of the present disclosure;
1B is a view showing each step of a control method according to an embodiment of the present disclosure;
Figure 2 is a view for explaining the implementation of each step of Figure 1b according to an embodiment of the present disclosure;
3 is an exemplary control flow diagram illustrating a sub-process according to an embodiment of the present disclosure;
4 is another exemplary control flow diagram illustrating a sub-process according to an embodiment of the present disclosure;
5 is another exemplary control flow diagram illustrating a sub-process according to an embodiment of the present disclosure;
6 is another exemplary control flow diagram illustrating a sub-process according to an embodiment of the present disclosure;
7 is another exemplary control flow diagram illustrating a sub-process according to an embodiment of the present disclosure;
8 is a diagram illustrating an exemplary implementation of the sub-process of FIG. 7 in accordance with an embodiment of the present disclosure;
9 is another exemplary control flow diagram illustrating a sub-process according to an embodiment of the present disclosure;
10 is another exemplary control flow diagram illustrating a sub-process according to an embodiment of the present disclosure;
11 is a diagram illustrating an example implementation according to an embodiment of the present disclosure;
12 is a diagram illustrating another example implementation according to an embodiment of the present disclosure;
13 is a diagram illustrating another system architecture implementing various modules and sub-modules in accordance with the disclosure;
14 is a diagram illustrating a computing device-based implementation according to an embodiment of the present disclosure;
15 is a diagram illustrating an example implementation for specifying a result according to an embodiment of the present disclosure;
16 is a diagram illustrating an example implementation for specifying a result according to an embodiment of the present disclosure;
17 is a diagram illustrating an exemplary implementation for specifying a result according to an embodiment of the present disclosure;
18 is a diagram illustrating an exemplary implementation for specifying a result according to an embodiment of the present disclosure.
Additionally, those skilled in the art will appreciate that elements in the figures are shown for simplicity and may not necessarily be drawn to scale. For example, a flowchart represents a method in terms of the most salient steps to help improve understanding of aspects of the present disclosure. Further, from the point of view of the configuration of the device, one or more components of the device may have been represented by conventional reference numerals in the drawings, which are details that will be readily apparent to those skilled in the art having the benefit of the description herein. In order not to obscure the drawings with details, only specific details suitable for understanding an embodiment according to the present disclosure may be shown.

Although exemplary implementations of embodiments of the present disclosure are shown below, it should be understood from the outset that the present disclosure may be implemented using any of a variety of techniques, whether presently known or existing. This disclosure is not to be limited in any way to the exemplary implementations, drawings and descriptions shown below, including the exemplary designs and implementations shown and described herein, but to be modified within the scope of the appended claims together with the full scope of equivalents. can

As used herein, the term “some” is defined as “none, or one, or more than one, or all.” Accordingly, the terms "none", "one", "more than one", "more than one but not all" or "all" shall all fall within the definition of "some". The term “some embodiments” may refer to no embodiments, one embodiment, some embodiments, or all embodiments. Accordingly, the term “some embodiments” is defined to mean “no embodiments, one embodiment, or more than one embodiment, or all embodiments”.

The terminology and structure employed herein is for the purpose of describing, teaching, and clarifying some embodiments and specific features and elements thereof, and does not limit, limit, or diminish the spirit and scope of the claims or their equivalents.

More specifically, any term used herein as such, but not limited to "include", "comprise", "has", "consist" and grammatical variations thereof does not specify exact limitations or limitations, unless otherwise specified, does not explicitly exclude the possible addition of one or more features or elements, and also "MUST comprise" or "NEEDS TO include." It should not be construed as excluding the possible removal of one or more of the listed features and elements, unless otherwise specified in limiting language such as ".

Regardless of whether a particular feature or element is limited to use only once, it may still be referred to in any way as "one or more feature" or "one or more element" or "at least one feature" or "at least one element". have. Also, use of the term "one or more" or "at least one" feature or element is not otherwise specified in the limiting language, such as "needs to have one or more..." or "requires one or more elements." However, the absence of such features or elements is not excluded.

Unless defined otherwise, all terms used herein, and particularly any technical and/or scientific terms, are to be understood to have the same meaning as commonly understood by one of ordinary skill in the art.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1A is a diagram for briefly explaining a method of controlling an electronic device according to an embodiment of the present disclosure.

As shown in FIG. 1A , the electronic device according to the present disclosure may receive a first user input for acquiring an image ( S101 ).

An 'electronic device' according to the present disclosure is a device capable of acquiring an image using a neural network model, and any device configured to perform each step of a control method as described below is an electronic device according to the present disclosure regardless of the type It may correspond to the device. For example, the electronic device may be implemented in various types, such as a smart phone, a tablet PC, a notebook computer, and a digital TV. According to an embodiment, the electronic device is a server for providing a web site for selling a product, and may be configured to generate various images of a product using a neural network model.

A 'neural network model' according to the present disclosure refers to an artificial intelligence model including an artificial neural network, and may be learned by deep learning. In particular, the neural network model according to the present disclosure may be a generative adversarial network (GAN) for generating an image.

The 'first user input' refers to a user input for obtaining an image according to a user's request, for example, a user touch input through a display of the electronic device, a user voice received through a microphone of the electronic device, or an electronic device It may be received based on an input of a physical button provided in the , a control signal transmitted by a remote control device for controlling an electronic device, and the like.

When the first user input is received, the electronic device may identify a domain corresponding to the first user input from among a plurality of predefined domains to classify images ( S103 ).

The term 'plural domains' is a pre-defined classification criterion for classifying images that can be generated by a neural network model, and may be replaced with terms such as category or class. For example, when the electronic device according to the present disclosure is implemented as a server for providing a website for selling products, and the neural network model is implemented to generate and output images of various types of products, a plurality of The domain may include domains such as “jacket”, “plate” and “sofa” according to the type of product.

Identifying the domain corresponding to the first user input means identifying the selected domain when the first user input includes information for selecting a specific domain from among a plurality of domains. For example, when the first user input includes information for selecting a product called “jacket”, the electronic device may identify a domain called “jacket” as a domain corresponding to the first user input. Meanwhile, in describing the present disclosure, the domain identified as corresponding to the first user input may be replaced with the term 'target domain'.

The electronic device identifies whether information related to at least one domain property among a plurality of predefined domain properties is included in the first user input in order to distinguish properties of images corresponding to the identified domains, and at least according to the identification result At least one domain attribute for acquiring one image may be identified.

The 'domain attribute' is a pre-defined detailed classification criterion to classify the attributes of images for each domain, and it can be said that it is a sub-concept for classifying the upper concept of a domain. For example, the domain attribute for the domain "jacket" may include "color", "material", "brand", etc. which are detailed classification criteria for classifying images of various types of jackets. In describing the present disclosure, the term 'domain attribute' may be replaced with the term 'target domain attribute (TDA)'.

The 'domain attribute-related information' is used as a general term for information corresponding to a predefined domain attribute among information included in the first user input. Specifically, the information related to the domain attribute may include at least one of direct information and first indirect information.

Here, 'direct information' refers to information for directly selecting at least one domain attribute from among a plurality of predefined domain attributes. For example, when information “red jacket” is included in the first user input, the information “red” may be direct information for selecting a color “red” from among a plurality of domain attributes.

The 'first indirect information' is not information for directly selecting at least one domain property from among a plurality of predefined domain properties, but information related to at least one domain property, which can be mapped to at least one domain property say information For example, when information “a jacket to wear in Russia” is included in the first user input, the information “Russia” may be first indirect information for selecting a material called “thick material” among a plurality of domain properties. . In describing the present disclosure, the term 'first indirect information' is distinguished from 'second indirect information' as described below, and may be replaced with the term 'source domain attribute (SDA)'.

As described above, when it is identified whether information related to at least one domain attribute is included in the first user input, the electronic device may identify a domain attribute for acquiring at least one image according to the identification result.

Specifically, if information related to at least one domain attribute is not included in the first user input (S105-N), the electronic device may acquire at least one image corresponding to a plurality of domain attributes through the neural network model ( S107-1). That is, if the first user input does not include information related to at least one domain property, the electronic device randomly combines all of the plurality of domain properties defined in order to classify images corresponding to the identified domain to obtain at least one domain property. You can create an image.

On the other hand, when information related to at least one domain attribute is included in the first user input (S105-Y), the electronic device receives at least one image included in the domain identified through the neural network model and corresponding to the at least one domain attribute. can be obtained (S107-2). That is, when the first user input includes information related to at least one domain property, the electronic device does not randomly combine all of the plurality of domain properties, but at least one image based on the information related to at least one domain property. At least one domain attribute for obtaining , may be identified, and at least one image may be generated by combining only the identified at least one domain attribute.

Specifically, when the first indirect information is included in the first user input, the electronic device may map the first indirect information to a domain property preset as corresponding to the first indirect information. In addition, the electronic device identifies at least one domain property including a domain property corresponding to the direct information and a domain property mapped to correspond to the first indirect information, and adds to the at least one domain property identified using the neural network model. At least one corresponding image may be acquired.

For example, if the first user input includes information “a red jacket to wear in Russia”, the electronic device may display “temperature in Russia”, “altitude in Russia”, “ It is possible to obtain information such as "wind speed in Russia" and "weather forecast in Russia", and map the first indirect information "Russia" to the domain attribute "thick material" based on the obtained information. And, the electronic device corresponds to at least one domain attribute including a domain attribute of “red” corresponding to the direct information of “red” and a domain attribute of “thick material” corresponding to the first indirect information of “Russia”. At least one image may be acquired.

When at least one image is obtained as described above, the electronic device may provide the at least one obtained image (S109). Specifically, the electronic device may display the acquired image on the display of the electronic device, or transmit it to an external device connected to the electronic device to be displayed on the display of the external device.

According to the above-described embodiment, the electronic device not only belongs to the domain of the image that the user wants to obtain based on the user input, but also generates images having the domain attribute desired by the user and provides the image to the user through the neural network model. can provide In particular, even when the user input includes information indirectly related to the domain property (first indirect information), the electronic device maps the indirect information to the domain property to provide the user with an image having a detailed property desired by the user. be able to provide

In particular, when the electronic device according to the present disclosure is implemented as a server for providing a website for selling products, and the neural network model is implemented to generate and output images of various types of products, the electronic device is a user An image of a product having an attribute that the user wants to purchase may be newly created and provided to the user, and thus user convenience may be remarkably improved.

Meanwhile, according to an embodiment of the present disclosure, the electronic device may check whether at least one image obtained through the above-described process conforms to the user's intention.

Specifically, when at least one image is obtained, the electronic device may obtain second indirect information related to the at least one domain attribute from the at least one image. Here, the 'second indirect information' is not information for directly selecting at least one domain property from among a plurality of predefined domain properties like the first indirect information, but information related to at least one domain property and includes at least one Refers to information that can be mapped to domain properties of However, the first indirect information is to refer to information included in the first user input, whereas the second indirect information is to refer to information obtained from at least one image obtained through a neural network model. Sleep is distinct.

When the second indirect information matches the first indirect information, the electronic device may provide at least one image. That is, as a result of obtaining at least one image based on the first indirect information included in the first user input, the second indirect information obtained from the at least one image is combined with the first indirect information included in the first user input If matching, it may be considered that an image matching the user's intention has been obtained, and thus the electronic device may provide the obtained at least one image to the user.

On the other hand, if the second indirect information does not match the first indirect information, the electronic device may retrain the neural network model. Here, retraining the neural network model may include adjusting a domain property corresponding to the first indirect information so that the second indirect information can match the first indirect information. That is, as a result of obtaining at least one image based on the first indirect information included in the first user input, the second indirect information obtained from the at least one image is combined with the first indirect information included in the first user input If not, it cannot be considered that an image matching the user's intention has been obtained, so the electronic device may obtain an image matching the user's intention by adjusting the domain property corresponding to the first indirect information to another domain property. have.

Meanwhile, according to an embodiment of the present disclosure, the electronic device may reflect the user's feedback on at least one image obtained through the above-described process.

Specifically, when the second user input including the user's feedback information on the at least one image provided to the user through the process as described above is received, the electronic device adjusts at least one domain property based on the feedback information, , at least one image corresponding to the adjusted at least one domain attribute may be acquired.

Here, the 'feedback information' may include positive feedback information on the first image among the at least one image and negative feedback information on the second image among the at least one image. In this case, the adjusted at least one domain property may include one or more domain properties excluding a plurality of domain properties corresponding to the second image among a plurality of domain properties corresponding to the first image.

In the above, an embodiment of the present disclosure has been briefly described with reference to FIG. 1A. Hereinafter, specific methods for technically implementing various embodiments of the present disclosure including the above-described embodiments will be described.

1B is a diagram illustrating each step of a control method according to an embodiment of the present disclosure. A control method according to the present disclosure includes an image generation method in a computing environment based on artificial intelligence (AI)-based technology. As an example, an artificial neural network (ANN) refers to a generative adversarial network (GAN) for image generation.

The method includes receiving an external input for image generation by the artificial neural network (S102). The ANN may be configured to operate in relation to a plurality of target domain attributes (TDAs) for the target domain. An ANN configured to generate an image in the target domain is defined by a plurality of characteristics, such as a distinct TDA representation for rendering a starting point in latent space for content or image generation. The plurality of initialized vectors are defined by at least one of a cluster sampling vector (CSV) and a cluster preference vector (CPV).

In one embodiment, receiving the external input includes electronically or acoustically receiving one or more user labels to generate an image, wherein the user labels optionally carry one or more source domain attributes. User labels are mapped to one or more target domain attributes to facilitate the shortlisting of TDAs, while one or more accompanying source domain attributes are mapped to shortlisted TDAs. In another embodiment, receiving the external input comprises receiving the external input as an automatically generated trigger based on prediction of an intermediate label in the latent space by the machine based on existing latent codes. Based on this, a plurality of TDAs are distinguished based on the predicted mid-labels until attainment of a threshold defined by the previously provided user feedback.

In one embodiment, based on the external input, one or more source domain attributes are filtered based on criteria defined by a combination of causality criteria and correlation criteria. The filtered source domain attributes are modeled as multiple TDAs.

The method also includes short listing ( S104 ) a plurality of TDAs from a plurality of TDAs based on at least one of the external inputs and one or more clusters associated with the plurality of TDAs. Shortlisting the TDA further includes identifying one or more clusters within the latent space associated with the shortlisted TDA and identifying one or more cluster preference vectors (CPV) based thereon.

A user-preference vector (UPV) is computed based on received external input including a user label and optionally source domain information. A plurality of short-listed TDAs associated with an external input are ranked by combining a cluster preference vector (CPV) and a UPV. A ranked and shortlisted TDA and one or more combinations thereof are defined to initiate interpolation within the latent space.

The method also includes interpolating the data within the latent space defined by the representation of the short-listed TDA (S 106). The direction of interpolation is determined based on a sampling vector computed based on (a) an external input and (b) at least one of the one or more clusters. The operation of the current sampling vector is based on the derivation of a first vector as a user sampling vector (USV) from an external input. A second vector is derived by performing a cluster preference vector (CSV) from the cluster associated with the short-listed TDA. The first vector and the second vector are combined to result in a current-sampling vector. In another example, the operation includes computing a USV from an external input and a mapping drawn between a source domain attribute and a target domain attribute. The USV and the sampling vector of the identified cluster are combined to provide the resulting sampling vector. Based on the resulting sampling vector, one or more clusters are updated based on the resulting sampling vector. One or more clusters are defined by a cluster comprising one or more clusters linked to at least one of the short-listed TDA and the short-listed TDA.

In one embodiment, the direction of interpolation is defined based on a statistical running average of a sampling vector that is historically computed based on user input and the one or more clusters. Specifically, the logged sampling vector is aggregated together with the currently computed sampling vector to become an aggregated vector through weighted averaging. The change pattern between existing latent codes is based on the automatically learned relationship in the latent space through a neural network. The aggregated direction is derived based on the aggregated vector and the direction associated with the calculated pattern, thereby becoming a federated update to the interpolation direction.

Further, the method includes generating at least one latent code based on the interpolation along the determined direction in the latent space ( S108 ). A magnitude and direction associated with the resulting sampling vector are determined, and a plurality of latent codes are generated in the latent space by interpolating along the direction of the resulting sampling vector based on the determined magnitude. Latent-space interpolation based on an external input includes decomposing the resulting sampling vector into a unit directional vector and a corresponding magnitude, and generating a plurality of latent codes along the unit directional vector based on one or more magnitudes.

In one embodiment, the orientation of the latent space is determined based on auto-learned relationships within the latent code of the latent space to predict the latent code. Interpolation of the latent space based on auto-learned relationships within the latent space includes retrieving, through a subspace defined by the shortlisted TDA, one or more latent codes configured to generate an image in the target domain. In addition, the neural network is trained to compute the relationships between the latent codes in the subspace, and thereby can predict additional latent codes based on the computed relationships.

Based on the generated latent code, at least one image is generated by the ANN (S 110). In addition to generating the image, the further received external input includes receiving a plurality of user feedback regarding the generated image as part of the latent spatial interpolation. Within the interpolated latent space, the optimal vector value for a user for a particular combination of target domain attributes is based on an optimal user sampling vector (USV) based on the associated mean distance for positive feedback and the relationship between positive and negative feedback is calculated as one or more of the obtained optimal user preference vectors (UPV).

FIG. 2 is a diagram illustrating a training step of an AI model-based ANN according to an embodiment of the present disclosure, and corresponds to step S104 of FIG. 1B .

Step S202 represents the collection of a plurality of user labels for each image of the target domain.

Step S204 represents, for example, conversion of the label into a numeric format performed through one hot-encoding. Subsequently, user-labels are aggregated into numerical feature vectors.

Step S206 represents obtaining a feature vector based on the numerical feature vector of step S204. The number of conditional labels (attributes) of the target domain is predefined. Corresponding to each target domain attribute, a feature vector is obtained from the neural network.

Step S208 represents selectively masking the feature vector output in step S206 to isolate a specific number of target attributes related to image formation. For example, from N target attributes, 2 ^N combinations are possible.

Step S210 represents a step (S210a) of training an embedding layer forming a part of the neural network to project ^{2N combinations into a common subspace.} Considering this example, in the case of N TDAs, there are N rows selected from ^{2 N combinations.} Since an embedding layer corresponds to an embedding matrix of rows and columns, the length of each row represents at least a dimension of a common subspace. The output feature map is supplied as input to an image generator 201 forming part of a generative adversarial network (GAN) (step S210b).

Step S212 represents the operation on the discriminator side, and reconstruction-loss helps to reproduce the target domain attribute.

Step S214 represents reproduction of a plurality of sets of target domain attributes with equal accuracy. To isolate or shortlist one of these sets, the user label is reconstructed by extending the reconstruction loss to the user label.

Overall, the steps described above continue, and training is performed until the best representation of the relevant target domain attribute is revealed. This forms the property that the latent space is changed accordingly to create a user experience. In addition, the trained GAN obtained may generate an image or any other multimedia content based on the latent code forming part of the latent space.

As can be understood, the artificial intelligence AI model obtained through training is based on a plurality of pieces of training data by means of a training technique or a predefined operating rule configured to perform a desired characteristic (or purpose) or an artificial intelligence model. It means that it is obtained by training an artificial intelligence model. An artificial intelligence model may include multiple neural network layers. Each of the plurality of neural network layers includes a plurality of weights, and a neural network operation is performed by an operation between the operation result by the previous layer and the plurality of weights.

3 is a diagram illustrating another training step of an AI model-based ANN according to an embodiment of the present disclosure, and corresponds to step S104 of FIG. 1B .

In one embodiment, random noise Z obtained via a probability distribution function (PDF) is processed as a segmentation space 302 to generate an image. The generative architecture 201 typically takes a random noise vector as input and produces a change in the target domain. However, random sampling from an unknown probability distribution cannot learn the representation directly responsible for interpolation. Thus, in accordance with the present disclosure, input noise is fed to a neural network 304 that is transformed into a discrete space associated with complex level features. Based on this, the input is rendered into a generative model, which in turn provides a representation directly responsible for interpolation and image generation during the training phase as well as the inference phase.

4 is a diagram illustrating mapping of a user input to a target domain attribute according to an embodiment of the present disclosure, and corresponds to step 102 of FIG. 1B .

Step S402 represents reception of an external user input with or without a source domain (SD) attribute. When receiving the SD attribute from the user in step S402a, the control flow proceeds to step S404. Otherwise, if the SD attribute is not received in step S402b, the external input is considered as a user request for direct personalization in the target domain. In this scenario, the flow proceeds to step S406.

In step S404, the source domain is mapped with the target domain attribute (TDA) based on the description of FIG. 5 . Upon identification of the corresponding TDA, a vector is computed for each TDA. The vectors are normalized such that each vector is a unit vector and this set of unit vectors is considered a user sampling vector (USV) expressed in S step 410 .

In step S406a, the clustering logic assigns the first user associated with the external input to a specific cluster in the latent space. However, if the user is already associated with the cluster, the control flow moves to step S406b.

In step S406b, a predefined sampling vector associated with the cluster associated with the user (providing an external input) is fetched.

In step S408, the provided user input is converted into a vector through, for example, hot encoding. Based thereon, based on the vector fetched in step S406, an equivalent vector is calculated and regarded as a user sampling vector (USV) as expressed in step S410.

5 is a diagram illustrating mapping of a source-domain to a target-domain attribute according to an embodiment of the present disclosure, and corresponds to step S102 of FIG. 1B .

Step S502 corresponds to a filtering process defined by causality filtering performed by a human entity by providing a target domain attribute to a source domain attribute linked to his/her knowledge. As part of the correlation-mechanism, a correlation between a source domain and a target domain is established by fitting hypotheses to (X, Y) data points in the domain. Source domains that yield an accuracy greater than a certain threshold (ie, 1 to n) are filtered or shortlisted from the inputted source domains 1 to k. By implementing the intersection of causality and correlation modules, the source domain exhibits a bijective nature. That is, a predicate relationship is achieved between representations of the source and target domains by diverting the basic pre-defined conditions for uncorrelated domain reconstruction.

Step S503 corresponds to establishing a relationship between the source domain and the target domain attribute. Once the source domain is selected, it becomes necessary to model it. In a general sense, each source domain maintains a one-to-many mapping with all target domain attributes. However, as the number of source domains increases, modeling this one-to-many relationship linearly increases the number of model parameters required. To solve this problem, a common neural network is obtained by combining an encoder and a decoder. A common network is trained to learn these relationships regardless of the number of source domains.

The mapping module shown in step S503 includes two components called a multi-mode encoder and a multi-mode decoder. As part of the operation of the mapping module, a data set or lookup table containing corresponding values of the source domain and the target domain is fetched as a precursor. The first phase of the operation of the mapping module is the training phase.

The encoder/decoder portion of the network includes a separate ANN network for each source domain. The source domain values are converted into a common target domain representation by the encoder via "Bootstrap aggregating" or Bagging, which enables later reconstruction at the decoder side. "Bagging" refers to an aggregator-network that learns to mix data from multiple modalities and project its output feature maps to the same common target domain representation step. The “amount” of information from each source domain to be blended at each level of the common network is maintained as a learnable parameter that is trained during the optimization phase of the training process.

After training, the “individual network” for each modality of the encoder can be removed, while the individual decoder for each source domain remains. This allows the network to reconfigure multiple multi-mode source domains when needed.

6 is a diagram illustrating generation of a target domain sample based on an external input according to an embodiment of the present invention, and corresponds to steps S106 and S108 of FIG. 1B .

Step S602 shows a state achieved when it has undergone a step corresponding to FIG. 4 . In another example, the current state in step S602 represents a default state of an ANN that may be deployed without training. At this stage, the trained ANN network or GAN has the following two functions:

(i) the creation of a variety of realistic images in the target domain; and

(ii) a distinct representation of the target domain attributes whose combination yields an external label provided to the constructor.

That is, in step S602, a condition in which the generative model is trained in the target domain and classification of the target domain attribute is performed indicates a condition. Combinations of target domain attributes condition the generator to produce transformations in smaller clusters or subspaces. The functions described above allow a trained or constructed ANN to generate subsamples in the target domain that best describe a particular combination of target domain attributes.

In step S604 a specific cluster of TDAs (as mentioned in step S602) is selected. The cluster is associated with the external-input provided by the user in step S402 of FIG. 4 . The external input may be a plain request by the user, for example "show me shirt design", thereby corresponding to a direct request for personalization in the target domain. In another example, the request may be accompanied by an SD attribute, and corresponds to an indirect personalization request of the target-domain. An example of indirect personalization with an SD attribute could be a request such as "Show me a shirt to wear in high temperatures in Africa."

In step S606, a sampling vector associated with the cluster selected in step S604 is selected.

In step S608, the user sampling vector obtained in step S412 of FIG. 4 is combined with the cluster sampling vector of step S606 to provide a final sampling vector. One or more clusters may be updated based on the resulting sampling vector, where such clusters are associated with the shortlisted TDA. The sampling vector is decomposed into a unit-direction vector and a corresponding magnitude. In an embodiment, the previously logged sampling vector may be aggregated together with the currently calculated sampling vector to generate the aggregated vector through weighted-averaging. A change pattern between existing latent codes is determined based on a relationship automatically learned in the latent space through a neural network. Accordingly, an aggregated interpolation direction may be derived based on the aggregated vector and the direction associated with the calculated pattern, thereby resulting in an associative update to the interpolation direction.

In step S610, the preference vector of one or more clusters selected in step S604 and the user preference vector calculated from the external input are combined to generate a rank of the TDA associated with the user. An appropriate number K of the N target domain attributes is selected. A random number of M target attributes are selected from among the K attributes. A distinct representation of the M attributes is maintained and the remaining N-K are masked to yield a label for the generator. As part of the validation drive, called the potential traversal preparation phase, the generative model is placed in an evaluation mode with conditioning applied.

In step S612, interpolation in the latent space is performed along the sampling direction determined from the resultant sampling vector in step S608 to generate a plurality of latent codes and corresponding images. The images are presented as arranged in FIG. 8 .

7 illustrates interpolation of data in a latent space for generating an image according to step S612 according to an embodiment of the present disclosure. As can be appreciated, a latent code corresponds to a point in the latent space that isolates a specific image generated by the GAN. Latent code formation is achieved by adding a sampling vector at the beginning of the latent space. Due to the distinct TDA representation, a separate sampling vector is modeled for each TDA.

Referring to FIG. 6 , a user/cluster sampling vector is combined to provide a final sampling vector. The sampling vector is decomposed into a magnitude corresponding to the unit direction vector. By keeping the magnitude constant, multiple latent codes along the direction of the sampling vector are obtained in the latent code generator 702 from the external input provided by the user reference step S402 of FIG. Each of these latent codes, along with features learned in a separate space, is sent to a generator to generate an image.

In another example, latent codes may be automatically generated without any external input based on predictions of intermediate labels in the latent space based on existing latent codes. For this prediction, the subspace defined by the shortlisted TDA is automatically searched to define one or more latent codes configured to generate an image in the target domain. Neural networks can be trained using neural networks to compute relationships between the latent codes in subspaces, thereby enabling prediction of additional latent codes based on the relationships computed. A plurality of TDAs are then discriminated based on the predicted mid-label until attainment of a threshold defined by user feedback. A short-listed TDA is obtained based on the segmented TDA to enable interpolation.

8 is a diagram illustrating a latent spatial interpolation for an exemplary user input according to the present disclosure, and corresponds to step S110 of FIG. 1B .

In one embodiment, the GAN may be trained by FIG. 3 in terms of “shirt” as the target domain. Through the training of Figure 3, the best factors such as shape, color and texture that directly affect shirt formation are identified. By the end of this step, the GAN achieves the ability to create a shirt for any user (ie Mr. Kim) given only numbers (ie, shirt, brown shirt, white shirt) as external input.

Based on FIG. 4 , an external input provided by a user (ie a request to generate a shirt image) is received in real time, which is mapped to a target domain. Also, as part of the indirect personalization offering, the user may also provide the source domain attribute by saying that he would like to purchase a shirt for a friend in Russia. Indirect SD factors such as Russian weather are mapped to direct TD factors such as shape, color, etc. identified during the training phase shown in FIGS. 2 and 3 by the mapping network of FIGS. 4 and 5 .

Accordingly, a plurality of images are generated based on data interpolation in the latent space as outputs. In the case of latent spatial interpolation, the decision to select a target domain attribute is made by the preference vectors mentioned in Figs. 6 and 7 which rank the target domain attributes in decreasing order of importance. Now, with respect to a specific target domain attribute, an important image along the latent space is determined based on the sampling vector. As can be appreciated, the sampling vector determines the direction in which the latent spatial interpolation is performed accordingly. Also, in addition to having a separate entity, each user also has a group identity, ie, can belong to a larger group. Thus, preference and sampling vectors are considered at both the user and cluster level.

Turning now to the representation shown in Figure 8, the same represents the concept of latent spatial interpolation for external input or user label "shirt". As shown, the interpolation may take into account multiple properties, such as texture and shape.

A preference vector regulates the decision as to which particular combination of characteristics is taken for a user. In one embodiment, the hierarchy or ranking of rows is performed according to a preference vector. Each ranked row represents which properties of the latent space change with interpolation, and thus target domain properties (eg, texture, shape).

With respect to a particular row, such as "shape," an infinite image can be created across the columns based on the sequence determined by the sampling vector. The sampling vector determines what the distance between any two consecutive images presented to the user should be. In another example, the row may also depict a TDA that is not readily interpretable by a human but can be easily deciphered by a machine. In an example, this particular TDA may be a combination of various TDAs, such as “shape + texture + color + temperature”.

Also, according to an embodiment, the current image may be ranked. In particular, when the interpolation stops, then the user is satisfied with the final image generated so that the image can be ranked. Thus, the column associated with the “liked” image is the final column and can be ranked such that the rows can be arranged in the desired sequence. The top row of the ranked column shows "liked" images. Images or rows below the hierarchy show decreasing order of similarity with favorite images. That is, the final rank column may simply be referred to as a result of iteratively performed interpolation, that is, “iterative interpolation output”.

Overall, as depicted in the description of Figures 2-7, the image creation process via user labels can be summarized as follows:

(i) Learning complex concepts through neural networks.

(ii) obtaining extensive user feedback on the concept;

(iii) A training example was constructed by mapping the user feedback and the internal feature vector of the complex concept in (i).

(iv) guiding further training of the network in an unsupervised/partially supervised manner until a certain accuracy threshold is reached.

This phenomenon also bridges the gap between neural networks and human representations, making it possible to label complex concepts. The description of Figures 2-7 represents a mapping of complex learned concepts to a wide range of user feedback. The following figures illustrate content or image creation based on iterative feedback.

9 is a diagram illustrating generation of target domain samples based on user feedback. Overall, FIG. 9 shows a mechanism of image generation driven by user feedback, thereby identifying relevant samples among variously generated images.

Step S902 represents image generation according to the latent spatial interpolation of FIGS. 7 and 8 .

Step S904 corresponds to obtaining a user's "like" or "dislike" for the currently generated image of the performed object. In the case of "dislike", the control flow proceeds to step S906, and in the case of "like", the control flow proceeds to step S908.

Step S906 represents the adjustment of the latent-code, thereby changing the interpolation variable to generate a new image through the further performed interpolation. By changing the sampling vector of one of the target domain attributes, a change in the target domain is provided to the user.

The control flow moves back to step S904 to confirm the user opinion. The backward and forward operation between steps S904 and S906 continues until the user is satisfied with the image generated in the target domain and the control flow moves to step S908.

In step S908, in case of "like" or positive feedback by the user, the sampling-distance employed to achieve the liked image is noted to compute an optimal sampling vector for the user for a particular combination of target domain attributes. More specifically, an optimal user sampling vector (USV) is determined based on the associated average distance with respect to positive feedback. For example, the average distance between any two positive feedbacks is used to compute an optimal sampling vector for the user for a particular combination of target domain attributes. In addition, an optimal user preference vector (UPV) is obtained based on the relationship between the positive feedback and the negative feedback. In an example, the ratio of positive feedback to negative feedback is used to obtain the user's preference vector for the attribute. In another example, the preference vector may also be obtained from any other degree of change associated with user feedback, ie, a first derivative, a second derivative. Overall, user satisfaction is calculated according to the percentage of positive responses gathered from user feedback. Further, positive feedback may be interpreted to indicate any feedback-based logic that provides an indication to the system described above to pause the creation process in the target-domain.

In another example, a statistical running average of historically computed sampling/preference vectors is also considered for calculation of the optimal vector. More specifically, the logged sampling vector is aggregated together with the currently computed sampling vector to generate an aggregated vector through weighted-averaging. The same can be applied to the calculation of the optimal preference vector.

In step S910, the optimal vector calculated in step S908 is stored.

In step S912, based on the optimal sampling vector, the TDA is updated, eventually leading to the update of the corresponding cluster.

Overall, target sample generation according to FIG. 9 allows direct mixing of TDA factors to form a shirt. The user may or may not like the formed shirt. With this iterative feedback, the system obtains the ideal amount of each direct factor that needs to be mixed. In addition, generation is made based on feedback received repeatedly, and personal data may not be required.

10 is a diagram illustrating reconstruction of a source domain attribute in which a target domain sample is used to reconstruct information of the source domain. The requirement for the reconstruction may be an optimal sample of the target domain generated through the differentiated TDA and user feedback of FIG. 9 . Thus, as part of the reconstruction step, the target domain samples are used to reconstruct the information of the source domain.

In step S1002, the optimal sampling vector for the user calculated in FIG. 9 in relation to the "liked" target domain sample is obtained. In another example, the computed optimal vector value specified for the user and the shortlisted TDA provide the aggregated feature vector.

In step S1004, it is checked whether the source domain information exists in the optimal vector. Otherwise, control may move to step S1006 where the source domain information may be attempted to be fetched from the name entity corresponding to the optimal vector and target domain sample. Once the source domain information is successfully fetched, control moves to step S1008. However, in case of fetching failure, control moves back to step S902 in Fig. 9, where the target domain sample is deformed and regenerated. If the changed target domain sample is “liked” in step S908, the control flow passes to step S1002.

In step S1008, information on each of the plurality of source domains is reconstructed based on the feature vector aggregated through the decoder forming a part of the common network as shown in FIG. 5 .

In step S1010, the reconstructed information is compared with the source domain information received in the user input to compute the efficiency of the short-listed TDA.

In step S1012, the efficiency can be found optimally. In this scenario, the clustering information associated with the preference vector (PV) and the sampling vector (SV) is updated.

In step S1010, the efficiency may also be found to be suboptimal. In this scenario, the short-listed TDA is updated to increase efficiency, and based on this, one or more of USV, UPV, CSV, CPV and at least one cluster associated with the TDA are updated. More specifically, step S1010 leads to transfer of control to FIG. 9 , and then decomposed into the following sub-steps.

a) receiving other user feedback with reference to step S904 to increase the efficiency;

b) updating the shortlisted TDA;

c) Updated cluster sampling vector and cluster preference vector after optimization of sampling vector for user for shortlisted TDA reference step S908. Optionally, updating of at least one cluster associated with one or more of the short-listed TDAs may be performed;

d) Passing the optimized vector to step S1002 to resume the procedure of FIG. 10 .

This diagram allows the user to predict the weather conditions in which the shirt may be worn, at least for a new configured "shirt" that the user likes. This is compared to the actual weather at the desired location and confirms that the result is useful to the user.

Overall, examples of real benefits emerging from the description of the preceding figures are:

[1] An infinite shirt can be created based on a combination of existing shirts in the inventory.

[2] Once the shirt is formed, user feedback is collected. This allows fine-tuning of the created shirt according to the user's preferences, a step beyond the usual similarity and search approaches on e-commerce websites.

[3] Indirect factors, such as high temperatures, regulate the way users buy shirts. The present disclosure also takes many of these “indirect factors” as valid inputs to shirt formation.

11 is a diagram illustrating an exemplary implementation of method steps according to a client-server implementation. Current implementations refer to "knowledge distillation," in which a deeper model is first trained to extract complex patterns from data. These complex patterns are then taught to the smaller child models. This results in a much smaller memory footprint, but with similar accuracy.

According to one embodiment, a mapping network and a generator network at the server 1102 utilize knowledge distillation to train a smaller model on the device of the client 1104 . The server 1102 implements a first version of the ANN to generate the image, the first version of the ANN being trained and then a large model configured to train a smaller model.

The client 1104 implements a second version of the ANN configured to be trained by the first version of the ANN as part of the knowledge distillation, wherein the client 1104 computes an optimal vector value and caches the values calculated thereby. is configured to More specifically, the client 1104 uses a 'local' copy of the generative network to compute an optimal sampling vector. The local copy obtained is an extract from knowledge distillation and works in real time. Multiple user experiences are obtained on the same device. The optimal sampling vector is stored in the local cache as user metadata.

User metadata moves to server 1102 as a single, slow federated update, which saves bandwidth costs. User metadata includes "numbers" instead of text/image information to ensure user privacy. This is a unique advantage over conventional recommendation systems that crawl previously purchased user items.

12 is a diagram illustrating an exemplary implementation of method steps according to a client-server implementation. Implementation refers to learning on unlabeled user data using knowledge extension. Through the concept of knowledge expansion and distillation, the present implementation can personalize the experience for the user even if labeled data is not provided. Overall, knowledge expansion is performed prior to knowledge distillation to obtain the same model size for the client-device.

At the server end 1202, an unlabeled image of the target domain is received from the user by the server 1202 implementing the student and teacher ANN. The first ANN or teacher ANN derives intermediate labels for unlabeled images by perceptual image-similarity criteria, wherein the intermediate labels include a plurality of TDAs and one or more associated with the unlabeled images. It is defined by a sampling vector. More specifically, the combination of target domain attributes and sampling vectors that produce an unlabeled image is calculated by perceptual image-similarity constraints. Expressions are stored as “soft pseudo-labels” or intermediate labels for the generated samples.

A second ANN of the same or larger size is trained based on the labeled data and intermediate-labelled data for image generation. Larger student networks are trained on labeled and pseudo-labelled data to learn more complex representations. Then, the first ANN is replaced by the second ANN to predict the intermediate label with respect to the unlabeled image, and based on this, the second ANN is retrained. This process repeats until the user is satisfied with the output generated. The satisfaction threshold is calculated by the relative probability of a positive user response among the total responses collected.

Upon detection of positive user feedback on the imaging capacity of the second ANN, a compressed version of the trained second ANN is instantiated on the client 1204 device based on the detection. That is, the ideal larger network thus obtained is compressed back to the client-device 1204 by distillation.

13 is a diagram illustrating an exemplary architecture 1300 that provides the tools and development environment described herein for technical realization of the implementations of FIGS. 1B and 12 via an AI model-based computing device. 13 is merely a non-limiting example, and it will be understood that many other architectures may be implemented to facilitate the functionality described herein. The architecture may be executed on hardware such as computing machine 1400 of FIG. 14 including, among others, a processor, memory, and various application specific hardware components.

Architecture 1300 may include an operating system, library, framework, or middleware. An operating system can manage hardware resources and provide common services. An operating system may include, for example, a kernel, services and drivers that define a hardware interface layer. A driver may be responsible for controlling or interfacing with the underlying hardware. Drivers include, for example, display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (such as Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and Other drivers may be included depending on the hardware configuration.

The hardware interface layer includes libraries that may include system libraries (eg, C standard library) such as file-system that may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library includes an audiovisual media library (e.g., a multimedia data library that supports the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), a database library (e.g. For example, it may include an API library such as SQLite that can provide various relational database functions, a web library (eg, WebKit that can provide a web browsing function), and the like.

Middleware may provide a higher level common infrastructure such as various graphical user interface (GUI) functions, high level resource management, high level location services, and the like. Middleware may provide a broad spectrum of other APIs that may be used by applications or other software components/modules, some of which may be specific to a particular operating system or platform.

As used herein, the term “module” may refer to a specific unit comprising one of hardware, software, and firmware, or any combination thereof. A module may be used interchangeably with, for example, a unit, logic, logic block, component, or circuit. A module may be a minimum unit or a part of performing one or more specific functions. A module may be formed mechanically or electronically. For example, a module according to the present disclosure may include at least one of a known or planned application-specific integrated circuit (ASIC) chip, a field-programmable gate array (FPGA) and a programmable-logic device.

Architecture 1300 also illustrates an aggregation of computing device-based mechanisms and ML/NLP-based mechanisms according to embodiments of the present disclosure. The user interface defined as input and interaction 1301 refers to the entire input. This may include one or more of a touch screen, a microphone, a camera, and the like. A first hardware module 1302 depicts specialized hardware for an ML/NLP based mechanism. In an example, the first hardware module 1302 includes one or more of a neural processor, FPGA, DSP, GPU, or the like.

The second hardware module 1312 depicts specialized hardware for executing device-related audio and video simulations. The ML/NLP based framework and API 1304 corresponds to a hardware interface layer for executing the ML/NLP logic 1306 based on the underlying hardware. In an example, the framework may be one or more of Tensorflow, Cafe, NLTK, GenSim, ARM Compute, and the like. The simulation framework and API 1314 may include one or more of Device Core, Device Kit, Unity, Unreal, and the like.

Database 1308 depicts a pre-trained multimedia content database comprising clusters of pre-formed multimedia content in the latent space. The database 1308 may be accessed remotely via the cloud by the ML/NLP logic 1306 . In another example, the database 1308 may reside partly on the cloud and partly on a device based on usage statistics.

Another database 1318 refers to a computing device DB that will be used to store multimedia content. Database 1318 may be accessed remotely via the cloud. In another example, database 1318 may reside partly on the cloud and partly on a device based on usage statistics.

A rendering module 1305 is provided for rendering the multimedia output and triggering further utility actions as a result of user authentication. The rendering module 1305 may appear as a display/touch screen, monitor, speaker, projection screen, or the like.

The general-purpose hardware and driver module 1303 corresponds to the computing device 1400 mentioned in FIG. 14 , and instantiates drivers for general-purpose hardware units as well as application-specific units 1302 , 1312 .

In one embodiment, the NLP/ML mechanism and VPA simulation underlying the present architecture 1300 may be remotely accessible and cloud-based, thereby remotely accessible via a network connection. A computing device, such as a VPA device, may be configured to remotely access an NLP/ML module, and the simulation module may include skeletal elements such as a microphone, camera, screen/monitor, speaker, and the like.

In addition, at least one of the plurality of modules of FIGS. 2 and 3 may be implemented through AI based on the ML/NLP logic 1306 . Functions related to AI may be performed through a non-volatile memory, a volatile memory, and a processor constituting the first hardware module 1302, that is, specialized hardware for an ML/NLP-based mechanism. A processor may include one or a plurality of processors. In this case, the one or more processors include a general-purpose processor such as a central processing unit (CPU), an application processor (AP), etc., a graphics-only processing unit such as a graphic processing unit (GPU), a visual processing unit (VPU), and/or It may be an AI-only processor, such as a neural processing unit (NPU). The above-described processor generally corresponds to the processor 1402 of FIG. 14 .

One or more processors control processing of input data according to pre-defined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. Pre-defined operating rules or artificial intelligence models are provided through training or learning.

Here, being provided through learning means that an AI model with predefined operating rules or desired characteristics is created by applying learning logic/technology to a plurality of learning data. Learning according to an embodiment may be performed in the device (ie, architecture 1300 or device 1600) in which AI is performed, and/or may be implemented through a separate server/system.

An AI model may consist of multiple neural network layers. Each layer has a plurality of weights, and the layer operation is performed through the calculation of the previous-layer and the calculation of the plurality of weights. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann Machine (RBM), and a deep trust network (DBN). : deep belief networks), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

The ML/NLP logic 1306 is a method to train a pre-defined target device (eg, a robot) using a plurality of training data to induce, allow, or control the target device to make a decision or prediction. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, or semi-supervised learning or reinforcement learning.

14 is a diagram illustrating another exemplary implementation according to an embodiment, and shows another typical hardware configuration of a system 1300 in the form of a computer system 1400 . Computer system 1400 may include a set of instructions that may be executed to cause computer system 1400 to perform any one or more of the disclosed methods. Computer system 1400 may operate as a standalone device or may be connected to other computer systems or peripheral devices using, for example, a network.

In a networked deployment, computer system 1400 may be configured as a client user computer in the capacity of a server or in a server-client user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. can operate as Computer system 1400 may also specify an action to be taken by a VR device, personal computer (PC), tablet PC, personal digital assistant (PDA), mobile device, palmtop computer, communication device, web appliance, or such machine. may be implemented as, or integrated across, various devices, such as any other machine capable of executing a set of instructions (sequential or otherwise) capable of executing a set of instructions. Also, although a single computer system 1400 is depicted, the term “system” also refers to any of a system or sub-system that individually or jointly executes a set or plurality of sets of instructions to perform one or more computer functions. It will be understood to include a set of

The computer system 1400 may include a processor 1402 , for example, a central processing unit (CPU), a graphics processing unit (GPU), or both. The processor 1402 may be a component of various systems. For example, the processor 1402 may be part of a standard personal computer or workstation. Processor 1402 may include one or more general processors, digital signal processors, application specific integrated circuits, field programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or currently known or future data for analyzing and processing data. It may be another device to be developed in The processor 1402 may implement a software program, such as manually generated (ie, programmed) code.

Computer system 1400 can include memory 1404 , such as memory 1404 , that can communicate via bus 1408 . Memory 1404 includes, but is not limited to, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, and the like. It may include, but is not limited to, computer-readable storage media such as, but not limited to, various types of volatile and non-volatile storage media. In one example, memory 1404 includes cache or random access memory for processor 1402 . In an alternative example, memory 1404 is separate from processor 1402 , such as the processor's cache memory, system memory, or other memory. Memory 1404 may be an external storage device or database for storing data. Memory 1404 is operable to store instructions executable by processor 1402 . Functions, acts, or tasks shown or described in the figures may be performed by programmed processor 1402 executing instructions stored in memory 1404 . A function, action, or task may be performed by software, hardware, integrated circuit, firmware, micro-code, etc., operating alone or in combination, independent of any particular type of instruction set, storage medium, processor, or processing strategy. . Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, and the like.

As shown, computer system 1400 is a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, a cathode ray tube (CRT), a projector, or a presently known or future developed for outputting determined information. It may or may not further include a display unit 1410 such as another display device to be used. Display 1410 may serve as an interface through which a user may view the functions of processor 1402 , or specifically as an interface with software stored in memory 1404 or drive unit 1416 .

Computer system 1400 may also include an input device 1412 configured to allow a user to interact with any component of system 1400 . Computer system 1400 may also include a disk or optical drive unit 1416 . The disk drive unit 1416 may include a computer-readable medium 1422 in which one or more sets of instructions 1424, eg, software, may be embodied. Further, instructions 1424 may implement one or more of the methods or logic as described. In a particular example, the instructions 1424 may reside fully or at least partially within the memory 1404 or the processor 1402 while being executed by the computer system 1400 .

This disclosure includes instructions 1424 or in response to a propagated signal to enable devices connected to the network 1426 to communicate voice, video, audio, images, or any other data via the network 1426. 1424) are contemplated for receiving and executing a computer-readable medium. In addition, the commands 1424 may be sent or received via the communication port or interface 1420 or via the network 1426 using the bus 1408 . The communication port or interface 1420 may be part of the processor 1402 or may be a separate component. Communication port 1420 may be created in software or may be a physical connection of hardware. Communication port 1420 may be configured to connect with network 1426 , external media, display 1410 , or any other component of system 1400 , or a combination thereof. The connection to the network 1426 may be a physical connection, such as a wired Ethernet connection, or may be established wirelessly, as discussed below. Likewise, additional connections with other components of system 1400 may be physical or may be established wirelessly. Network 1426 may alternatively be directly connected to bus 1408 .

Network 1426 may include a wired network, a wireless network, an Ethernet AVB network, or a combination thereof. The wireless network may be a cellular telephone network, an 802.11, 802.16, 802.20, 802.1Q or WiMax network. Additionally, network 1426 may be a public network, such as the Internet, a private network, such as an intranet, or a combination thereof, and may include, but are not limited to, a variety of networking currently available or to be developed in the future, including, but not limited to, TCP/IP-based networking protocols. protocol is available. The system is not limited to operating with any particular standards and protocols. For example, standards for transport over the Internet and other packet switched networks (eg, TCP/IP, UDP/IP, HTML, HTTP) may be used.

According to an embodiment of the present disclosure, in a system for generating an image in a computing-environment based on artificial intelligence (AI)-based technology, the system includes at least one sensor for receiving an external-input for generating an image. A receiving unit (1301) to shortlist a plurality of target domain attributes (TDAs) from the plurality of TDAs based on one or more clusters associated with the plurality of TDAs, and interpolate data within a latent space defined by a representation of the shortlisted TDAs; , the direction of interpolation is (i) a sampling-vector computed based on at least one of (a) the outer-input and (b) the one or more clusters, and/or (i) the latent codes for predicting the latent codes. determine based on an auto-learned relationship within the latent codes in space and generate at least one latent code based on the interpolation along the determined direction in the latent space, at least based on the latent-code and a processor 1303, configured to generate at least one image by the ANN, and a display module 1305 for displaying the at least one generated image.

15 is a diagram illustrating an example implementation of the present disclosure, illustrating real-time latent spatial interpolation associated with external feedback. Interpolation can result in wallpapers that change dynamically as displayed on a television in the living room.

State-of-the-art frame TVs have the ability to disguise wallpapers depending on the surface on which they are mounted. Real-time latent spatial interpolation associated with external feedback in accordance with the present disclosure provides an additional level of personalization. Health data collected from mobile devices and wearables such as smartwatches can be used to create dynamic wallpapers on the screen. In one embodiment, during a group exercise, each section of the screen may change color according to the person's heartbeat, acting as external feedback. By changing the parameters of the latent space based on external feedback and personalization, infinite wallpapers can be created.

In the example currently shown in FIG. 15 , the change in time of day coupled with the observed climate change during the day provides external feedback to dynamically generate new wallpapers via the GAN. In another example, the mood swings of a living room member may be captured as optimistic, sad, happy, and normal. Based on human emotions, the wallpapers change dynamically.

16 is a diagram illustrating an example implementation of the present disclosure, illustrating the ability of a latent space to interpolate based on external feedback, thereby completing an inventory of an online marketplace.

As is commonly observed, certain items may be out of stock during online shopping. The consumer does not have the option to see the missing items. According to the latent space of the present disclosure, the user only needs to select two items from the inventory. This disclosure creates infinite options amongst what a user can choose from. This may at least allow the seller to no longer have to spend a lot of money to advertise all possible items. For a blue shirt, for example, every shade no longer has to be advertised. Also, when the user selects a particular item, a new design created by latent spatial interpolation can be placed on a custom order. Sellers can use the newly created design to come up with more creative products.

17 is a diagram illustrating an example implementation of the present disclosure, illustrating the ability of latent spaces to interpolate and provide various user experiences.

As can be appreciated, a fixed point in latent space needs to be reached at a certain time. However, these points are loosely defined. For example, while watching a movie, there are some important moments where everyone needs the same experience. These "moments" are definite points in latent space. There are certain instances that a group of users will experience during the same time-interval. For example, different people watch the same movie for the entire duration of that length.

By now adjusting the rate of sampling for each user in accordance with the present disclosure, a variety of experiences can be provided while still complying with timing constraints. In one embodiment, the present disclosure provides a different experience each time the same movie is viewed. For example, in a 5D movie, the type of smell provided to the user during movie time may be adjusted in real time.

18 illustrates an example implementation of the present disclosure, with reference to a personalized workout generator for rendering workout recommendations according to a user's physical condition. The source domain is used to select a dynamic final point in the latent space of the target domain. Based on this, the interpolation trajectory is adjusted accordingly.

A state-of-the-art fitness system system can recommend exercises and automatically correct a person's posture in real time. However, the predictions derived are usually very broad. The user is asked to specify the kind of goal to be achieved. In one embodiment, the goal may be 100 push-ups per day. By assuming that the goal is constant, the user's performance is evaluated and recommendations are provided.

In our system, we use user parameters such as health to change goals in real time. For example, if a person feels sick, his heart rate may be too high. Instead of expecting him to complete 100 push-ups of his choice, this disclosure predicts that he will do 70. Thus, the fitness system is automatically calibrated in real time.

Overall, the present disclosure provides significant advantages over the state of the art. Because user personalization is critical for the success of any service industry, this disclosure utilizes AI models such as generative adversarial neural networks (GANs) to deliver and compute infinite experiences to users using fixed memory, thereby enabling machines and Bridging the long-standing gap between humans.

As can be appreciated, neural networks learn complex patterns, but do not know which patterns relate to humans. On the other hand, humans can "see" patterns and choose patterns they like, but cannot quickly create new ones in their minds. This disclosure uses machines to create diversity and bridges this gap by rendering human-computer interfaces (HCIs) by asking humans whether they like them. The present disclosure enables a machine to learn based at least on human-specific biases to generate patterns that are related to humans.

Although specific language has been used to describe the present disclosure, any limitation that arises in this regard is not intended. As will be apparent to those skilled in the art, various working modifications may be made to a method for implementing the concepts of the present disclosure as taught herein.

The drawings and the foregoing description provide examples of embodiments. A person skilled in the art will understand that one or more of the described elements may be well combined into a single functional element. Alternatively, a particular element may be divided into a plurality of functional elements. Elements from one embodiment may be added to another embodiment. For example, the order of the processes described herein may vary and is not limited in the manner described herein.

Also, the actions in the flowchart need not be implemented in the order shown; Not all actions have to be performed. Also, these actions that do not depend on other actions may be performed in parallel with other actions. The scope of the embodiments is by no means limited by these specific examples. Numerous variations are possible, such as differences in structure, dimensions, and material use, whether or not specified herein. The scope of the embodiments is at least as broad as given by the following claims.

Advantages, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, an advantage, advantage, solution to a problem and all component(s) which give rise to or make it more pronounced an advantage, advantage, or solution shall be significant, necessary, or necessary in any or all claims. It should not be construed as an essential feature or component.

Claims (23)

In an electronic device,
a memory for storing neural network models for image generation; and
receiving a first user input for obtaining an image;
Identifies a domain corresponding to the first user input from among a plurality of domains defined in order to classify images,
When information related to at least one domain property among a plurality of domain properties defined in order to classify properties of images corresponding to the identified domain is included in the first user input, it is transmitted to the identified domain through the neural network model. is included and acquires at least one image corresponding to the at least one domain attribute,
a processor that provides the obtained at least one image; An electronic device comprising a.
According to claim 1,
Information related to the domain attribute,
at least one of direct information for selecting the at least one domain attribute and first indirect information related to the at least one domain attribute,
The processor is
When the first indirect information is included in the first user input, the first indirect information is mapped to a domain property preset to correspond to the first indirect information,
Identifies the at least one domain attribute including a domain attribute corresponding to the direct information and a domain attribute mapped to correspond to the first indirect information,
The electronic device acquires the at least one image corresponding to the identified at least one domain attribute.
3. The method of claim 2,
The processor is
when the at least one image is obtained, obtain second indirect information related to the at least one domain attribute from the at least one image;
When the second indirect information matches the first indirect information, providing the at least one image,
When the second indirect information does not match the first indirect information, the electronic device retrains the neural network model.
According to claim 1,
The processor is
When a second user input including user feedback information on the provided at least one image is received, the at least one domain attribute is adjusted based on the feedback information,
The electronic device acquires the at least one image corresponding to the adjusted at least one domain attribute.
5. The method of claim 4,
The feedback information includes positive feedback information on a first image of the at least one image and negative feedback information on a second image of the at least one image,
The at least one adjusted domain attribute includes one or more domain attributes excluding a plurality of domain attributes corresponding to the second image from among a plurality of domain attributes corresponding to the first image.
A method for generating an image in a computing-environment based on artificial intelligence (AI)-based technology, the method comprising:
Receiving an external-input for image-generation by an artificial neural network (ANN) (S102), wherein the ANN operates on a plurality of target domain attributes (TDA) for a target domain configured to: receiving (S102);
shortlisting a plurality of target domain attributes (TDAs) from the plurality of TDAs based on at least one of the external inputs and one or more clusters associated with the plurality of TDAs (S104);
interpolating (106) data within a latent space defined by the representation of the shortlisted TDA, wherein the direction of interpolation is:
(i) a sampling-vector computed based on at least one of (a) the outer-input and (b) the one or more clusters; and/or
(i) interpolating (S106) to predict latent codes, determined based on auto-learned relationships within latent codes in the latent space;
generating at least one latent code based on the interpolation along the determined direction in the latent space (S108); and
generating ( S110 ) at least one image by the ANN based at least on the latent-code.
7. The method of claim 6,
The direction of the interpolation is:
aggregating the logged sampling vectors and the currently computed sampling vector to result in an aggregated-vector through weighted-averaging;
calculating a variation pattern among existing latent codes based on the auto-learned relationship in the latent space through a neural network; and
deriving an aggregated-direction based on the aggregated vector and the direction associated with the computed pattern, thereby causing an associated-updation to the direction of the interpolation.
8. The method of claim 7,
The operation of the current sampling-vector is:
derivation of a first vector as a user sampling vector (USV) from the external input;
derivation of a second vector as a cluster sampling vector (CSV) from the clusters associated with the short-listed TDA; and
and combining the first vector and the second vector to result in the current-sampling vector.
7. The method of claim 6,
Receiving the external input includes:
receiving one or more user-labels electronically or acoustically to generate images, the user-labels optionally carrying one or more source domain attributes;
mapping the user-label and one or more target domain attributes to facilitate shortlisting of the TDA; and
mapping the short-listed TDA with the one or more accompanying source domain attributes.
7. The method of claim 6,
Receiving the external input includes:
predicting intermediate-labels in the latent space by a machine based on an existing latent-code; and
classifying the plurality of TDAs based on the predicted mid-labels until attainment of a threshold defined by user-feedback; and
and receiving the external input as an auto-generated trigger based on obtaining the shortlisted TDA based on the segmented TDA to enable interpolation.
7. The method of claim 6,
The ANN configured to generate images in the target-domain is:
a) a differentiated TDA representation for rendering a starting point in the latent space for image generation; and
b) a method defined by a plurality of characteristics as one or more of a plurality of initialized vectors defined by one or more of the cluster sampling vector (CSV), a cluster preference vector (CPV).
7. The method of claim 6,
The short listing of the TDA is:
identifying one or more clusters within the latent space associated with a short-listed TDA, and based thereon identifying one or more cluster preference vectors (CPV);
calculating a user-preference vector (UPV) based on the received external input comprising a user-label and optionally source domain information;
ranking the plurality of short-listed TDAs associated with the external input by combining the cluster preference vector (CPV) and the UPV; and
defining the ranked and shortlisted TDA and one or more combinations thereof to initiate the interpolation within the latent space.
10. The method of claim 9,
The interpolation in the latent space based on the sampling vector is:
computing a user sampling vector (USV) from the external input and a mapping drawn between the source domain attribute and the target domain attribute;
combining the USV and the sampling vector of the identified cluster to provide a resulting sampling vector;
determining a magnitude and direction associated with the resulting sampling vector; and
generating a plurality of latent codes in the latent space by interpolating along the direction of the resulting sampling vector based on the determined magnitude.
13. The method of claim 12,
updating one or more clusters based on the resulting sampling vector, wherein the one or more clusters include:
a) a cluster comprising the short-listed TDA; and
b) defined by one or more clusters linked to at least one of the short-listed TDAs.
7. The method of claim 6,
The interpolation of the latent space based on the external input is:
decomposing the resulting sampling vector into a unit direction vector and a corresponding magnitude; and
generating a plurality of latent codes along the unit direction vector by maintaining a constant magnitude.
7. The method of claim 6,
The interpolation of the latent space based on the auto-learned relationship within the latent space is:
retrieving, through the subspace defined by the shortlisted TDA, one or more latent codes configured to generate images in the target domain; and
computing a relationship between the latent codes in the subspace, thereby training a neural network to enable prediction of additional latent codes based on the computed relationship.
7. The method of claim 6,
The reception of the external input is:
receiving a plurality of user-feedback related to images generated as part of the latent spatial interpolation;
In the interpolated latent space,
an optimal user sampling vector (USV) based on the associated mean distance for positive feedback; and
Optimal user preference vector (UPV) obtained based on the relationship between positive and negative feedback
computing optimal vector values for the user for the particular combination of target domain attributes as one or more of
18. The method of claim 17,
aggregating the short-listed TDA with the calculated optimal vector values specific to the user according to claim 7 to provide an aggregated feature vector;
reconstructing information on each of the plurality of source domains based on the aggregated feature vector through a decoder forming part of a common network; and
comparing the reconstructed information with the source domain information received in the user input to compute the efficiency of the short-listed TDA;
updating the short-listed TDA to increase the efficiency, and updating at least one of the USV, UPV, CSV, and CPV and at least one cluster associated with the TDA based on this
reconstructing a plurality of source-domain attributes based on the target domain attributes via one or more of:
19. The method of claim 18,
receiving other user feedback to increase the efficiency;
updating the shortlisted TDA to further reconstruct the source domain attributes based on the received other feedback;
Based on the further reconstructed source domain properties,
a further optimized sampling vector for the user for the short-listed TDA;
an updated cluster sampling vector and the cluster preference vector; and
Update of at least one cluster associated with one or more of the short-listed TDAs.
The method further comprising the step of achieving one or more of
7. The method of claim 6,
The creation of the one or more images comprises:
feeding the random noise associated with a probability distribution to a deep learning network to extract one or more features underlying the random noise; and
sending the extracted features as input to the ANN to facilitate image generation.
7. The method of claim 6,
The ANN is:
a server implementing a first version of the ANN to generate an image, wherein the first version of the ANN is a large model that is trained and then configured to train smaller models;
A client implementing a second version of the ANN configured to be trained by the first version of the ANN as part of knowledge distillation, wherein the client computes optimal vector values, whereby the calculated value A method, operating through a client-server architecture defined by a client, configured to cache
7. The method of claim 6,
The ANN is:
receiving an unlabeled image in a target domain by a server implementing the ANN;
deriving, by a first ANN, intermediate-labels for the unlabeled image by perceptual image-similarity criteria, wherein the intermediate labels are in a plurality of TDAs and one or more sampling vectors associated with the unlabeled image. deriving, defined by;
storing intermediate-labels for one or more combinations of shortlisted TDA for the generated image;
training a second ANN of the same or larger size based on the labeled data and the intermediate-labeled data for image generation;
replacing the first ANN with the second ANN to predict intermediate-labels for an unlabeled image, and retraining the second ANN based thereon;
detecting positive user-feedback on the image generating capacity of the second ANN; and
instantiating the compressed version of the trained second ANN on a client device based on the detection;
A method, executed through a client-server architecture as defined by
7. The method of claim 6,
filtering one or more source domain attributes from the external-input based on a criterion defined by a combination of a causality criterion and a relevance criterion; and
modeling the filtered source domain properties into the plurality of target domain properties (TDA).

Images