KR102150720B1 - Image embedding apparatus and method for content-based user clustering - Google Patents

Abstract

The present invention relates to a fashion content embedding apparatus for user clustering and a method thereof. To this end, the fashion content embedding apparatus includes: a fashion item classification module which is configured to receive fashion item image information from a fashion item detection module, input each fashion item image information to a previously learned artificial neural network, and output fashion item classification information for each fashion item image information; and an embedding module which is connected to a fully connected layer (FC Layer) of the artificial neural network of the fashion item classification module and output an output vector of the FC layer of each artificial neural network for each fashion item image information as an embedding vector for each fashion item image information. Therefore, it can be automatically clustered by the embedding vector without an administrator tagging a style of interest or classifying users for the fashion content or fashion items which each user reacts.

Description

Fashion content embedding apparatus and method for user clustering TECHNICAL FIELD [Image embedding apparatus and method for content-based user clustering]

The present invention relates to an apparatus and method for embedding fashion contents for user clustering.

With the recent development of smartphones, various social networks are flooding, and the fashion market is undergoing catastrophic change. Users include Styleshare, Instagram, YouTube, fashion itembook, Snap, Snow, Pinterest, Tumblr, TikTok Appreciate fashion content including fashion items uploaded by other users, OOTD (Outfit of the day), V-log, etc. on social networks such as (Tiktok), and perform social actions such as like, comment, and share. They show empathy for fashion items and communicate with each other. In addition, according to the recent trend, fashion item advertisement content is also consumed as a single fashion content, and this flow in which fashion item advertisement content or fashion brand advertisement content is included in fashion content is especially the image of Instagram and Style Share. It appears prominently in social networks based. These recent trends are changing fashion commerce and fashion commerce advertisements.

The first change in fashion commerce and fashion commerce advertising is the rapid growth of creator-based fashion item commerce, and the second change is the rapid growth of identity-based fashion online commerce. Representative examples of the rapid growth of creator-based fashion item commerce include Kylie Cosmetics (Kylie Jenner), Huda Beauty (Huda Kattan), and Imvely. In the creator-based fashion item commerce, consumption characteristics are found to communicate with the creator or to resemble the creator, and the social network channel of the creator itself plays a role of an advertising channel. In addition, identity-based fashion online commerce includes FarFetch, YOOX, Net-A-Porter, Stylenanda, and Musinsa, and actively utilizes social networks as advertising channels, and considerable resources for advertising optimization such as recommendation algorithms and redirects. It has the characteristic of putting in.

Republic of Korea Patent Publication 10-2019-0029567, Product recommendation method using style characteristics, Omnious Co., Ltd.

In accordance with changes in social networks and fashion commerce, there have been many attempts to optimize fashion commerce using data generated from social networks. For example, Korean Patent Application Laid-Open No. 10-2019-0029567 provides the same style feature that recommends fashion items that have the same style feature to users but exist in different categories based on an image classification model that classifies style features of fashion images. It presents a way to recommend the products you have.

However, it was not easy because of the technical difficulty to cluster users' styles and identities by using Unsupervised Learning of log data generated in social networks to optimize fashion commerce. For example, the recommendation of a fashion item with the same style characteristics suggested in Korean Patent Application Laid-Open No. 10-2019-0029567 relates to an algorithm that determines whether it is simply the same shape without taking into account the overall atmosphere or matching harmony, etc. It cannot be used to cluster identities. When users are clustered and used for optimizing fashion commerce, the MD of fashion commerce categorizes the characteristics of users that are difficult to understand, such as a fashion item recommendation service, an advertisement service using a detailed user cluster, and a creator recommendation suitable for brand identity. The effect of optimizing fashion commerce in the form of fashion occurs.

Accordingly, an object of the present invention is to provide an apparatus and method for embedding fashion contents for user clustering, which enables clustering of user styles or identities by using log data generated in a social network.

Hereinafter, specific means for achieving the object of the present invention will be described.

An object of the present invention is to receive fashion content information of interest, which is fashion content information that a specific user inputs or uploads during a specific period of time in a user client or a social network web server, and a fashion content image included in the fashion content information of interest A fashion content receiving module that generates information and reaction information; The fashion content receiving module receives fashion content image information, detects an area of a fashion item from the fashion content image information to generate a plurality of fashion item area information, and the fashion item area information A fashion item detection module that generates fashion item image information, which is image information of at least part of fashion content image information, as many as the number of fashion items detected in the fashion content image information; The fashion item detection module is configured to receive the fashion item image information, input each of the fashion item image information to a pre-learned artificial neural network, and output fashion item classification information for each of the fashion item image information. Fashion item classification module; And a fully connected layer of the artificial neural network of the fashion item classification module, and an output vector of the FC layer of each artificial neural network with respect to each of the fashion item image information is obtained from each of the fashion item image information. And an embedding module that outputs an embedding vector for, wherein the loss function of the artificial neural network includes a distance between the embedding vectors for each of the fashion item image information, and in the learning step of the artificial neural network, the Learning the weight of the artificial neural network to minimize the loss value of the loss function, and using the embedding vector for each of the fashion item image information as input information of a user clustering module or a fashion item clustering module, This can be achieved by providing a fashion content embedding device for user clustering.

Another object of the present invention is that the fashion content receiving module receives fashion content information of interest, which is fashion content information that a specific user inputs or uploads reaction information for during a specific period in a user client or a social network web server, and the fashion content of interest A fashion content receiving step of generating fashion content image information and response information included in the information; A fashion item detection module receives fashion content image information from the fashion content receiving module, detects an area of a fashion item from the fashion content image information, and generates a plurality of fashion item area information. A fashion item detection step of generating fashion item image information, which is image information of at least part of the fashion content image information with respect to item area information, as many as the number of fashion items detected in the fashion content image information; A fashion item classification module classifies fashion items for each fashion item image information by receiving the fashion item image information from the fashion item detection module and inputting each of the fashion item image information to a previously learned artificial neural network A fashion item classification step, configured to output information; And the embedding module is connected to a fully connected layer of the artificial neural network of the fashion item classification module, so that the output vector of the FC layer of each artificial neural network for each of the fashion item image information And an embedding step of outputting an embedding vector for fashion item image information, wherein the loss function of the artificial neural network includes a distance between the embedding vectors for each of the fashion item image information, and the artificial neural network Learning the weight of the artificial neural network to minimize the loss value of the loss function in the learning step, and using the embedding vector for each of the fashion item image information as input information of the user clustering module or the fashion item clustering module. Characterized, it can be achieved by providing a fashion content embedding method for user clustering.

Another object of the present invention is a memory module for storing a fashion content embedding program code; And a processing module for processing the fashion content embedding program code, wherein the fashion content embedding program code is fashion content information input or uploaded by a specific user during a specific period in a user client or a social network web server. A fashion content receiving step of receiving fashion content information of interest and generating fashion content image information and reaction information included in the fashion content information of interest; A fashion item image that detects an area of a fashion item from the fashion content image information to generate a plurality of fashion item area information, and is image information of at least a part of the fashion content image information for each of the fashion item area information A fashion item detection step of generating information as much as the number of fashion items detected in the fashion content image information; A fashion item classification step, configured to output fashion item classification information for each fashion item image information by inputting each of the fashion item image information to a previously learned artificial neural network; And an embedding step of outputting an output vector of the FC layer of each of the artificial neural networks for each of the fashion item image information as an embedding vector for each of the fashion item image information, on a computer, The loss function of the artificial neural network includes a distance between the embedding vectors for each of the fashion item image information, and the weight of the artificial neural network to minimize the loss value of the loss function in the learning step of the artificial neural network. And using the embedding vector for each of the fashion item image information as input information of a user clustering module or a fashion item clustering module.It can be achieved by providing a fashion content embedding device for user clustering. have.

As described above, the present invention has the following effects.

First, according to an embodiment of the present invention, an effect of automatically clustering by an embedding vector occurs without having to manually tag a style of interest or classify users for fashion content or fashion items that users respond to.

Second, according to an embodiment of the present invention, by recommending a product based on fashion content/fashion items uploaded or reacted by a specific user, an effect of improving the conversion rate of a recommendation algorithm or an advertisement algorithm is generated.

Third, according to an embodiment of the present invention, an interest-targeted advertisement of a specific brand is possible by recommending a group of users or creators based on fashion contents/fashion items of a specific brand.

The following drawings attached to the present specification illustrate preferred embodiments of the present invention, and serve to further understand the technical idea of the present invention together with the detailed description of the present invention, so the present invention is limited to the matters described in such drawings. And should not be interpreted.
1 is a schematic diagram showing a fashion content embedding apparatus for user clustering according to an embodiment of the present invention;
2 is a schematic diagram showing an operation relationship of the fashion content receiving module 10 according to an embodiment of the present invention;
3 is a schematic diagram showing the application of a smoothing filter by a smoothing filter application module according to an embodiment of the present invention;
4 is a schematic diagram showing a result of generating candidate information of a candidate output module according to an embodiment of the present invention;
5 is a schematic diagram showing a non-candidate removal module according to an embodiment of the present invention;
6 is a schematic diagram showing the operational relationship of the artificial neural network of the fashion item classification module 12 according to an embodiment of the present invention;
7 is a schematic diagram showing an example of an artificial neural network of the fashion item classification module 12 according to an embodiment of the present invention;
8 is a schematic diagram showing embedding vector mapping in a clustering module according to an embodiment of the present invention.

Hereinafter, exemplary embodiments in which the present invention can be easily implemented by those of ordinary skill in the art will be described in detail with reference to the accompanying drawings. However, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

In addition, the same reference numerals are used for parts having similar functions and functions throughout the drawings. Throughout the specification, when a specific part is said to be connected to another part, this includes not only the case that it is directly connected, but also the case that it is indirectly connected with another element interposed therebetween. In addition, the inclusion of a specific component does not exclude other components unless specifically stated to the contrary, but means that other components may be further included.

Fashion content embedding device for user clustering

1 is a schematic diagram showing a fashion content embedding apparatus for user clustering according to an embodiment of the present invention. As shown in Fig. 1, the fashion content embedding device 1 for user clustering includes a fashion content receiving module 10, a fashion item detection module 11, a fashion item classification module 12, and an embedding module 13 , May include a clustering module 14. In addition, the fashion content embedding device 1 for user clustering according to an embodiment of the present invention is provided by a processing module of a computing device such as a specific web server, a virtual server such as a cloud server, a smartphone, a tablet PC, and a desktop PC. It can be configured to be processed and stored in a memory module of each device. The fashion content embedding device 1 for user clustering according to an embodiment of the present invention receives the user's reaction information (like, share, comment, etc.) or the fashion content of interest, which is the fashion content uploaded by the user, to a user client. It may be configured to receive directly from 100 or to receive from the social network web server 50 through a social network API.

Fashion content receiving module 10, the user's reaction information (Like, Share, Comment, etc.) from the user client 100 or the social network web server 50 is input or fashion content of interest that is uploaded by the user It is a module that receives 2 is a schematic diagram showing an operation relationship of the fashion content receiving module 10 according to an embodiment of the present invention. As shown in Figure 2, the fashion content receiving module 10 according to an embodiment of the present invention, a specific period for a specific user in the social network application module of the user client 100 or the social network web server 50 Receives fashion content information (fashion content image information, text information, tag information, comment information, like information, view count information, etc.) of interest during the period, and transmits fashion content image information received to the fashion item detection module 11 It is a module.

The fashion item detection module 11 receives fashion content image information from the fashion content receiving module 10, detects a fashion item area from the fashion content image information, and generates fashion item area information, This module generates fashion item image information, which is at least a part of fashion content image information, based on fashion item area information.

The fashion item detection module 11 according to an embodiment of the present invention may use a fashion item detection algorithm fine-tuning YOLO, RCNN, Faster RCNN, and the like. Alternatively, a fashion item detection algorithm that fine-tuned a network such as AlexNet previously learned with ImageNet can be used. Furthermore, it is possible to use existing computer vision algorithms such as Boosting the Haar-like Feature of Viola-jones.

The fashion item detection module 11 according to an embodiment of the present invention may include a smoothing filter application module, a candidate output module, and a non-candidate removal module. The fashion item detection by the fashion item detection module 11 according to an embodiment of the present invention is specifically configured to perform fashion item detection in fashion content image information, and the detected fashion item image information The fashion item expressing as a feature vector is transmitted to the embedding module 13 that performs feature embedding to vectorize each fashion item. In the clustering module 14, the embedding feature vectors are compared to form a cluster between vectors. It may be configured to perform clustering to output cluster information including at least one fashion item image information and user information clustered.

Fashion item detection of the fashion item detection module 11 according to an embodiment of the present invention includes a rectangle (or polygon, oval, or shape) configured to surround at least one fashion item included in the fashion content image information. A candidate output module that outputs candidate information, which is a fashion item box (circular, curved, etc.), non-candidate that outputs non-candidate classification information in which an image in a fashion item box among the candidate information is classified as not a fashion item It may be performed by the removal module, and may be configured to provide the candidate information whose non-candidate classification information is less than or equal to a specific value (excluding candidate information classified as non-candidate).

The smoothing filter application module according to an embodiment of the present invention generates at least one smoothing fashion content image information by applying a smoothing filter to the fashion content image information, and transmits the generated smoothing fashion content image information to the candidate output module. Can be configured. The smoothing filter of the smoothing filter application module may be composed of a smoothing filter such as a Gaussian filter, a bilateral filter, and a median filter, and the fashion content image information is gradually It may be configured to generate a plurality of smoothing fashion content image information by applying a smoothing weight and input it as input information of the artificial neural network of the candidate output module. For example, when a Gaussian filter is configured as a smoothing filter, the value of sigma, which is the standard deviation of the Gaussian kernel in the x and y directions, is gradually applied to a large value (for example, the first smoothing fashion content image It is configured to generate a plurality of smoothing fashion content image information while applying sigma to 1 for information, sigma to 2 for the second smoothing fashion content image information, etc.), and the smoothing fashion to which the smoothing weight of the smoothing filter is gradually applied. It may be configured to input content image information and fashion content image information to which a smoothing filter is not applied to the artificial neural network of the candidate output module. 3 is a schematic diagram illustrating application of a smoothing filter by a smoothing filter application module according to an embodiment of the present invention. As shown in FIG. 3, the smoothing filter application module according to an embodiment of the present invention generates at least one smoothing fashion content image information whose smoothing degree is gradually increased by gradually increasing Sigma for fashion content image information. Can be configured to

The candidate output module according to an embodiment of the present invention includes a pre-learned artificial neural network, and the input information of the artificial neural network of the candidate output module is a smoothing fashion in which a smoothing filter is applied to the fashion content image information and the fashion content image information. Content image information can be configured, and the output information is fashion item classification information (whether it is a fashion item) for a plurality of candidate information and fashion item area information of the candidate information (for example, coordinate information of each vertex, or a center cell). It may be configured to output coordinate information and width/height information). Specifically, the inference session of the previously learned artificial neural network of the candidate output module includes the fashion content image information and smoothing fashion content image information for a specific user who wants to perform user clustering as input information. Fashion item presence information, which is a confidence score for the existence of a fashion item in a specific area, and coordinate information of the corresponding area are output as an output vector. In the training session of the artificial neural network of the candidate output module, a plurality of fashion content image information about a plurality of users previously stored in the database of the social network web server through the social network web server 50 is used as source data. , In the fashion content video information, the zone information set for a specific fashion item is used as the ground-truth of artificial neural network learning, and the following loss information is merged to be used as a loss function, and the loss value of the loss function is minimized. The weight of each layer of the artificial neural network is learned.

[Loss 1] The difference between the predicted center coordinates of the area predicted to exist and the ground-truth center coordinates

[Loss 2] The difference between the width/height of the area predicted to have a fashion item and the width/height of the ground-truth

[Loss 3] The difference between the reliability of the area where the fashion item is predicted to exist (information on the presence or absence of fashion items) and the existence of the actual fashion item in the corresponding area of Ground-truth

[Artificial neural network loss function of candidate output module] = loss 1 + loss 2 + loss 3

In addition, in the artificial neural network of the candidate output module according to an embodiment of the present invention, the entire layer may be composed of only a convoulution layer and a pooling layer, and a platen layer or a fully connected layer. Layer may not be configured. Accordingly, since candidate information, which is output information, is configured in three dimensions, no dimensional loss occurs, and a plurality of candidate information is output at a relatively high speed compared to a general convolutional neural network (CNN). For example, the artificial neural network of the candidate output module according to an embodiment of the present invention includes a 3x3 convolution filter and 2x2 max pooling on 12x12x3 input information (fashion content image information or smoothing fashion content image information). A convolution layer that outputs a 5x5x10 vector by applying a filter (Max Pooling Filter), a convolution layer that outputs a 3x3x16 vector by applying a 3x3 convolution filter to a 5x5x10 vector, and a 3x3x16 vector Convolutional layer that outputs a 1x1x32 vector by applying a convolution filter of, and fashion item availability information for a plurality of candidate information output as a 1x1x2 vector by applying a convolution filter to a 1x1x32 vector, and 1x1x4 vector. It may be configured to include fashion item area information of candidate information output as a vector as output information.

The artificial neural network of the candidate output module according to an embodiment of the present invention may use a cross entropy loss as a cost function for learning the presence or absence of fashion items. That is, the weight of the hidden layer of the artificial neural network of the candidate output module is updated so that the cross entropy loss is reduced (optimized) for each candidate information at a specific learning rate. At this time, optimization methods that can be used include a gradient descent method, a momentum method, and the like, and a back propagation algorithm may be used to easily apply gradient descent. In addition, the artificial neural network of the candidate output module according to an embodiment of the present invention may use Euclidean loss as a loss function for coordinate points such as each vertex for learning fashion item area information of each candidate information. have.

The artificial neural network included in the candidate output module according to an embodiment of the present invention may be configured as an artificial neural network including fewer layers than the non-candidate removal module. Accordingly, there is an effect that the output of candidate information proceeds very quickly.

4 is a schematic diagram showing a result of generating candidate information by a candidate output module according to an embodiment of the present invention. As shown in FIG. 4, according to the candidate information generation module according to an embodiment of the present invention, fashion item area information having a relatively small size is generated from fashion content image information, and relatively size is generated from smoothing fashion content image information. It may be configured to generate information about a large fashion item area. According to this, since candidate information is output for each fashion content image information to which a smoothing filter is not applied and a plurality of smoothing fashion content image information to which a smoothing filter is applied, a plurality of different sizes included in one fashion content image information There is an effect of being able to detect all parts of a fashion item. For example, for fashion content image information to which a smoothing filter is not applied, a fashion item portion having the smallest size is detected, and for smoothing fashion content image information to which a smoothing filter is applied, a fashion item portion having a larger size is detected. The effect occurs. This is because the smoothing fashion content video information disappears leaving only features that are relatively larger in scale than the fashion content video information.

In addition, the candidate output module according to an embodiment of the present invention may be configured to remove redundant candidate information, which is redundant candidate information, among the plurality of candidate information output from the artificial neural network. The redundant candidate information removal of the candidate output module may be performed in the following order.

(1) The candidate output module sorts the plurality of candidate information output from the artificial neural network in the order of the highest fashion item presence information (the highest reliability, the highest confidence).

(2) Among candidate information that overlaps with the candidate information having the highest fashion item presence information, candidate information whose ratio of the overlapping area to the total area of the candidate information is more than a certain value is determined to have detected the same fashion item and removed. (For example, if the ratio of the overlapping area to the total area of candidate information is 50% or more, it is removed)

(3) Among the candidate information that was not removed in step 2, step 2 is performed in order of the candidate information with the highest fashion item presence information

Accordingly, it is possible to prevent duplicated output of candidate information for the fashion content image information or the same fashion item included in the smoothing fashion content image information. This effect generates a chain effect that reduces the computational load of the non-candidate removal module.

The non-candidate removal module according to an embodiment of the present invention is a fashion content image corresponding to the candidate information (fashion item presence information for the candidate information and fashion item area information of the candidate information) output from the candidate output module. It may include a pre-learned artificial neural network that uses image information in the information as input information and non-candidate classification information of the candidate information as output information. In addition, the non-candidate removal module according to an embodiment of the present invention includes candidate information classified as not including a fashion item among candidate information based on the non-candidate classification information of the candidate information (the non-candidate classification information is equal to or greater than a specific value). Candidate information) may be removed to output the remaining candidate information (including fashion item area information, fashion item presence information, and non-candidate classification information) from which the non-candidate candidate has been removed as fashion item image information. 5 is a schematic diagram showing a non-candidate removal module according to an embodiment of the present invention. 5, the non-candidate removal module according to an embodiment of the present invention receives candidate information generated by the candidate output module, and removes candidate information whose reliability of non-candidate classification information is greater than or equal to a specific value among candidate information. Afterwards, it is a module that outputs video information or area information of candidate information remaining without being removed as fashion item video information and transmits it to the fashion item classification module 12. Accordingly, it is possible to remove candidate information classified as not including a fashion item among the candidate information with a high probability.

The fashion item classification module 12 inputs each fashion item image information included in the received fashion content image information into a pre-learned artificial neural network (especially, ConvNet), and stores fashion item classification information for each fashion item image information. It is a module that is configured to output (inference stage of artificial neural network). 6 is a schematic diagram showing an operation relationship of the artificial neural network of the fashion item classification module 12 according to an embodiment of the present invention. As shown in FIG. 6, each fashion item image information included in the fashion content image information received by the fashion item classification module 12 is converted into a pre-learned artificial neural network (especially, ConvNet) of the fashion item classification module 12. It may be configured to input to and output fashion item classification information for each fashion item image information. In the inference step of the previously learned artificial neural network of the fashion item classification module 12, fashion content image information of a user who wants to perform user clustering is used as input information.

In the learning step, the artificial neural network of the fashion item classification module 12 according to an embodiment of the present invention uses a plurality of fashion content image information of a plurality of users previously stored in the database of the social network web server 50 as source data. The fashion item label information on which fashion item category or fashion item is the fashion item image information is used as the ground-truth of artificial neural network learning, and the following loss information is merged and used as a loss function. The weight of each layer of the artificial neural network is learned to minimize the loss value of the loss function. In this case, the source information (input information) for learning may be a plurality of fashion content image information in a social network, and may include reaction information (like feeling information, comment information, sharing information, etc.) of the fashion content.

[Loss 1] The difference between the fashion item classification information on which label the fashion item video information is classified and the actual fashion item label information on the corresponding fashion item video information of Ground-truth

[Loss 2] The distance between vectors output from the FC layer (Fully Connected Layer) of each artificial neural network for each fashion item video information (if one fashion item is detected in one fashion content video information, the distance is 0 )

[Artificial neural network loss function of fashion item classification module] = loss 1 + loss 2

7 is a schematic diagram showing an example of an artificial neural network of the fashion item classification module 12 according to an embodiment of the present invention. As shown in FIG. 7, the artificial neural network of the fashion item classification module 12 may be constructed as [INPUT-CONV-RELU-POOL-FC]. The fashion item image information, which is input information, may be composed of a matrix having a channel of 32 width, 32 height, and height n, and an input size of [32x32xn]. The CONV layer (Conv. Filter) 101 is connected to a partial area of fashion item image information, and a dot product of the connected area and weight is calculated. The resulting volume will have the same size as [32x32x12]. The RELU layer is an activation function applied to each element, such as max(0,x). The RELU layer does not change the volume size ([32x32x12]). As a result, Activation map 1 (102) is generated. The POOL layer (pooling, 103) performs downsampling on the "horizontal and vertical" dimensions and outputs a reduced volume (Activation map 2, 104) such as [16x16x12]. After the fully-connected (FC) layer 106 connected to the n-th activation map n 105, class scores are calculated, and a volume (output layer, 107) having a size of [m x m x 1] is output. Loss 2 of the loss function of the artificial neural network of the fashion item classification module 12 uses the output vector of the FC layer 106, which has a reduced dimension of the fashion item image information, and the FC for each fashion item image information The distance between the output vectors of the layer is composed of the loss value of loss 2.

In particular, the weight of loss 2 may be configured to be proportional to the number of reaction information (like information, comment information, sharing information, etc.) of the corresponding fashion content video information. For example, in the case of fashion content having a lot of response information, loss 2 may have a higher weight than loss 1. That is, in the fashion content having more response information, the distance between the vectors of the FC layer of each artificial neural network with respect to each fashion item image information in the corresponding fashion content image information may be outputted closer.

Regarding the loss 2 of the artificial neural network of the fashion item classification module, for example, when learning the artificial neural network of the fashion item classification module shown in FIG. 6, the artificial neural network and boots for the boot 1 part (fashion item image information) Artificial neural network for part 2, artificial neural network for jeans part, artificial neural network for leather jumper The output vector (embedding vector) of the Fully Connected Layer before the output layer, not each output layer (Boots 1-Boots 2) The artificial neural network is set to minimize the value of the loss function that combines the loss 1 and the loss 2, using the combined value of the distance between embedding vectors, boots 1-the distance between jeans embedding vectors, etc.) You will learn. According to an embodiment of the present invention, fashion items included in one fashion content image information have a distance between the output vectors of the FC layer close to each other, and further, fashion content with more response information is between the output vectors of the FC layer. As the distance of is outputted closer, the effect of being outputted so that matching fashion items have close vector values occurs. According to this, the existing fashion item classification artificial neural network is used to classify the matching fashion items when combined with each other only by changing the loss function, even without tagging each fashion item for each fashion item. There is an effect that items can be clustered by style.

The embedding module 13 is a module that is connected to the FC layer of the artificial neural network of the fashion item classification module 12 and receives an output vector of the FC layer as an embedding vector for a corresponding fashion item. For example, as shown in FIG. 6, when there are four fashion item image information in one fashion content image information, the embedding module 13 generates four embedding vectors corresponding to each fashion item image information.

The clustering module 14 generates cluster information by clustering an embedding vector of source data (fashion item image information of at least part of fashion content image information of a social network) previously used for learning, and the embedding module 13 This module maps the embedding vector for each fashion item image information generated in the cluster information. According to the clustering module 14, a plurality of fashion items are detected with respect to at least one fashion content to which a specific user inputs reaction information in a specific social network, and the plurality of fashion items are previously learned through the clustering module 14. Since it is mapped to a plurality of clusters, at least one cluster is mapped for one user, and each cluster represents the style identity of the user, and the number of fashion items included in each cluster represents the importance of the style identity. Can be interpreted as.

That is, according to the clustering module 14 according to an embodiment of the present invention, the style identity of a specific user, which is a cluster composed of a plurality of fashion items by inputting information on fashion content of interest of a specific user (a plurality of It may be configured to output the fashion item image information). In addition, according to the clustering module 14 according to an embodiment of the present invention, information about fashion contents of interest of each of a plurality of users is input, and a group of users (user clustering information) that groups a plurality of users by style identity is output. Can be configured to Further, according to the clustering module 14 according to an embodiment of the present invention, an individual fashion item or a fashion item of a specific brand is input by the clustering module 14 to output a fashion item corresponding to a cluster such as a corresponding fashion item. It may be configured to input reaction information to a fashion item corresponding to a cluster such as a corresponding fashion item or to output an uploaded user cluster (user clustering information).

8 is a schematic diagram showing embedding vector mapping in a clustering module according to an embodiment of the present invention. As shown in Figure 8, the clustering module 14 according to an embodiment of the present invention may include a plurality of pre-learned cluster information, each of the clusters, such as shoes, tops, bottoms, hats, jewelry, etc. It may include embedding vectors of various categories of fashion items. In addition, the embedding vector of fashion items included in at least one fashion content in which a specific user inputs reaction information in a specific social network may be configured to be mapped to the previously learned clusters, and each cluster to which fashion items are mapped Represents the style identity of the specific user. Using this style identity of a specific user, a recommendation algorithm or advertisement algorithm can be configured to recommend or advertise other fashion items in the same cluster. Alternatively, an algorithm for recommending creators (based on uploaded fashion content) in the same cluster as the cluster to which the fashion item of a specific brand belongs may be configured. Alternatively, an algorithm for outputting users in the same cluster as the cluster to which the product of a specific brand belongs may be constructed in order to compose an advertisement or event targeted to users having a similar brand and style identity.

As the clustering algorithm of the clustering module 14 according to an embodiment of the present invention, a general clustering algorithm such as K-means may be used. In addition, in the clustering algorithm according to an embodiment of the present invention, if there are more than a certain number of points within a radius e (epsilon) based on a core point among the embedding vectors for a plurality of fashion item image information, one cluster is formed. It can be configured in a method of forming an arbitrary number of clusters by repeatedly performing the recognition process. Accordingly, there is an effect of automatically finding the number of clusters of fashion item image information without the need to preset the number of clusters, and an effect of preventing deterioration of clustering performance due to outliers is generated. In addition, even when it is not known how many fashion items will appear in the specific fashion content video information, the effect of clustering occurs, and since the FC layer of the fashion item classification module 12 is used as an embedding vector, parameters suitable for clustering are previously The effect of being fixed occurs.

Fashion content embedding method for user clustering

The fashion content embedding method for user clustering according to an embodiment of the present invention may include a fashion content reception step, a fashion item detection step, a fashion item classification step, an embedding step, and a clustering step.

In the fashion content receiving step, the fashion content receiving module 10 includes fashion content information of interest, which is fashion content information that a specific user inputs or uploads response information for a specific period in the user client 100 or the social network web server 50 Fashion content video information, text information, tag information, comment information, like information, view count information, etc.) are received, and the received fashion content video information is transmitted to the fashion item detection module 11.

In the fashion item detection step, the fashion item detection module 11 receives fashion content image information from the fashion content receiving module 10, detects a fashion item area from the fashion content image information, and detects fashion item area information. And generating fashion item image information that is at least a part of fashion content image information based on fashion item area information.

In the fashion item classification step, the fashion item classification module 12 inputs each fashion item image information included in the received fashion content image information into a pre-learned artificial neural network to provide fashion item classification information for each fashion item image information. It is a step that is configured to be output (inference step of the artificial neural network).

In the embedding step, the embedding module 13 is connected to the FC layer of the artificial neural network of the fashion item classification module 12 and receives the output vector of the FC layer as an embedding vector for the fashion item.

In the clustering step, the clustering module 14 generates cluster information by clustering an embedding vector of source data (fashion item image information of at least part of fashion content image information of a social network) previously used for learning, and the embedding module ( This is a step of mapping the embedding vector for each fashion item image information generated in 13) to cluster information.

As described above, those skilled in the art to which the present invention pertains will appreciate that the present invention can be implemented in other specific forms without changing the technical spirit or essential features thereof. Therefore, the above-described embodiments are illustrative in all respects and should be understood as non-limiting. The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present invention.

The features and advantages described herein are not all inclusive, and in particular many additional features and advantages will become apparent to those skilled in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used herein has been selected primarily for readability and for teaching purposes, and may not be chosen to describe or limit the subject matter of the invention.

The above description of embodiments of the present invention has been presented for purposes of illustration. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the above disclosure.

Therefore, the scope of the invention is not limited by the detailed description, but by any claims in the application on which it is based. Accordingly, the disclosure of the embodiments of the present invention is illustrative and does not limit the scope of the present invention described in the following claims.

1: Fashion content embedding device for user clustering
10: Fashion content receiving module
11: Fashion item detection module
12: Fashion item classification module
13: embedding module
14: clustering module
50: social network web server
100: user client

Claims (3)

Receive fashion content information of interest, which is fashion content information uploaded or input by the specific user, to a social network web server, which is a web server of a specific social network service, through a user client of a specific user, and include it in the fashion content information of interest. A fashion content receiving module that generates fashion content image information;
The fashion content receiving module receives fashion content image information, detects an area of a fashion item from the fashion content image information to generate a plurality of fashion item area information, and the fashion item area information A fashion item detection module that generates fashion item image information, which is at least part of image information of fashion content image information, as many as the number of fashion items detected in the fashion content image information; And
The fashion item detection module receives the fashion item image information and is connected to a fully connected layer of a pre-learned artificial neural network using each fashion item image information as input data, and each fashion item image An embedding module for outputting an output vector of the FC layer of the artificial neural network for information as an embedding vector for each fashion item image information;
Including,
The loss function of the artificial neural network includes a distance between the embedding vectors for each of the fashion item image information included in one of the fashion content information, and a loss value of the loss function decreases as the distance is closer. And in the learning step of the artificial neural network, the weight of the artificial neural network is learned in a direction in which the loss value decreases,
Characterized in that performing user clustering or fashion item clustering based on the embedding vector for each of the fashion item image information,
Fashion content embedding device for user clustering.
The fashion content receiving module receives fashion content information of interest, which is fashion content information uploaded or input by the specific user, to a social network web server that is a web server of a specific social network service through a user client of a specific user, A fashion content receiving step of generating fashion content image information included in the fashion content information of interest;
A fashion item detection module receives fashion content image information from the fashion content receiving module, detects an area of a fashion item from the fashion content image information, and generates a plurality of fashion item area information. A fashion item detection step of generating fashion item image information, which is image information of at least part of the fashion content image information with respect to item area information, as many as the number of fashion items detected in the fashion content image information;
The embedding module receives the fashion item image information from the fashion item detection module and is connected to a fully connected layer of a pre-learned artificial neural network using each of the fashion item image information as input data. An embedding step of outputting an output vector of the FC layer of the artificial neural network for the fashion item image information as an embedding vector for each fashion item image information;
Including,
The loss function of the artificial neural network includes a distance between the embedding vectors for each of the fashion item image information included in one of the fashion content information, and a loss value of the loss function decreases as the distance is closer. And in the learning step of the artificial neural network, the weight of the artificial neural network is learned in a direction in which the loss value decreases,
Characterized in that performing user clustering or fashion item clustering based on the embedding vector for each of the fashion item image information,
Fashion content embedding method for user clustering.
A memory module that stores a fashion content embedding program code; And
A processing module that processes the fashion content embedding program code;
Including,
The fashion content embedding program code,
Receive fashion content information of interest, which is fashion content information uploaded or input by the specific user, to a social network web server, which is a web server of a specific social network service, through a user client of a specific user, and include it in the fashion content information of interest. A fashion content receiving step of generating fashion content image information;
A fashion item image that detects an area of a fashion item from the fashion content image information to generate a plurality of fashion item area information, and is image information of at least a part of the fashion content image information for each of the fashion item area information A fashion item detection step of generating information as much as the number of fashion items detected in the fashion content image information; And
An embedding step of outputting an output vector of a fully-connected layer of the artificial neural network, which has the fashion item image information as input data, as an embedding vector for each fashion item image information;
Including program code to execute on a computer,
The loss function of the artificial neural network includes a distance between the embedding vectors for each of the fashion item image information included in one of the fashion content information, and a loss value of the loss function decreases as the distance is closer. And in the learning step of the artificial neural network, the weight of the artificial neural network is learned in a direction in which the loss value decreases,
Characterized in that performing user clustering or fashion item clustering based on the embedding vector for each of the fashion item image information,
Fashion content embedding device for user clustering.

Images