JP2023548906A - Determining visual themes within media item collections - Google Patents

Abstract

A media application determines media item clusters such that media items within each cluster have visual similarities based on pixels of images or videos from the media item collection. Media item collections are associated with user accounts. The media application selects a subset of media clusters from the corresponding media item clusters based on media items in each cluster that have a visual similarity within a similarity threshold. A media application displays a user interface that includes a subset of media clusters.

Description

REFERENCES TO RELATED APPLICATIONS This application is filed on March 11, 2021 and entitled "Determining Visual Themes from Pixels in a Media Item Collection," U.S. Provisional Application No. 63/187390 and May 17, 2021. No. 63/189,658, filed Oct. 25, 2021, and entitled "Determining Visual Themes from Pixels in a Media Item Collection," filed Oct. 25, 2021. No. 17/509,767, entitled "Determining Visual Themes in a Media Item Collection," each of which is incorporated herein in its entirety.

Background Users of devices such as smartphones or other digital cameras take and store large numbers of photos and videos in image libraries. Users utilize such libraries to recall various events such as birthdays, weddings, vacations, trips, etc. by viewing their photos and videos. A user may have a large image library containing thousands of images taken over a period of time.

The background description provided herein is for the purpose of generally presenting the context of the disclosure. To the extent described in this Background section, the work of currently named inventors is not explicitly or implicitly cited as prior art to this disclosure, as is any description that is not considered prior art at the time of filing. is not recognized.

SUMMARY A computer-implemented method includes generating vector representations of media items from a collection of media items associated with a user account using a trained machine learning model, and determining media item clusters such that media items within the clusters have visual similarity, the vector distance between the vector representations of the pair of media items is indicative of the visual similarity of the media items, and the clusters include: A vector distance between each pair of media items in the cluster is selected to be outside a visual similarity threshold, and the method further includes displaying a user interface including the subset of the media item clusters.

In some embodiments, each media item has an associated timestamp, media items obtained within a predetermined time period are associated with an episode, and selecting a subset of media item clusters based on the corresponding associated timestamps such that the corresponding media items in the subset meet a temporal diversity criterion that excludes more than a predetermined number of corresponding media items from a particular episode. In some embodiments, the method further includes excluding from the media item collection media items associated with categories in the prohibited category list before selecting the subset of media item clusters. In some embodiments, the method further includes excluding media items corresponding to categories in the prohibited category list before determining the media item cluster. In some embodiments, each media item is associated with a location, and in response to the subset of media item clusters including more than a predetermined number of media items, selecting the subset of media item clusters includes: A subset of media item clusters is based on location such that it satisfies a locational diversity criterion. In some embodiments, media item clusters are further determined based on corresponding media items associated with labels that have semantic similarity. In some embodiments, the method determines whether each media item in the subset of media item clusters is based on analyzing the likelihood that a user associated with the user account will view the media item and perform a positive action. The method further includes scoring the item and selecting the media item from the subset of the media item cluster based on the corresponding score meeting the threshold score. In some embodiments, a method includes receiving feedback from a user regarding one or more media items in a subset of media item clusters, and determining one or more media items in the subset of media item clusters based on the feedback. and changing the corresponding score of the item. In some embodiments, the feedback is an explicit action demonstrated by removing one or more media items from a subset of media item clusters from a user interface, or a corresponding media item within a subset of media item clusters. or sharing a corresponding media item in a subset of media item clusters. In some embodiments, the method includes receiving aggregate feedback from a user of an aggregate subset of media item clusters and providing the aggregate feedback to a trained machine learning model, the method comprising: determining parameters of the trained machine learning model; is updated, and the method further includes changing the media item cluster based on updating parameters of the trained machine learning model. In some embodiments, the method selects a particular media item from each cluster within the subset of media item clusters based on the particular media item containing the largest number of objects corresponding to the visual similarity. Further comprising selecting as a cover photo for each cluster within the subset of clusters. In some embodiments, the method further includes adding a title to each cluster in the subset of media item clusters based on the type of visual similarity and common expression. In some embodiments, the user interface is displayed at predetermined intervals. In some embodiments, the method further includes providing a notification to a user associated with the user account that a subset of media item clusters is available, the notification including clusters within the subset of media item clusters. Contains the title corresponding to each. In some embodiments, the method includes determining computations to be performed on individual devices to optimize the computations and creating a trained machine learning model based on the computations performed on the individual devices. and implementing the method on a plurality of devices.

In some embodiments, a method includes receiving media items from a media item collection associated with a user account as input to a trained machine learning model, and using the trained machine learning model to generate media item clusters. and generating an output image embedding of the cluster, wherein the media items in each cluster have visual similarity, and media items with visual similarity are closer to each other in vector space than media items that are dissimilar. partitioning the vector space generates media item clusters, and the method further calculates the media item clusters based on the corresponding media items in each cluster having visual similarity within a visual similarity threshold. , selecting a subset of media item clusters, and displaying a user interface including the subset of media item clusters.

In some embodiments, the functional images are removed from the media item collection before the media item collection is provided to the trained machine learning model. In some embodiments, the trained machine learning model is trained with feedback from the user including reactions to the media item set or changes to the title of the media item set.

Embodiments may further include a system that includes one or more processors and memory that stores instructions for execution by the one or more processors. The instructions include determining media item clusters such that media items in each cluster have visual similarity based on pixels of images or videos from the media item collection, and the media item collection is associated with a user account. selecting a subset of media item clusters based on corresponding media items in each cluster that are associated and have a visual similarity within a visual similarity threshold; and a user interface comprising the subset of media item clusters. including displaying. In some embodiments, each media item has an associated timestamp, media items obtained within a predetermined time period are associated with an episode, and selecting a subset of media item clusters based on the corresponding associated timestamps such that the corresponding media items in the subset meet a temporal diversity criterion that excludes more than a predetermined number of corresponding media items from a particular episode.

Embodiments may further include a non-transitory computer-readable medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform the following operations. The operations include determining media item clusters such that media items in each cluster have visual similarity based on pixels of images or videos from the media item collection, and the media item collection is associated with a user account. selecting a subset of media item clusters based on corresponding media items in each cluster that are associated and have a visual similarity within a visual similarity threshold; and a user interface comprising the subset of media item clusters. including displaying. In some embodiments, each media item has an associated timestamp, media items obtained within a predetermined time period are associated with an episode, and selecting a subset of media item clusters based on the corresponding associated timestamps such that the corresponding media items in the subset meet a temporal diversity criterion that excludes more than a predetermined number of corresponding media items from a particular episode.

The present invention advantageously uses machine learning models to identify similar images (or other media items) without having to manually identify images or manually provide categories for images (or other media items). Describe the method for identifying clusters. In this way, an improved method for classifying images or other media items can be provided. This method can, for example, provide a classification of events that more reliably reflects underlying trends in the data than would be the case with predefined classifications or categories. Furthermore, the machine learning model can advantageously reduce power consumption and increase efficiency by using a static training set and updating the event machine learning model in response to the update size being less than a threshold size. Can be done.

1 is a block diagram illustrating an example network environment in accordance with some embodiments described herein. FIG. 1 is a block diagram illustrating an example computing device in accordance with some embodiments described herein. FIG. In accordance with some embodiments, FIG. 3A is an exemplary set of various media items, each matching a particular visual theme, and FIG. A first set of media items is shown that matches a second visual theme that includes three images and a third visual theme that includes a cat in a shark stuffing in different positions. FIG. 3B is an exemplary set of various media items, each matching a particular visual theme, in accordance with some embodiments described herein. A fourth visual theme is shown in which the same object (a backpack) is seen in each image taken at a location. 2 is an example of a visual theme of natural images of different mountain ranges having both temporal and spatial diversity, in accordance with some embodiments; 2 is an example of a user interface including clusters with visual themes, in accordance with some embodiments described herein. 3 is a flowchart illustrating an example method for displaying a subset of media item clusters in accordance with some embodiments described herein. 2 is a flowchart illustrating an example method for generating embeddings of media item clusters and selecting a subset of media item clusters using a machine learning model, in accordance with some embodiments described herein.

Detailed description Network environment 100
FIG. 1 shows a block diagram of an example environment 100. In some embodiments, environment 100 includes media server 101, user device 115a, user device 115n, and network 105. Users 125a, 125n may be associated with user devices 115a, 115n, respectively. In some embodiments, environment 100 may include other servers or devices not shown in FIG. 1, or may not include media server 101. In FIG. 1 and the other figures, the letters after a reference number, such as "115a", represent a reference to the element bearing that particular reference number. A reference number in text without a letter after it, such as "115", represents a general reference to the embodiment of the element bearing that reference number.

Media server 101 may include a processor, memory, and network communication hardware. In some embodiments, media server 101 is a hardware server. Media server 101 is communicatively connected to network 105 via signal line 102 . The signal line 102 may be a wired connection such as Ethernet, coaxial cable, fiber optic cable, or a wireless connection such as Wi-Fi, Bluetooth, or other wireless technology. good. In some embodiments, media server 101 sends and receives data to and from one or more user devices 115a, 115n over network 105. Media server 101 may include media application 103a and database 199.

Media application 103a, with the user's permission, operates to determine media item clusters based on pixels of images or videos from the media item collection such that media items within each cluster have visual similarity. A media item collection may be associated with a user account. For example, one cluster may include objects with similar shapes and colors, another cluster may include parks with similar environmental attributes, and another cluster may include images of pets in different situations. But that's fine. Media application 103a selects a subset of media item clusters based on corresponding media items in each cluster that have visual similarities within a visual similarity threshold. Media application 103a displays a user interface that includes a subset of media item clusters.

In some embodiments, media application 103a includes a central processing unit (CPU), field programmable gate array (FPGA), application specific integrated circuit (ASIC), any other type of processor, or a combination thereof. It may also be implemented using hardware. In some embodiments, media application 103a may be implemented using a combination of hardware and software.

Database 199 may store media collections associated with user accounts, training sets for machine learning models, user actions related to media (viewing, sharing, annotating, etc.). Database 199 may store media items that are indexed and associated with the identity of user 125 of user device 115. Database 199 may also store social network data related to user 125, user preferences of user 125, and the like.

User device 115 may be a computing device that includes memory and a hardware processor. For example, user device 115 may be a desktop computer, a mobile device, a tablet computer, a cell phone, a wearable device, a head-mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or access network 105. It may also include other electronic devices that can be used.

In the illustrated implementation, user device 115a is connected to network 105 via signal line 108 and user device 115n is connected to network 105 via signal line 110. Media application 103 may be stored on user device 115a as media application 103b or on user device 115n as media application 103c. Signal lines 108 and 110 may be wired connections such as Ethernet, coaxial cable, fiber optic cable, or wireless connections such as Wi-Fi, Bluetooth, or other wireless technologies. You can. User devices 115a and 115n are used by users 125a and 125n, respectively. User equipment 115a, 115n in FIG. 1 is used as an example. Although FIG. 1 shows two user devices 115a and 115n, the present disclosure applies to system architectures that include one or more user devices 115.

In some embodiments, a user account includes a media item collection. For example, users can capture images and videos from their own cameras (e.g., smartphones or other cameras), upload images from digital single-lens reflex (DSLR) cameras, and view media taken and shared by other users. Add to your media item collection. Media application 103 determines media item clusters such that media items within each cluster have visual similarities based on pixels of images or videos from the media item collection. For example, FIG. 3A shows a first visual theme 300 of images (brown objects with curved shapes) that have visual similarity. Specifically, the first object is a drink with ice in a glass, the second object is a heart latte in a coffee cup, and the third object is a drink with ice in a glass, and the third object is a heart-shaped latte in a coffee cup. A bowl made of wood. Other examples are mountain ranges, natural arcs, ocean waves with people, horizontally extending parallel lines (e.g. train tracks, roads, etc.), changes over time (plant growth, movement of the sun, etc.). , painting during work), etc.

A media item cluster may include images from the same episode, for example multiple images of the same work taken by a user from different angles. For example, FIG. 3A shows a second example 325 that includes three images. The three images are the same still life photographed in different ways so that the leaves on the tree become progressively more distinct.

Media application 103 selects a subset of media item clusters based on corresponding media items in each cluster that have visual similarities within a visual similarity threshold. The visual similarity threshold may be between media items that are very similar to media items that are more similar than items that are only remotely related. For example, the theme of the first example 300 of FIG. 3A is a brown circular object. This may be in the middle of the range of similarity thresholds. Conversely, the third example 350 of FIG. 3A is a cluster of media items with the theme of cats in shark stuffing, taken at different times of day. This is a more visually similar theme. The fourth example 375 of FIG. 3B includes the theme of an orange backpack used when a person goes on different trips. Yet another example of very similar media that is closer to the threshold similarity value is that the media item is a pink flower with a slightly different shape.

When media items are not visually similar enough, it is difficult to discern themes between them, and as a result, the media items appear more like a random collection of media items than what the user wants to see. there is a possibility. In some embodiments, the media application 103 limits the number of media items to create a visual theme, such that the media item collection does not appear to be a group of all available cat images from the user's library, for example. Keep it more consistent.

Media application 103 may display a user interface that includes a subset of media item clusters. In some embodiments, media application 103 displays a user interface that includes a subset of media item clusters at predetermined intervals. For example, media application 103 may display a user interface that includes a subset of clusters on a daily, weekly, monthly, etc. basis. Media application 103 may change the frequency with which the subset of clusters is displayed based on the feedback. For example, if the user views a subset of clusters each time it becomes available, media application 103 may maintain the frequency of display, but if the user views a subset of clusters less frequently, media application 103 may The application 103 may reduce the frequency of display.

Media application 103 may provide a notification that a subset of clusters is available to a user associated with the user account, along with a title corresponding to the subset of clusters. For example, media application 103 may provide daily notifications, weekly notifications, monthly notifications, etc. to the user. In some embodiments, the user interface includes options to limit the frequency of notifications and/or display of a subset of media item clusters.

Exemplary computing device 200
FIG. 2 is a block diagram of an example computing device 200 that may be used to implement one or more features described herein. Computing device 200 may be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 200 is user device 115 used to run media application 103. In another example, computing device 200 is media server 101. In yet another example, media application 103 is partially located on user device 115 and partially on media server 101.

One or more of the methods described herein can be implemented as a standalone program running on any type of computing device, as a program running on a web browser, as a mobile computing device (e.g., a mobile phone, a smartphone, Mobile applications (apps) running on tablet computers, wearable devices (e.g. watches, armbands, jewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, head-mounted displays), laptop computers) It can be executed as In the primary example, all calculations are performed in a mobile application on a mobile computing device. However, a client/server architecture can be used. For example, a mobile computing device transmits user input data to a server device and receives and outputs (eg, displays) final output data from the server. In another example, computations may be shared between a mobile computing device and one or more server devices.

In some embodiments, computing device 200 includes a processor 235, memory 237, I/O interface 239, display 241, camera 243, and storage 245. Processor 235 may be connected to bus 218 via signal line 222. Memory 237 may be connected to bus 218 via signal line 224. I/O interface 239 may be connected to bus 218 via signal line 226. Display 241 may be connected to bus 218 via signal line 228. Camera 243 may be connected to bus 218 via signal line 230. Storage device 245 may be connected to bus 218 via signal line 232.

Processor 235 may be one or more processors and/or processing circuits for executing program code and controlling basic operations of computing device 200. "Processor" includes any suitable hardware system, mechanism or component for processing data, signals or other information. A processor may include a general-purpose central processing unit (CPU) having one or more cores (e.g., a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., having a multi-processor configuration), a graphics processing unit ( GPUs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), complex programmable logic devices (CPLDs), specialized circuits to achieve functionality, and to perform processing based on neural network models. It may include a dedicated processor, a neural circuit, a system with a processor optimized for performing matrix calculations (eg, matrix multiplication), or other systems. In some implementations, processor 235 can include one or more coprocessors to perform neural network processing. In some implementations, processor 235 may be a processor that generates probabilistic output by processing data. For example, the output produced by processor 235 may be inaccurate or may be accurate to within an output expected value. Processing need not be limited to a particular geographic location or limited in time. For example, a processor may perform functions in real time, off-line, or in batch mode. Portions of the processing may be performed by different (or the same) processing systems at different times and different locations. A computer may be any processor in communication with memory.

Memory 237 is typically provided within computing device 200 for use by processor 235 and includes any suitable processor-readable storage medium for storing instructions for execution by a processor or set of processors. For example, it may be random access memory (RAM), read only memory (ROM), electrically erasable read only memory (EEPROM), or flash memory. Memory 237 may be located separately from and/or integrated with processor 235. Memory 237 can store software (including media applications 103) executed on computing device 200 by processor 235.

Memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, camera applications, image library applications, image management applications, image gallery applications, media display applications, communication applications, web hosting engines or applications, mapping applications, media sharing applications, and the like. One or more methods disclosed herein can operate, for example, as a standalone computer program that can run on any type of computing device, as a web application with web pages, on a mobile computing device, etc. may run as a mobile application (“app”) on a number of environments and platforms.

Application data 266 may be data generated by other applications 264 or hardware on computing device 200. For example, application data 266 may include images captured by camera 243, user actions identified by other applications 264 (eg, social networking applications), and the like.

I/O interface 239 may provide functionality that allows computing device 200 to interface with other systems and devices. The interfacing device may be included as part of computing device 200 or may be separate but in communication with computing device 200. For example, network communication devices, storage devices (eg, memory 237 and/or database 199), and input/output devices can communicate via I/O interface 239. In some embodiments, the I/O interface 239 includes interface devices, such as input devices (keyboards, pointing devices, touch screens, microphones, cameras, scanners, sensors, etc.) and/or output devices (display devices, speaker devices, etc.). , printer, monitor, etc.). For example, when a user provides touch input, I/O interface 239 sends data to media application 103.

Some example interface connection devices that can be connected to I/O interface 239 display user interfaces for the content described herein, such as images, video, and/or output applications, and It may also include a display 241 that can be used to receive touch (or gesture) input. For example, display 241 may be used to display a user interface that includes a subset of media item clusters. Display 241 can be any suitable display device, such as a liquid crystal display (LCD), light emitting diode (LED), or plasma display screen, cathode ray tube (CRT), television, monitor, touch screen, three-dimensional display screen, or the like. may include a visual display device. For example, display 241 may be a flat display screen on a mobile device, multiple display screens embedded in an eyeglass form factor or headset device, or a monitor screen on a computing device.

Camera 243 may be any type of image capture device capable of capturing images and/or video. In some embodiments, camera 243 captures images or videos that I/O interface 239 sends to media application 103.

Storage device 245 stores data related to media application 103. For example, storage device 245 may store a media item collection associated with a user account, a subset of media clusters, a training set for a machine learning model, and the like. In embodiments where media application 103 is part of media server 101, storage 245 is the same as database 199 of FIG.

Exemplary media application 103
FIG. 2 depicts an example media application 103. Media application 103 includes a filtering module 202, a clustering module 204, a machine learning module 205, a selection module 206, and a user interface module 208. In some embodiments, media application 103 uses either clustering module 204 or machine learning module 205.

Filtering module 202 excludes media items from the media item collection that correspond to categories in the prohibited category list. In some embodiments, filtering module 202 includes a set of instructions executable by processor 235 to exclude media items that correspond to categories in the prohibited category list. In some embodiments, filtering module 202 may be stored in memory 237 of computing device 200 and may be accessible and executable by processor 235.

In some embodiments, filtering module 202 excludes media from the media item collection before clustering module 204 performs clustering. In an alternative embodiment, filtering module 202 excludes media from the media item collection after clustering module 204 performs clustering. For example, filtering module 202 excludes media items associated with visual similarity to categories from the prohibited category list. The prohibited category list may include media items that are taken not for photographic value but as functional images, such as receipt images, document images, parking meter images, screenshot images, etc.

In some embodiments where media application 103 includes machine learning module 205, filtering module 202 excludes functional images from the media item collection before the media item collection is provided to the machine learning model. For example, filtering module 202 filters out receipts, instructions, documents, and screenshots before the media item collection is provided to the machine learning model.

Clustering module 204 determines media item clusters such that media items within each cluster have visual similarities based on pixels of images or videos from the media item collection. In some embodiments, clustering module 204 includes a set of instructions executable by processor 235 to generate media item clusters. In some embodiments, clustering module 204 may be stored in memory 237 of computing device 200 and may be accessible and executable by processor 235.

In some embodiments, clustering module 204 accesses a media item collection associated with a user account, such as a library associated with the user. If filtering module 202 excludes a media item, clustering module 204 accesses media item collections that do not include media items corresponding to the prohibited category list. Clustering module 204 may determine media item clusters such that media items within each cluster have visual similarities based on pixels of images or videos from the media item collection. In some embodiments, clustering uses an N-dimensional Gaussian diversity function to determine visual similarity.

In some embodiments, machine learning module 205 includes a machine learning model trained to generate output image embeddings of media clusters such that media items within each cluster have visual similarities. In some embodiments, machine learning module 205 includes a set of instructions executable by processor 235 to generate image embeddings. In some embodiments, machine learning module 205 may be stored in memory 237 of computing device 200 and may be accessible and executable by processor 235.

In some embodiments, machine learning module 205 may use vectors (embeddings) in a multidimensional feature space to determine visual similarity of clusters. Images with similar features may have similar feature vectors. For example, vector distances between feature vectors of images with similar features may be smaller than vector distances between dissimilar images. The feature space is based on various factors of the image, such as the depicted subject matter (objects detected from the image), image composition, color information, image orientation, image metadata, and specific objects recognized from the image ( For example, with user permission, it may be a function such as a known face).

In some embodiments, training may be performed using supervised learning. In some embodiments, machine learning module 205 includes a set of instructions executable by processor 235. In some embodiments, machine learning module 205 may be stored in memory 237 of computing device 200 and may be accessible and executable by processor 235.

In some embodiments, machine learning module 205 may use training data (obtained with permission for training) to generate a trained model, specifically a machine learning model. For example, the training data may include ground truth data in the form of media clusters associated with descriptions of visual similarity of the clusters. In some embodiments, the visual similarity description may include user feedback as to whether the clusters are related and include distinct themes. In some embodiments, visual similarity descriptions may be added automatically by image analysis. Training data may be obtained from any source, such as a data repository designated for training, data that has been granted permission for use as training data for machine learning.

In some embodiments, the training data is synthetic data generated for training purposes, e.g., data that is not based on activity in the situation being trained, e.g., generated from simulations or computer-generated images/videos. It may also include data that was In some embodiments, machine learning module 205 uses unedited/transferred weights obtained from another application. For example, in these embodiments, the trained model may be generated on a different device and provided as part of the media application 103, for example. In various embodiments, the trained model has a model structure or morphology (e.g., defining the number and type of neural network nodes, connections between nodes, and organization of nodes into layers) and associated weights. It may be provided as a data file containing. Machine learning module 205 can read the trained model data file and implement a neural network including node connections, layers, and weights based on the model structure or morphology specified in the trained model.

Machine learning module 205 produces a trained model, referred to herein as an event machine learning model. In some embodiments, machine learning module 205 identifies one or more features within the input media item by applying an event machine learning model to data, such as application data 266 (e.g., input media); The system is configured to generate feature vectors (embeddings) representing media items. In some embodiments, machine learning module 205 may include software code executed by processor 235. In some embodiments, machine learning module 205 can specify circuitry (e.g., a programmable processor, field programmable gate array (FPGA)) that enables processor 235 to apply the machine learning model. In some embodiments, machine learning module 205 may include software instructions, hardware instructions, or a combination thereof. In some embodiments, machine learning module 205 may provide an application programming interface (API). Operating system 262 and/or other applications 264 can utilize this API to output image embeddings of media clusters by calling machine learning module 205 and, for example, applying a machine learning model to application data 266. can. In some embodiments, media items that match visual similarity are closer to each other in vector space than dissimilar images. Therefore, media item clusters are generated by partitioning the vector space.

In some embodiments, the machine learning model is a classifier that receives a media item collection as input. Examples of classifiers include neural networks, support vector machines, k-nearest neighbors, logistic regression, Naive Bayes, decision trees, perceptrons, and the like.

In some embodiments, a machine learning model may include one or more model forms or structures. For example, the model form or structure may implement any type of neural network, e.g. a linear network, multiple layers (e.g. "hidden layers" between input and output layers, each layer being a linear network). A deep neural network that a sequence-to-sequence neural network (e.g., a network that receives sequential data as input, such as words in a sentence, or frames in a video, and produces a sequence of results as output); network).

The model morphology or structure can specify the connections between various nodes and the organization of nodes organized into layers. For example, a first layer (eg, input layer) node may receive data as input data or application data 266. For example, when a machine learning model is used to analyze an input image, eg, a first image, associated with a user account, such data may include, eg, one or more pixels per node. Subsequent intermediate layers can receive as input the outputs of the nodes of previous layers according to the connections specified in the model form or structure. These layers are sometimes called hidden layers. The final layer (eg, output layer) produces the output of the machine learning model. For example, this output may be an image embedding of the media cluster. In some embodiments, the model morphology or structure specifies the number and/or type of nodes in each layer.

The features output by the machine learning module 205 include the subject (e.g., sunset vs. a specific person), the colors present in the image (green hills vs. blue lake), color balance, illumination source, angle and intensity (e.g., three This may include the location of objects in the image (adhering to the segmentation method), the relative location of objects (eg, depth of field), location of the shot, focus (foreground vs. background), or shadows. While the aforementioned features are human-understandable, the output features may be embedded or other numerical values that are representative of the image and not human-analyzable (e.g., individual feature values are (may not correspond to specific features such as the color present, the location of the object, etc.) However, the trained model is robust to images, so it outputs similar features for similar images and different features for significantly different images.

In some embodiments, the model form is a CNN that includes network layers, each network layer extracting image features at a different level of abstraction. The CNN used to identify features within an image may also be used to classify the image. The model architecture may include layer combinations and orders consisting of multidimensional convolution, average pooling, max pooling, activation functions, regularization, regularization, and other layers and modules that are actually used in applied deep neural networks. good.

In different embodiments, a machine learning model can include one or more models. One or more models may include multiple nodes arranged in layers according to model structure or morphology. In some embodiments, a node may be a memoryless computational node configured to process a unit of input and produce a unit of output, for example. The computations performed by a node may include, for example, multiplying each of a plurality of node inputs by a weight, obtaining a weighted sum, and adjusting the node output by adjusting the weighted sum with a bias or intercept value. It may also include the step of generating. For example, machine learning module 205 can adjust each weight based on the feedback in response to automatically updating one or more parameters of the machine learning model.

In some embodiments, the calculation performed by the node may also include applying a step/activation function to the adjusted weighted sum. In some embodiments, the step/activation function may be a non-linear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, computations performed by multiple nodes may be performed using, for example, multiple processor cores of a multi-core processor, individual processing units of a graphical processing unit (GPU), or dedicated neural circuits. may be used and executed in parallel. In some embodiments, a node may include memory. A node may, for example, remember one or more previous inputs and use one or more previous inputs in processing subsequent inputs. For example, a node having memory may include a long short term memory (LSTM) node. LSTM nodes may use memory to maintain state that allows the node to operate like a finite state machine (FSM). Models containing such nodes are useful when processing sequential data, such as words in a sentence or paragraph, a sequence of images, frames in a video, dialogue, or other audio. would be useful. For example, the heuristics-based model used for the gating model may remember one or more previously generated features for previous images.

In some embodiments, the machine learning model may include embeddings or weights for individual nodes. For example, a machine learning model may be initialized as multiple nodes organized into layers as specified by model morphology or structure. At the time of initialization, respective weights may be applied to the connections between each pair of nodes connected according to the model form, eg, each pair of nodes in successive layers of the neural network. For example, each weight may be randomly assigned or initialized to a default value. The training set of media clusters can then be used, for example, to train a machine learning model to generate results. In some embodiments, a subset of the entire architecture may be reused from other machine learning applications as a transfer learning method to leverage pre-trained weights.

For example, training can include applying supervised learning techniques. In supervised learning, the training data includes multiple inputs (e.g., media items from a media item collection associated with a user account) and an expected output (e.g., an image embedding of a media cluster) corresponding to each input. be able to. For example, automatically adjusting weight values based on a comparison of the machine learning model's output and the expected output to increase the probability that the machine learning model will produce the expected output when provided with similar inputs.

In some embodiments, training can include applying unsupervised learning techniques. In unsupervised learning, only input data (e.g., media items from a media item collection associated with a user account) may be provided, and the machine learning model uses, for example, image features to distinguish between the data. It may be trained to cluster into multiple groups.

In various embodiments, a trained model includes a set of weights that correspond to a model structure. In embodiments that omit the training set, machine learning module 205 may generate a machine learning model based on prior training, such as by the developer of machine learning module 205 or a third party. In some embodiments, a machine learning model may include a set of fixed weights downloaded from a server that provides weights.

In some embodiments, machine learning module 205 may be implemented in an offline manner. Implementing machine learning module 205 may include using a static training set that does not include updates when data in the static training set changes. This advantageously results in increased efficiency of processing performed by computing device 200 and reduced power consumption of processing device 200. In these embodiments, a machine learning model may be generated in a first stage and provided as part of machine learning module 205. In some embodiments, small updates to a machine learning model may be implemented in an online manner where updates to training data are included as part of training the machine learning model. A small update is an update that has a size smaller than a threshold size. The size of an update is related to the number of variables in the machine learning model that are affected by the update. In such embodiments, an application invoking machine learning module 205 (e.g., operating system 262, one or more other applications 264, etc.) utilizes image embedding of media item clusters to identify visually similar clusters. can be identified. Machine learning module 205 may also generate system logs periodically, for example, every hour, every month, or every three months. System logs may be used to update machine learning models, for example to update embeddings of machine learning models.

In some embodiments, machine learning module 205 may be implemented in a manner compatible with the particular configuration of computing device 200 on which machine learning module 205 is executed. For example, machine learning module 205 can determine a computational graph that utilizes available computational resources, such as processor 235. If the machine learning module 205 is implemented as a distributed application on multiple devices, e.g., if the media server 101 includes multiple media servers 101, the machine learning module 205 may be implemented on each individual device to optimize computation. It is possible to determine the calculations to be performed on the In another example, when machine learning module 205 determines that processor 235 includes a GPU having a certain number (e.g., 1000) of GPU cores, machine learning module 205 determines that processor 235 includes a (as a process or a thread).

In some embodiments, machine learning module 205 may implement a set of trained models. For example, an event machine learning model can include multiple trained models, each applicable to the same input data. In these embodiments, machine learning module 205 may select a particular trained model based on, for example, available computational resources, success rate using previous inferences, etc.

In some embodiments, machine learning module 205 can run multiple trained models. In these embodiments, the machine learning module 205 may, for example, use majority voting to score the outputs obtained by applying each trained model, or by selecting one or more particular outputs. Outputs can be combined. In some embodiments, such selectors are part of the model itself and serve as a connection layer between trained models. Additionally, in these embodiments, machine learning module 205 applies a time threshold (e.g., 0.5 ms) for applying individual trained models and utilizes only individual outputs available within the time threshold. can do. Outputs that are not received within a time threshold are not utilized and may, for example, be discarded. For example, such an approach may be appropriate when there is a time limit specified between invoking machine learning module 205, such as by operating system 262 or one or more other applications 264. In this way, we limit the maximum time that machine learning module 205 takes to perform a task, e.g., to identify one or more features of an input media item and generate a feature vector (embedding) representing the media item. The responsiveness of the media application 103 can be improved because the machine learning module 205 can provide the best classification in real time.

In some embodiments, machine learning module 205 receives feedback. For example, machine learning module 205 can receive feedback from a user or set of users via user interface module 208. When a user provides feedback, machine learning module 205 provides the feedback to the machine learning model, and machine learning model 205 uses the feedback to update parameters of the machine learning model to create media item clusters. Fix the output image embedding of . When a set of users provides feedback, machine learning module 205 provides aggregate feedback to the machine learning model, and machine learning model 205 uses the aggregate feedback to update parameters of the machine learning model to Change the output image embedding for item clusters. For example, aggregate feedback may include a subset of media clusters and a user's response to the subset of media clusters. The user's responses can be to view only one image and refuse to view the rest of the media, view all of the corresponding media items in the subset, share the corresponding media item, and the corresponding media item. provide approval or disapproval instructions (e.g., thumbs up/thumbs down, likes, +1, etc.), remove/add individual media items from a subset of media item clusters, change titles, etc. include. Machine learning module 205 can modify media clusters based on updating parameters of the machine learning model.

In some embodiments, the machine learning model is trained with user feedback, including reactions to a subset of clusters and changing the title of one of the clusters within the subset. Machine learning module 205 provides feedback to the machine learning model to identify clusters of media items that have a certain type of visual similarity (e.g., are visually similar but not the type of media that the user would like to see). The parameters are changed to exclude images of ocean waves and images of surfers on waves taken at different times and/or different locations.

Selection module 206 selects a subset of media item clusters based on visual similarity determined by clustering module 204. In some embodiments, selection module 206 includes a set of instructions executable by processor 235 to select a subset of media item clusters. In some embodiments, selection module 206 may be stored in memory 237 of computing device 200 and may be accessible and executable by processor 235.

In some embodiments, selection module 206 selects a subset of media item clusters whose media items have visual similarity within a visual similarity threshold. For example, the range may be between 0.05 and 0.3 out of 0 and 4. Other ranges and scales are also possible. A subset of media item clusters within a visual similarity threshold may be considered to have visual themes that are recognized as related and cohesive.

In some embodiments, if the number of media item clusters exceeds a predetermined number (e.g., greater than 15 media items), selection module 206 imposes additional restrictions when selecting the subset of media item clusters. Can be done. For example, the selection module 206 may identify a timestamp associated with each media item and determine whether multiple media items (e.g., media items associated with the same time period and same location) are associated with the same episode based on the timestamp. selecting a subset of media item clusters based on the timestamps to which the subset of media item clusters are associated, such that the subset of media item clusters satisfies a temporal diversity criterion that identifies that a media item is present in the media and excludes more than a predetermined number of media items from a particular episode; (i.e., by imposing temporal diversity by selecting a subset of media item clusters based on associated timestamps such that no more than a certain number (e.g., 3) of media items are associated with the same episode) be able to. This avoids media item clusters that are too similar and potentially overlapping, since a user may take multiple images of an object at the same location at the same time of day. This also avoids a situation where a user takes the same image and edits it for posting to another photo sharing application, for example. Selection module 206 may use temporal diversity to select a subset of clusters that represent the progress of an object over a period of time. For example, a cluster may include different images of a child taken at different time periods to show the child growing large, or different images of plants from seedlings to flowering bushes.

In some embodiments, selection module 206 imposes geographic diversity on a subset of media clusters. For example, the selection module 206 can identify a location associated with each media item such that the number of corresponding media items available for the cluster exceeds a predetermined number (e.g., exceeds 10 media items). If so, selection module 206 selects a subset of media item clusters based on location such that the subset of media item clusters satisfies a locational diversity criterion. FIG. 4 includes an example 400 of visual themes for natural images of different mountain ranges that have both temporal diversity because the images were taken in different months and years, and locational diversity because the images were taken in different locations. . There are two types of diversity in images, but hidden similarities emerge through visual themes.

In some embodiments, selection module 206 imposes a semantic theme on a subset of media clusters. Selection module 206 may identify labels associated with images and group subsets of media item clusters based on corresponding media items having the same or similar labels. For example, selection module 206 may select a subset of media item clusters ranging from puppies to adult dogs using labels that identify depictions of dogs in images. In some embodiments, the media application 103 combines the semantic theme of the Golden Gate Bridge with the visual theme of other bridges that are visually similar to the golden color of the Golden Gate Bridge.

In some embodiments, the selection module 206 selects within the subset of media item clusters based on analyzing the likelihood that a user associated with the user account will view the media item and perform a positive action. Score each media item. Affirmative actions can include viewing the subset, sharing the subset, ordering prints from the subset, and so on. The selection module 206 selects media items that are more interesting if the subject matter is more interesting, for example, if the subject matter includes babies, people the user knows, places the user has visited, etc. can be scored as related to the likelihood of performing. Conversely, selection module 206 may determine that the user is less likely to perform a positive action associated with a particular object, eg, a static object such as a bunk bed. In some embodiments, the selection module 206 scores the subset of media item clusters based on personal information associated with the user or based on aggregate information regarding the user's general reaction to the media. In some embodiments, the selection module 206 selects a media item that is too blurry because the quality of the media item reduces the likelihood that a user associated with the user account will perform a positive action associated with the media item. Score media items based on their quality.

Selection module 206 may select media items within the subset of media item clusters if the score corresponding to each media item satisfies a threshold score. In some embodiments, the threshold score is a static value that is the same for all users. In some embodiments, the threshold score is user-specific. In some embodiments, the threshold score is specified by the user.

Once the selection module 206 determines the subset of clusters, it can direct the user interface module 208 to display a user interface that includes the subset of clusters. In some embodiments, a user may provide feedback related to a subset of clusters. For example, a user can view the subset, provide indications of approval of the subset, share the subset, and order printing of photos from the subset.

In some embodiments, selection module 206 receives feedback and changes scores corresponding to the subset of media item clusters based on the feedback. For example, feedback may be indicated by explicit action indicated by deleting a subset of clusters from a user interface, or by one or more of viewing a subset of clusters, viewing a subset of clusters, or sharing a subset of clusters. This can include implicit actions that have been taken. In some embodiments, selection module 206 can identify patterns within the feedback. For example, if positive feedback occurs when objects in a cluster are of a particular type (baby, family, tree, etc.), selection module 206 scores a subset of clusters to include objects of similar types. can be corrected. In another example, the pattern may indicate that the user prefers themes with lower visual similarities than higher visual similarities, and selection module 206 selects themes with lower visual similarities. The score can be modified to select more frequently.

In some embodiments, selection module 206 can receive feedback from a set of users using media application 103 and aggregate the feedback. For example, the selection module 206 can create aggregate feedback of the user for a subset of media clusters and can change the scoring based on the aggregate feedback.

User interface module 208 generates a user interface. In some embodiments, user interface module 208 includes a set of instructions executable by processor 235 to generate a user interface. In some embodiments, user interface module 208 may be stored in memory 237 of computing device 200 and accessible and executable by processor 235.

User interface module 208 displays a user interface that includes a subset of media clusters. FIG. 5 illustrates an example user interface 500 that includes clusters 505 with visual themes, in accordance with some embodiments described herein. In this example, cluster 505 is displayed at the top of the user interface, along with a group of recent highlights and a group of images from a year ago. User interface 500 also includes an image taken yesterday (March 9th) in San Francisco.

In some embodiments, user interface module 208 suggests subsets of clusters and generates a user interface for viewing, editing, and sharing media. As shown in FIG. 5, the user interface may include, for example, a cluster located at the top of the user interface, and when the user selects an image, the user interface includes options to edit or share the image.

In some embodiments, in response to a user selecting a cluster in the user interface, user interface module 208 displays corresponding media items from the cluster at predetermined intervals. For example, user interface module 208 may display each media item for 2 seconds, 3 seconds, etc.

In some embodiments, user interface module 208 provides cover photos for a subset of clusters. The cover photo may be the newest photo, the photo with the highest score, etc. In some embodiments, user interface module 208 selects a particular media item from each cluster within the subset of media item clusters based on the particular media item that contains the greatest number of objects corresponding to the visual similarity. , as the cover photo for each cluster in the subset of media item clusters. For example, a cluster may have a visual theme of a group of people skiing, and the user interface module 208 displays a cover that shows an image depicting the highest number of people from the cluster of people skiing. Photos may be selected into clusters. In another example, if the cluster has a visual theme of people engaging in outdoor aquatic activities, the user interface module 208 may indicate that the people are near water rather than in water (e.g., building sandcastles, or images of people surfing are the most representative cover media item compared to other images where people are not engaged in more vigorous outdoor activities (e.g. tanning along the water) The user interface module 208 may also select the cover photo based on having the highest visual quality (e.g., clear, high resolution, unblurred, well exposed, etc.) among the clusters. You can also choose.

In some embodiments, user interface module 208 adds a title to each cluster in the subset of media item clusters based on the type of visual theme and/or template representation. For example, the title could be the action that occurred in the image (e.g. "surf's up" for the ocean cluster, "into the blue" for the sky cluster, "on the road" for the road cluster, "stairway to heaven), food metaphors (e.g., "mixed nuts", "smorgasbord", "mixed bag", "goody bag", "wine flight", "cheese pairing", "sampler", "treasure trove", "overlooked") photo trails (e.g., "photo detective," "photo mystery," "mystery photos," "photo sphinx"), creative combinations, correlations (e.g., connections, photo, photo club, photo weaving, pattern, coincidence, cause and effect, slot machine, one of these is the same as the other, similar), title referring to a pattern (e.g. beta pattern, pattern Hunter, small pattern, pattern portal, connect the dots, image pattern, photo pattern), synonyms of patterns, such as themes (e.g. photo story, photo story, photo story, story of two photos, photo theme, lucky theme) , a set (e.g., photo set, surprise set), or a match (e.g., memory match), an onomatopoeic word (e.g., zigzag, boom, pan-pan, bang, bang, photo, photobish), a verb (e.g., "look what we found ``in the couch cushions'', ``look what appeared'', ``help us sleuth'', ``will it blend'', ``time flies'', ``some things never change''), or a high confidence score that the guess is correct. It can refer to the title of the cluster (eg, Magical Pattern, Light Feeling) that refers to the connections it will have. In some embodiments, the template expression may be a funny or common expression that is more colloquial and appealing than simply adding a title such as "Birthdays 1997-2001." In some embodiments, the user interface module 208 identifies a general title such as "Look what we found" and a theme such as "your orange backpack brought you far" with respect to the fourth example 375 of FIG. 3B. It can include a title that includes a subtitle.

In some embodiments, user interface module 208 provides a notification to a user associated with the user account that a subset of the clusters is available for viewing. User interface module 208 may provide notifications on a regular basis, such as daily, weekly, monthly, etc. In some embodiments, if the user no longer views notifications when they are provided daily (weekly, monthly, etc.), user interface module 208 may generate notifications less frequently. User interface module 208 may additionally provide a notification that includes a title corresponding to the subset of clusters.

Exemplary Flowchart FIG. 6 is a flowchart illustrating an example method 600 for displaying a subset of media item clusters, according to some embodiments. The method illustrated in flowchart 600 may be performed by computing device 200 of FIG. 2.

Method 600 may begin at block 602. At block 602, a request is generated for access to a media item collection associated with a user account. In some embodiments, the request is generated by user interface module 208. Block 604 may be executed after block 602.

At block 604, permission interface elements are displayed. For example, user interface module 208 can display a user interface that includes a permission interface element for requesting that a user provide permission to access a media item collection. Block 606 may be executed after block 604.

At block 606, it is determined whether permission has been granted by the user to access the media item collection. In some embodiments, block 606 is performed by user interface module 208. If the user has not provided permission, the method ends. If the user provides permission, block 606 may be followed by block 608.

At block 608, media item clusters are determined such that media items within each cluster have visual similarity based on pixels of images or videos from the media item collection. Media item collections are associated with user accounts. In some embodiments, block 606 is performed by clustering module 204. Block 610 may be executed after block 608.

At block 610, a subset of media item clusters is selected based on corresponding media items in each cluster that have visual similarities within a visual similarity threshold. In some embodiments, block 610 is performed by selection module 206. Block 612 may be executed after block 610.

At block 612, a user interface is displayed that includes a subset of media clusters. In some embodiments, block 610 is performed by user interface module 208.

FIG. 7 is a flowchart illustrating an example method 700 for generating embeddings for media item clusters and selecting a subset of media item clusters using a machine learning model, according to some embodiments. The method illustrated in flowchart 700 may be performed by computing device 200 of FIG.

Method 700 may begin at block 702. At block 702, a request is generated for access to a media item collection associated with a user account. In some embodiments, the request is generated by user interface module 208. Block 704 may be executed after block 702.

At block 704, permission interface elements are displayed. For example, user interface module 208 can display a user interface that includes a permission interface element for requesting that a user provide permission to access a media item collection. Block 706 may be executed after block 704.

At block 706, it is determined whether permission has been granted by the user to access the media item collection. In some embodiments, block 706 is performed by user interface module 208. If the user has not provided permission, the method ends. If the user provides permission, block 706 may be followed by block 708.

At block 708, the trained machine learning model receives as input media items from a media item collection associated with the user account. In some embodiments, block 708 is performed by machine learning module 205. Block 710 may be performed after block 708.

At block 710, the trained machine learning model generates an output image embedding of the media item cluster. Media items in each cluster have visual similarity, and media items that have visual similarity are dissimilar in vector space, such that media items with visual similarity generate media item clusters by partitioning the vector space. Closer to each other than the items. In some embodiments, block 710 is performed by machine learning module 205. Block 712 may be executed after block 710 .

At block 712, a subset of media item clusters is selected based on corresponding media items in each cluster that have visual similarities within a visual similarity threshold. In some embodiments, block 712 is performed by machine learning module 205. Block 714 may be executed after block 712.

At block 714, a user interface is displayed that includes the subset of media item clusters. In some embodiments, block 714 is performed by user interface module 208.

In addition to the above description, the systems, programs, or features described herein may provide user information (e.g., information about the user's media items such as photos or videos, the user's interactions with media applications that display the media items, the user's information about your social networks, your social behavior or activities, your occupation, your preferences for viewing image-based creations, your settings for hiding people or pets, your user interface preferences, or your current location. The user may be given controls to choose whether and when to enable the collection of content or information () and to send content or information from the server. Additionally, certain data may be processed in one or more ways to remove identifiable personal information prior to storage or use. For example, a user's ID can be processed so that the user's personal information cannot be identified. Also, when obtaining location information (eg, city, zip code, or state level), the user's geographic location can be generalized such that the user's location cannot be determined. Thus, the user can control what user information is collected, how the information is used, and what information is provided to the user.

In the above description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various described embodiments may be practiced without these specific details. In other instances, structures and devices are shown in block diagrams in order to avoid obscuring the description. For example, embodiments may be described above with primarily reference to a user interface and specific hardware. However, embodiments are applicable to any type of computing device that can receive data and commands, and any peripheral device that provides services.

Reference herein to "some embodiments" or "some instances" means that a particular feature, structure, or characteristic described in connection with the embodiment or instance is present in at least one implementation of the description. It means that it can be included in the form. The appearances of the phrase "in some embodiments" in various places in this specification are not necessarily all referring to the same embodiment.

Some portions of the detailed descriptions presented above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally conceived herein to be a series of consistent steps that lead to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, but not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to such data as bits, numbers, elements, symbols, characters, terms, numbers, or the like.

It should be understood that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless stated otherwise or as is clear from the discussion, any discussion that utilizes terms including "process," "compute," "compute," "determine," or "display" throughout the description , computer system Refers to the operations and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical quantities in memory, registers, or other information storage, transmission, or display devices.

Embodiments herein also relate to a processor for performing one or more steps of the methods described above. The processor may be a dedicated processor that is selectively activated or reconfigured by a computer program stored in the computer. Such computer programs may be of any kind including, but not limited to, optical disks, ROMs, CD-ROMs, magnetic disks, RAM, EPROMs, EEPROMs, magnetic or optical cards, flash memory including USB keys with non-volatile memory. The information may be stored on non-transitory computer readable storage media including disks or any type of medium suitable for storing electronic instructions, each connected to a computer system bus.

This specification may include some entirely hardware embodiments, some entirely software embodiments, or some embodiments that include both hardware and software elements. In some embodiments, the present specification is implemented in software, including but not limited to firmware, resident software, microcode, and the like.

Additionally, the instructions may take the form of a computer program product accessible from a computer usable or computer readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable medium is any medium that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, machine, or device. The device can be

A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. Memory elements include local memory used during the actual execution of program code, mass storage, and at least some program code to reduce the number of times the code must be retrieved from mass storage during execution. and a cache memory that provides temporary storage of the information.

Claims (20)

A computer-implemented method, the method comprising:
generating a vector representation of a media item from a media item collection associated with a user account using the trained machine learning model;
determining media item clusters based on the vector representations of the media items such that the media items in each cluster have a visual similarity, and the vector distance between the vector representations of a pair of media items is , the visual similarity of the media items is selected such that the vector distance between each pair of media items in the cluster is outside a visual similarity threshold; Furthermore,
selecting a subset of the media item clusters based on corresponding media items in each cluster having the visual similarity within a visual similarity threshold;
displaying a user interface including the subset of the media item clusters. Each media item has an associated timestamp;
the media items obtained within a predetermined time period are associated with an episode;
Selecting the subset of the media item clusters includes a temporal diversity in which the corresponding media items in the subset of the media item clusters exclude more than a predetermined number of the corresponding media items from a particular episode. 2. The method of claim 1, wherein the method is performed on the basis of corresponding associated timestamps to meet the criteria. 2. The method of claim 1, further comprising excluding from the media item collection media items associated with categories in a prohibited category list before selecting the subset of media item clusters. The method of claim 1, further comprising excluding media items corresponding to categories in a prohibited category list before determining the media item clusters. Each media item is associated with a location and
Selecting the subset of media item clusters in response to the subset of media item clusters including more than a predetermined number of media items, wherein the subset of media item clusters meets a spatial diversity criterion. 2. The method of claim 1, wherein the method is based on a location that satisfies the following. The method of claim 1, wherein the media item clusters are further determined based on the corresponding media items associated with labels that have semantic similarity. scoring each media item in the subset of the media item cluster based on analyzing the likelihood that a user associated with the user account will perform a positive action in reference to the media item; ,
2. The method of claim 1, further comprising: selecting the media item from the subset of media item clusters based on a corresponding score that meets a threshold score. receiving feedback from a user regarding one or more media items within the subset of media item clusters;
8. The method of claim 7, further comprising changing a corresponding score of the one or more media items within the subset of media item clusters based on the feedback. The feedback may include explicit action demonstrated by removing the one or more media items from the subset of media item clusters from the user interface, or the corresponding media within the subset of media item clusters. 9. The method of claim 8, comprising implicit behavior indicated by one or more of viewing an item or sharing the corresponding media item in the subset of the media item cluster. receiving aggregate feedback from a user of an aggregate subset of media item clusters;
providing the aggregate feedback to the trained machine learning model, parameters of the trained machine learning model are updated, the method further comprising:
The method of claim 1, further comprising changing the media item cluster based on updating the parameters of the trained machine learning model. From each cluster within the subset of media item clusters, select the particular media item from each cluster within the subset of media item clusters based on the particular media item that contains the maximum number of objects corresponding to the visual similarity. 2. The method of claim 1, further comprising selecting as a cover photo for each cluster of . 2. The method of claim 1, further comprising adding a title to each cluster in the subset of media item clusters based on visual similarity type and common representation. 2. The method of claim 1, wherein the subset of media item clusters is displayed on the user interface at predetermined intervals. further comprising providing a notification to a user associated with the user account that the subset of media item clusters is available;
The method of claim 1, wherein the notification includes a title corresponding to each of the clusters in the subset of media item clusters. determining calculations to be performed on individual devices to optimize calculations;
2. The method of claim 1, further comprising: implementing the trained machine learning model on a plurality of devices based on the calculations performed on the individual devices. A computer-implemented method, the method comprising:
receiving media items from a media item collection associated with the user account as input to the trained machine learning model;
using the trained machine learning model to generate an output image embedding of a cluster of media items, the media items in each cluster having a visual similarity, and the media having the visual similarity. partitioning the vector space generates the media item clusters by items being closer to each other than dissimilar media items in the vector space;
selecting a subset of the media item clusters based on corresponding media items in each cluster having a visual similarity within a visual similarity threshold;
displaying a user interface including the subset of the media item clusters. 17. The method of claim 16, wherein functional images are removed from the media item collection before the media item collection is provided to the trained machine learning model. 17. The method of claim 16, wherein the trained machine learning model is trained with feedback from a user including reactions to a set of media items or changes to a title of the set of media items. A system,
a processor;
a memory connected to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform the following operations;
The said operation is
determining media item clusters such that the media items in each cluster have visual similarity based on pixels of images or videos from the media item collection, the media item collection being associated with a user account; is,
selecting a subset of the media item clusters based on corresponding media items in each cluster having the visual similarity within a visual similarity threshold;
displaying a user interface including the subset of the media item clusters. Each media item has an associated timestamp;
the media items obtained within a predetermined time period are associated with an episode;
Selecting the subset of the media item clusters includes a temporal diversity in which the corresponding media items in the subset of the media item clusters exclude more than a predetermined number of the corresponding media items from a particular episode. 20. The system of claim 19, wherein meeting the criteria is done based on corresponding associated timestamps.

Images