EP4662566A1

EP4662566A1 - Performing a task relating to a query using a machine-learned model pretrained using first and second datasets

Info

Publication number: EP4662566A1
Application number: EP23716001.5A
Authority: EP
Inventors: Junjie Ke; Keren YE; Jiahui YU; Yonghui Wu; Peyman Milanfar; Feng Yang
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2023-03-09
Filing date: 2023-03-09
Publication date: 2025-12-17
Also published as: WO2024186318A1

Abstract

A computer-implemented method includes receiving a query relating to at least one image, processing the query, by a machine-learned model which is pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing. The one or more tasks include one or more of outputting an aesthetic score relating to the at least one image, outputting one or more images having an aesthetic characteristic specified by the query, outputting a caption generated by the machine-learned model, outputting an aesthetic classification, or outputting a selection of a highest aesthetically ranked query image from among a plurality of query images included in the query. The first dataset includes textual descriptions associated with a first plurality of images and the second dataset includes aesthetic textual descriptions associated with a second plurality of images.

Description

PERFORMING A TASK RELATING TO A QUERY USING A MACHINE-LEARNED

MODEL PRETRAINED USING FIRST AND SECOND DATASETS

FIELD

[0001] The disclosure relates to a vision language model that is pretrained on imagecomment pairs to leam image aesthetic information, for example using self-supervised learning. More particularly, downstream tasks including image aesthetic assessment, aesthetic captioning, and zero-shot learning for aesthetic style classification may be enabled.

BACKGROUND

[0002] Current image aesthetic assessment (IAA) methods can be based on human ratings which provide a single score (e.g., a score on a scale from 1 to 10 of 4.70, 5.67, 6.67, etc.). However, such a score does not capture diverse aesthetic factors including a composition, color, style, high-level semantics, etc. Applications of IAA include photo recommendations, photo selection, and photo editing. That is, a single score is limited and can be difficult for a user to understand.

SUMMARY

[0003] Aspects and advantages of embodiments of the disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments

[0004] One example aspect of the disclosure is directed to a computer-implemented method for performing a task aesthetically related to a query. The method includes receiving, by a computing system, a query relating to at least one image, processing the query , by a machine-learned model provided by the computing system, the machine-learned model being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing, by the computing system, the one or more tasks comprising one or more of: outputting an aesthetic score with respect to the at least one image, outputting one or more images having an aesthetic characteristic specified by the query, outputting a caption generated by the machine-learned model for the at least one image, outputting an aesthetic classification for the at least one image, or outputting a selection of a highest aesthetically ranked query image when a plurality of query images are included in the query. The first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset.

[0005] In some implementations, a number of the first plurality of images from the first dataset is greater than a number of the second plurality of images from the second dataset. [0006] In some implementations, the query includes the at least one image, and the one or more tasks include outputting the aesthetic score with respect to the at least one image.

[0007] In some implementations, processing the query includes the machine-learned model determining the aesthetic score with respect to the at least one image based on the machine-learned model utilizing a rank-based adapter which optimizes a relative aesthetic score ranking between the at least one image and at least one other image.

[0008] In some implementations, the query includes a search request for one or more images having the aesthetic characteristic, and the one or more tasks include outputting the one or more images having the aesthetic characteristic.

[0009] In some implementations, processing the query includes the machine-learned model determining an aesthetic rank of a plurality of candidate images having the aesthetic characteristic, and the one or more images included in the output are ranked highest aesthetically among the plurality of candidate images according to an aesthetic score generated by the machine-learned model.

[0010] In some implementations, the aesthetic characteristic includes a style of an image.

[0011] In some implementations, the query includes the at least one image, processing the query includes the machine-learned model generating the caption for the at least one image, and the one or more tasks include outputting the caption generated by the machine- learned model for the at least one image.

[0012] In some implementations, the query includes the at least one image, and the one or more tasks include outputting the aesthetic classification for the at least one image, the aesthetic classification being selected from a plurality of predetermined aesthetic classifications.

[0013] In some implementations, the query includes the plurality of query images, processing the query includes the machine-learned model aesthetically ranking each of the plurality of query' images, and the one or more tasks include outputting the selection of the highest aesthetically ranked query image from among the plurality of query' images.

[0014] Another example aspect of the disclosure is directed to a computer-implemented method for training an image aesthetic model. The method includes pretraining the image aesthetic model in a first manner using a first dataset, the first dataset including textual descriptions associated with a first plurality of images from the first dataset; pretraining the image aesthetic model in a second manner using a second dataset, the second dataset including aesthetic textual descriptions associated with a second plurality of images from the second dataset; and implementing the image aesthetic model based on the pretraining in the first manner and the pretraining in the second manner, to perform one or more tasks with respect to an input query relating to at least one image, the one or more tasks comprising one or more of: outputting an aesthetic score with respect to the at least one image, outputting one or more images having an aesthetic characteristic specified by the input query, outputting a caption generated by the machine-learned model for the at least one image, outputting a respective aesthetic classification for the at least one image, or outputting a selection of a highest aesthetically ranked query image when a plurality of query images are included in the input query.

[0015] In some implementations, a number of the first plurality of images from the first dataset is greater than a number of the second plurality of images from the second dataset. [0016] In some implementations, the input query includes the at least one image, and the one or more tasks include outputting the aesthetic score with respect to the at least one image. [0017] In some implementations, implementing the image aesthetic model includes the image aesthetic model determining the aesthetic score with respect to the at least one image based on the image aesthetic model utilizing a rank-based adapter which optimizes a relative aesthetic score ranking between the at least one image and at least one other image.

[0018] In some implementations, the input query includes a search request for one or more images having the aesthetic characteristic, and the one or more tasks include outputting a predetermined number of candidate images having the aesthetic characteristic by the image aesthetic model determining an aesthetic rank of a plurality of candidate images having the aesthetic characteristic and selecting the predetermined number of candidate images which are ranked highest aesthetically among the plurality of candidate images according to an aesthetic score generated by the image aesthetic model for each of the plurality of candidate images.

[0019] In some implementations, the aesthetic characteristic includes a style of an image.

[0020] In some implementations, the input query includes the at least one image, implementing the image aesthetic model includes the image aesthetic model generating the caption for the at least one image, and the one or more tasks include outputting the caption generated by the image aesthetic model for the at least one image. [0021] In some implementations, the input query includes the at least one image, and the one or more tasks include outputting the aesthetic classification for the at least one image, the aesthetic classification being selected from a plurality of predetermined aesthetic classifications.

[0022] In some implementations, the input query includes the plurality of query images, implementing the image aesthetic model includes the image aesthetic model aesthetically ranking each of the plurality of query images, and the one or more tasks include outputting the selection of the highest aesthetically ranked query image from among the plurality of query images.

[0023] Another example aspect of the disclosure is directed to a computing system. The computing system includes one or more processors; and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: receiving a query relating to at least one image, processing the query, by a machine-learned model provided by the computing sy stem, the machine-learned model being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing, the one or more tasks comprising one or more of: outputting an aesthetic score with respect to the at least one image, outputting one or more images having an aesthetic characteristic specified by the query, outputting a caption generated by the machine-learned model for the at least one image, outputting an aesthetic classification for the at least one image, or outputting a selection of a highest aesthetically ranked query image when a plurality of query images are included in the query, wherein the first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset.

[0024] Other aspects of the disclosure are directed to various systems, apparatuses, non- transitory computer-readable media, user interfaces, and electronic devices. In one or more example embodiments, a computer-readable medium (e.g., a non-transitory computer- readable medium) which stores instructions that are executable by one or more processors of a computing system or computing device is provided. In some implementations the computer-readable medium stores instructions which may include instructions to cause the one or more processors to perform one or more operations of any of the methods described herein (e.g., operations of the computing system or of the computing device). The computer- readable medium may store additional instructions to execute other aspects of the computing system or of the computing device and corresponding methods of operation, as described herein.

[0025] These and other features, aspects, and advantages of various embodiments of the disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the disclosure and, together with the descnption, serve to explain the related principles.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended drawings, in which: [0027] FIG. 1 A depicts a block diagram of an example computing system that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0028] FIG. IB depicts a block diagram of an example computing device that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0029] FIG. 1C depicts a block diagram of an example computing device that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0030] FIG. 2 depicts a flow chart diagram of an example method performed by a machine-learned model, according to example embodiments of the disclosure.

[0031] FIG. 3 depicts a block diagram of aesthetically pretraining an example machine- learned model, according to example embodiments of the disclosure.

[0032] FIG. 4 depicts a block diagram of a rank-based adapter implemented in a machine-learned model, according to example embodiments of the disclosure.

[0033] FIG. 5 depicts a flow chart diagram of an example method to perform a task based on processing of a query by a machine-learned model, according to example embodiments of the disclosure.

[0034] FIG. 6 depicts a flow chart diagram of an example method for training an image aesthetic model, according to example embodiments of the disclosure.

[0035] FIG. 7 depicts example ensemble prompts utilized for training a machine-learned model, according to example embodiments of the disclosure. [0036] FIG. 8 depicts images retrieved by a machine-learned model with respect to a prompt describing an image aesthetic style, according to example embodiments of the disclosure.

[0037] FIG. 9 depicts images retrieved by a machine-learned model with respect to various inputs describing a desired image aesthetic style, according to example embodiments of the disclosure.

[0038] FIG. 10 depicts images retrieved by a machine-learned model with respect to various inputs describing a desired image aesthetic style, according to example embodiments of the disclosure.

[0039] FIG. 11 depicts image captions generated by a machine-learned model with respect to various images, according to example embodiments of the disclosure.

[0040] Reference numerals that are repeated across plural drawings are intended to identify the same features in various implementations.

DETAILED DESCRIPTION

Overview

[0041] Reference now will be made to embodiments of the disclosure, one or more examples of which are illustrated in the drawings, wherein like reference characters across drawings are intended to denote like features in various implementations. Each example is provided by way of explanation of the disclosure and is not intended to limit the disclosure. [0042] Terms used herein are used to describe the example embodiments and are not intended to limit and / or restrict the disclosure. The singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. In this disclosure, terms such as "including", "having", “comprising”, and the like are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not preclude the presence or addition of one or more of the features, elements, steps, operations, elements, components, or combinations thereof.

[0043] It will be understood that, although the terms first, second, third, etc., may be used herein to describe various elements, the elements are not limited by these terms. Instead, these terms are used to distinguish one element from another element. For example, without departing from the scope of the disclosure, a first element may be termed as a second element, and a second element may be termed as a first element.

[0044] It will be understood that when an element is referred to as being “connected” to another element, the expression encompasses an example of a direct connection or direct coupling, as well as a connection or coupling with one or more other elements interposed therebetween.

[0045] The term "and / or" includes a combination of a plurality of related listed items or any item of the plurality of related listed items. For example, the scope of the expression or phrase "A and/or B" includes the item "A", the item "B", and the combination of items "A and B”.

[0046] In addition, the scope of the expression or phrase "at least one of A or B" is intended to include all of the following: (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Likewise, the scope of the expression or phrase "at least one of A, B, or C" is intended to include all of the following: (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B, and at least one of C.

[0047] According to example embodiments disclosed herein, an image and user comments about the image may be used in combination to provide rich aesthetic semantics and to mine aesthetic concepts about the image. For example, user comments about an image may include comments such as compliments about a composition of the image (e.g., “simple and nice composition, I like it”), about a quality of the image (“the photo is too blurry”), about a style of the image (e g., “not enough of the background here, imo”).

[0048] According to example embodiments disclosed herein, a computing system may be configured to pretrain an image aesthetic model with noisy image-comment pairs and efficiently adapt the model for downstream IAA tasks (e.g., image aesthetic assessment, aesthetic captioning, and zero-shot learning).

[0049] Various learning-based IAA methods have been proposed by leveraging deep models such as convolutional neural networks (CNN) and transformers. These approaches learn from human-labeled IAA datasets where images are paired with aesthetic ratings, and models are trained to regress towards the mean opinion scores (MOS).

[0050] Learning IAA models directly from human-labeled aesthetic ratings, e.g., MOS, is suboptimal because scores do not provide context for why an image is aesthetically pleasing or not Some methods attempt to integrate external knowledge such as theme, human eye fixation, and aesthetic attributes, to improve the IAA performance. These methods either use multitask training or cascade score prediction on top of a frozen attribute network. However, they require additional labeled dataset or off-the-shelf models, which can be costly to obtain. [0051] In comparison to the above approaches which require additional annotations, according to examples of the disclosure, the disclosed method utilizes abundant imagecomment pairs from existing sources, databases, and the like (e.g., aesthetic websites or photographic forums). For example, some image-comment pairs can be easily crawled from the Internet. In addition, the image-comment pairs contain rich aesthetic information (e.g., objects, themes, styles, positive or negative user emotions, etc.) as humans are better at expressing aesthetic preferences using natural languages than providing abstract scores. On some image sharing platforms user comments provide deep insights about how the aesthetics of a particular image is evaluated (e.g., comments including “very cool patterns and curls” and “little bit on the blurry side”, which demonstrate positive and negative aesthetic comments, respectively). Examples of the disclosure use the rich aesthetic semantics from this image-comment space to improve IAA methods.

[0052] Prior methods utilizing user comments for IAA employ very different task formulations from the IAA methods disclosed herein. For example, some methods aggregate visual and comment features for IAA, but require the image and comment to be fed as inputs at inference time. Since images are not always associated with comments, it is hard to apply such methods in real-world applications. Some approaches use the Latent Dirichlet Allocation (LDA) topics from the comments as pseudo labels to guide image representation learning. However, simplifying the comments into topics loses the rich context. In contrast, according to examples disclosed herein the raw comments are utilized for richer aesthetic textual information.

[0053] Examples of the disclosure are directed to a two-stage Vision-Language Aesthetics (VILA) learning framework using image-text pretraining for obtaining a model that generalizes to multiple downstream IAA tasks (e.g., image aesthetic assessment, aesthetic captioning, and zero-shot learning for aesthetic style classification). In the pretraining stage, an image-text model (VILA-P) is learned using the contrastive and text sequence generation objectives, to fully leverage the fine-grained knowledge from aesthetic image-comment pairs.

[0054] Other vision-language models such as contrastive language-image pretraining (CLIP), ALIGN, and Contrastive Captioner (CoCa), are trained to align vision and language feature spaces to capture semantic information. However, these vision-language models are pretrained using general image-text pairs on the web, and therefore aesthetic-related information is underrepresented. These generally pretrained vision-language models perform sub-optimally on aesthetic tasks. According to examples of the disclosure, a vision-language model is pretrained on aesthetic image-comment pairs from existing sources, databases, and the like (e.g., aesthetic websites or photographic forums, photograph sharing websites, etc.). [0055] After pretraining VILA-P on the image-comment pairs, the disclosed model may be fine-tuned in a second stage using a lightweight rank-based adapter (VILA-R), which is designed for adapting the pretrained model for downstream score-based IAA tasks. By adding feature residuals to the frozen image embeddings, images with good aesthetic quality are pushed closer to the textual anchor of “good image” and images with low aesthetic quality are pulled away from the same anchor The method can effectively rank the images according to human rated preferences. With 0.1% tunable parameters, the disclosed model outperforms previous works on IAA correlation metrics over the Aesthetic Visual Analysis (AV A) dataset.

[0056] As described above, various downstream tasks can be performed using the VILA learning framework according to the examples of the disclosure. In addition to the scorebased IAA, the VILA learning framework described herein can perform superior aesthetic comment generation as well as provide zero-shot learning (ZSL) capabilities for aesthetic style classification and quality analysis. By companng images to text queries such as “good photo” and “bad photo”, the ZSL model outperforms supervised learning models like neural image assessment (NIMA) which require labor intensive ratings as a ground truth. The example methods disclosed herein can learn rich image aesthetic concepts without human labels, greatly reducing the data collection cost.

[0057] The vision-language aesthetic framework (VILA) described herein learns rich image aesthetic features using image-comment pairs. In some implementations, a rank-based module adapts the model to downstream IAA tasks without perturbing the pretrained weights, effectively learning the aesthetic quality concepts with minimal injected parameters. The pretrained aesthetic model outperforms prior works for aesthetic captioning on AVA- Captions. For example, without any supervised labels, zero-shot performance is 69% mean average precision (mAP) on the AVA-Style dataset and 0.657 Spearman’s rank correlation coefficient (SRCC) on the AVA dataset, outperforming many supervised approaches. When tuned with a small number of parameters using the disclosed rank-based adapter, the disclosed method further achieves state-of-the-art performance on the AVA dataset.

[0058] IAA is widely useful for search, ranking, recommendation, and UI design, etc. Unlike technical quality assessment which focuses on distortion, cropping, or image noise, IAA aims at measuring the aesthetic quality. Previous methods were data-driven and gathered large-scale datasets containing images and human ratings. Based on these datasets, some methods built a rank-based model and approximated the ground truth score distributions. Different from these methods, the disclosed image aesthetic model utilizes an image-text pre-training framework and in some implementations utilizes a rank-based adapter by optimizing only a small set of learnable parameters, avoiding catastrophic forgetting which has occurred in previous methods, while preserving the rich knowledge from the pretrained model.

[0059] To preserve the high-resolution aesthetic information for IAA, some prior methods focused on the network design to allow perceiving the full-size images and original aspect ratios. For example, some convolutional neural network (CNN)-based methods may be designed to alleviate detrimental cropping and resizing. Other methods adopt a transformer architecture, where the input image is treated as visual tokens and variable-length sequences are naturally supported, preserving the image resolution and aspect ratios. Different from these methods, in some implementations the disclosed image aesthetic model may use a fixed input (e.g., a 224 x 224 input) without considering the original resolution and aspect ratios. [0060] Some methods utilize additional supervision for IAA. For example, some methods utilize natural language annotations in their curated datasets. However, these methods formulate IAA as one of multiple parallel tasks, or don’t generate quality related outputs, or require both the image and comment at inference time. Different from these methods, in some implementations the disclosed image aesthetic model may utilize user comments to leam meaningful aesthetic representations using contrastive and generative targets, and the learned image model can be used independently without text input.

[0061] Image-text pretraining utilizes the fact that paired image and text co-refer to each other. Some methods use a contrastive learning framework, aiming to draw image representation and the aligned text representation closer than image-text pairs that do not match. Some methods implement self-supervised learning objectives such as masked region reconstruction, masked object prediction, word region alignment, etc. However, these imagetext models all required an off-the-shelf visual detector, e g., Fast Region-based CNN (Fast R-CNN), to extract salient visual tokens. This hindered their generalization to large-scale pretraining, since training visual detectors also requires annotated object bounding boxes. Methods such as vision transformer (ViT), which treats an image as patch tokens, have been adopted for large-scale pretraining and end-to-end multimodal transformer-based methods have been implemented. CLIP, ALIGN, and CoCa train image-text foundation models using a large-scale image-text corpus such. Because the training data involves billions of image- text pairs, the learned models can generalize to many downstream tasks such as image- text/text-image retrieval, image classification, captioning, etc.

[0062] According to examples of the disclosure, pretraining the image aesthetic model VILA-P allows for learning a rich vision-language latent representation for image aesthetics in a self-supervised manner, using images and paired user comments.

[0063] The architecture used for pretraining the image aesthetic model described herein includes some elements from the CoCa architecture which unifies both contrastive learning and image-to-caption generation under the same architecture.

[0064] According to examples of the disclosure, the implemented architecture includes an image encoder, a unimodal text decoder, and a multimodal text decoder. The image representation (from the image encoder) and the unimodal text representation (from the appended [CLS] position of the unimodal text decoder) are aligned using a contrastive objective. The multimodal text decoder is trained to perform captioning by cross-attending to the image features.

[0065] The image encoder may be in the form of a vision transformer and be configured to perform image encoding by splitting an image into patches and treating the patches as tokens. The patches are then projected to D-dimensional features and fed to the transformer blocks to generate a sequence of visual embeddings V = {vi, ..., VK}, where K is the number of visual tokens.

[0066] The unimodal text decoder may be configured to first tokenize the text into a sequence of tokens, where each token is mapped to a D-dimensional word embedding vector. A [CLS] token is appended to the sequence, and the input tokens are then fed through a plurality of layers (e.g., 6-layers) of transformer blocks to generate the unimodal text representation W = {wi, ..., WL, Weis}, where Weis is output from the [CLS] token, and L is the number of text tokens. To enable the captioning objective, the transformer text decoder layers are trained with causally -masked self-attention, which prohibits tokens from attending to future tokens. The appended learnable token wcis is used as the contrastive text embedding. [0067] To apply the contrastive learning objective, the unimodal encoding modules are jointly optimized by a contrastive target which is used to align the image-text pairs:

[0068] where xi and yi are the normalized contrastive embeddings of the i-th image and text in the batch. L^12tcon is the image-to-text contrastive loss and L^t21con is the text-to-image counterpart, T is the learnable temperature, and N is the batch size.

[0069] To apply the generative learning objective and perform a captioning task, the multimodal text decoder may be configured to maximize the likelihood of generating the paired text conditioned on visual features in an autoregressive manner:

[0070] To cotrain the two targets (i.e., the contrastive and generative objectives), two task-specific attentional pooling layers are stacked on top of the image encoder to generate a contrastive image representation and a generative image representation. Finally, the combined pre-training target is a weighted sum of the contrastive loss and the generative loss, using hyper-parameters a and ?:

L ⁼ ClLcon + j3l_Gen (2).

[0071] Vision-language pretraining methods require large-scale data for learning the complex visual and textual dynamics in the real-world. Many of those methods are trained on large proprietary datasets with image-text pairs crawled from the web. Such a general pretraining strategy is proven to be useful for tasks such as image classification and retrieval. However, the majority of the paired text from the web is about the image content rather than aesthetic related image-text pairs. As a result, aesthetic information easily gets diluted in the vast amount of pretrained data. In this regard, the method disclosed herein initializes from the generally pretrained image-text model, and then further pretrains the model on aesthetic image-comment pairs. An example dataset source for general pre-training includes a filtered 650M subset of an openly available dataset LAION-5B-English. For aesthetic pre-training, as another example dataset source, an AVA-Captions dataset (which is the largest aesthetic comments dataset currently available) may be used. In AVA-Captions. each image is associated with one or more user comments which are often greatly informative about different aesthetic aspects for the image. During training, image-comment pairs may be constructed by randomly sampling one comment each time from the list of available comments.

[0072] Compared to known supervised learning with predefined labels or categories, such vision-language pretraining allows open-set aesthetic concepts to be learned through the noisy image-comment pairs. This allows the latent visual and textual representations to capture much richer and broader aesthetic concepts, which makes it more transferable to downstream tasks.

[0073] The pretrained image aesthetic model contains rich vision-language information which enables zero-shot aesthetic tasks. As discussed later, experimental results demonstrate that the pretrained model VILA-P has impressive ZSL ability, surpassing some supervised models.

[0074] According to some implementations, the VILA-P model’s IAA performance may be further improved for scenarios where the mean-opinion-score (MOS) as training labels is utilized. Since finetuning the whole model is expensive and may damage the pretrained model’s zeroshot and captioning ability, according to examples described herein, the model is adapted with a few tunable parameters while keeping the image and text backbone frozen.

For this purpose, a lightweight rank-based adapter module is implemented to efficiently adapt the pretrained vision-language model to downstream IAA tasks. It allows the model to inherit the training-free advantages of the pretrained backbone, while leveraging the rich aesthetic textual information for IAA tasks. The modified model having the adapter module is referred to as VILA-R herein.

[0075] IAA predicts the aesthetic score for a given image. For example, an image may be represented by a frozen image embedding extracted using the image encoder in VILA-P. Formally,

V = E(I, Ofrozen), (3) r = F(v, y), (4)

[0076] where I is the input image, v is the image features extracted using image encoder E with its frozen pretrained weights Ofrozen. F is the IAA scoring model with parameters y, and r is the predicted aesthetic score. [0077] Given an arbitrary pair of images represented by Vi and vj, and their corresponding MOS labels li and lj, IAA model output n and rj are trained to respect the order of li and lj . The effectiveness of the model F is measured by the correlation between r and 1. [0078] Given the contrast vely pretrained image embedding v and the corresponding text embedding w, their cosine similarity v^Tw can be used to measure to what extent the image involves the textual concept. This is due to the fact that the cosine similarity between paired image-text is maximized by the contrastive pretraining objective (Eq. 1). Thus, the text can be used as “prompts” to score images according to the textual concept (e g., measuring whether images are “good photos”). For example, experimental results show more than 0.6 correlation using text prompts for IAA scoring. This implies that the text decoder in VILA-P contains useful information about what is a pleasantly looking image, and can be utilized as an anchor to further improve the model’s IAA ranking capability.

[0079] With the contrastive and generative pretraining, the rich textual concept about aesthetically pleasing images is captured in the text decoder, and it is embedded in the same latent space as the image. So, the image embedding can be slightly adjusted to fit that rich textual concept of a good image. Concretely, the frozen text embedding of “good image” may be used by the computing system as an anchor to score the images, and the image representation may be adjusted to optimize the relative ranking between two images according to their MOS labels. For example, v represents the unnormalized contrastive image embedding from the frozen VILA-P. A learnable residual is added to obtain the rank- adjusted image embedding v as where H 6 R^DXD j_{s a} learnable matrix for modeling the residuals.

[0080] The “good image” is used as the text prompt, and its normalized frozen embedding w_P is extracted from the [CLS] position of the unimodal text decoder output. The cosine similarity between image v and the anchor w_P is used as the predicted IAA score for ranking, r = v^Tw_P (6)

[0081] To optimize the relative ranking between two images, the triplet ranking loss LRA is optimized using w_P as the anchor with a pair of images as input. where m is the margin hyper-parameter.

[0082] For example, m = 0.1 may be used as a default value, however other values of m may be used (e.g., 0.01, 0.05, 0.15, 0.2). The image with a higher MOS label li will be treated as positive vi, and the image with a lower MOS lj is the corresponding negative vj . The ranking loss ensures that the similarity between the positive sample and the anchor is greater than that of the negative sample, which effectively ranks the images according to its aesthetic ratings. The only tunable parameter in this example is H with D² parameters, which is only about 0. 1% of the total parameters of VILA-P.

[0083] In this example, only the frozen text embedding w_P is used which can be exported for training and inference without the text backbone. As a result, the final IAA model has the same computational and storage as a single image-encoder-only architecture, with only the image as input for IAA inference.

[0084] The disclosed framework for learning image aesthetics (VILA) includes pretraining vision-language models on image-comment pairs from external sources including image sharing websites. The model learns the rich aesthetic semantics in a self-supervised manner without the need for expensively labeled data. The pretrained model VILA-P achieves state-of-the-art performance compared to other methods and enables additional tasks including zero-shot learning for IAA, style classification or retrieval, outperforming many supervised baselines.

[0085] To efficiently adapt the pretrained model for IAA without hindering its powerful zero-shot abilities or damaging the rich representation, a lightweight rank-based adapter module may be implemented. By using the text embedding as an anchor, and explicitly modeling the ranking concept, state-of-the-art performance with respect to the IAA task may be achieved with only a small amount of injected parameters. The disclosed rank-based adapter module may be implemented to perform IAA, however the disclosed method is generally applicable in adapting largescale visual-language models to other ranking based tasks.

[0086] The disclosure provides numerous technical effects and benefits. Results show that the pretrained aesthetic vision-language model outperforms prior works on the image aesthetic captioning task over the AVA-Captions data set. Results show that the pretrained aesthetic vision-language model has a powerful zero-shot capability for aesthetic tasks such as zero-shot style classification and zero-shot IAA, surpassing supervised baselines. In implementations including the lightweight rank-based adapter, model state-of-the-art performance with respect to the IAA task is achieved over the AVA dataset.

[0087] With reference now to the drawings, example embodiments of the disclosure will be discussed in further detail.

Example Devices and Systems

[0088] FIG. 1A depicts a block diagram of an example computing system 100 that performs various tasks using a machine-learned model (e.g., a pretrained vision language model or an image aesthetic model) according to example embodiments of the disclosure. The system 100 includes a user computing system 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0089] The user computing system 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0090] The user computing system 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing system 102 to perform operations. [0091] In some implementations, the user computing system 102 can store or include one or more machine-learned models 120 (e.g., a pretrained vision language model or an image aesthetic model). For example, the machine-learned models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi -headed self-atention models (e.g., transformer models). Example machine-learned models 120 are discussed with reference to the drawings herein.

[0092] In some implementations, the one or more machine-learned models 120 can be received from the server computing system 130 over network 180, stored in the memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing system 102 can implement multiple parallel instances of a single machine-learned model 120 (e g., to perform parallel tasks across multiple instances of the machine-learned model 120).

[0093] More particularly, the machine-learned models disclosed herein (e.g., referred to as VILA-P and VILA-R, or generally as VILA), may be implemented to perform various tasks related to an input query which relates to an image. For example, the machine-learned models disclosed herein may be utilized for downstream score-based IAA tasks (e.g., providing a mean opinion score (MOS)) with respect to a query including an input image. For example, the machine-learned models disclosed herein may be utilized for downstream tasks including aesthetic captioning generation with respect to a query including an input image. For example, the machine-learned models disclosed herein may be utilized for providing zero-shot learning (ZSL) aesthetic tasks involving aesthetic style classification and quality analysis with respect to a query including an input image. For example, the machine- learned models disclosed herein may be utilized for providing recommendations or search results relating to a query for an image having a particular or specified aesthetic characteristic (e.g., a certain style). For example, the machine-learned models disclosed herein may be utilized for aesthetically ranking images in order to provide recommendations or search results or to select one or more best aesthetic images relating to a query for an image having a particular or specified aesthetic characteristic (e.g., a certain style).

[0094] Additionally or alternatively, one or more machine-learned models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing system 102 according to a client-server relationship. For example, the machine-learned models 140 can be implemented by the server computing system 130 as a portion of a web service (e.g., a recommendation service, a search service, an image analysis service, and the like). Thus, one or more machine-learned models 120 can be stored and implemented at the user computing system 102 and/or one or more machine- learned models 140 can be stored and implemented at the server computing system 130. [0095] The user computing system 102 can also include one or more user input components 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input (e.g., a camera which captures an image).

[0096] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0097] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0098] As described above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the machine-learned models 140 can be or can otherwise include various machine-learned models. Example machine- learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine- learned models can include multi-headed self-attention models (e.g., transformer models). Example machine-learned models 140 are discussed herein with reference to the drawings.

[0099] The user computing system 102 and/or the server computing system 130 can train the machine-learned models 120 and/or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.

[0100] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0101] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and/or 140 stored at the user computing system 102 and/or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and/or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0102] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0103] In particular, the model trainer 160 can train the machine-learned models 120 and/or 140 based on a set of training data 162. The training data 162 can include, for example, various datasets which may be stored remotely or at the training computing system 150. For example, in some implementations datasets utilized for training include LAION- 5B-English-Filtered, AVA Dataset, AVA-Captions, and AVA-Style. However, other datasets may be utilized (e.g., images from external websites, image-text pairs from external websites, etc.).

[0104] In some implementations, if the user has provided consent, the training examples can be provided by the user computing system 102. Thus, in such implementations, the machine-learned model 120 provided to the user computing system 102 can be trained by the training computing system 150 on user-specific data received from the user computing system 102. In some instances, this process can be referred to as personalizing the model. [0105] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 1 0 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media. [0106] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 180 can be carried via any type of wired and/or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

[0107] The machine-learned models described in this specification may be used in a variety of tasks, applications, and/or use cases.

[0108] In some implementations, the input to the machine-learned model (s) of the disclosure can be image data (e.g., one or more images). The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image classification output (e.g., aesthetically classifying the image as a “bad image” or a “good image”, aesthetically classifying the image according to a style of the image, etc.). As another example, the machine-learned model(s) can process the image data to generate a captioning output (e.g., providing an aesthetic caption which provides aesthetic information about the image data). As another example, the machine-learned model(s) can process the image data to generate a prediction output (e.g., providing an aesthetic score or rating which assesses the aesthetic value or qualities of an image).

[0109] In some implementations, the input to the machine-learned model (s) of the disclosure can be text or natural language data which relate aesthetically to an image (i.e., a request in the fomi of text or natural language data for an image having a specified aesthetic characteristic - e.g., “show me images with a vanishing point”). The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a prediction output or search output (e.g., retrieving one or more images which aesthetically correspond to the input text or natural language data which includes aesthetic information associated with an image).

[0110] In some implementations, the input to the machine-learned model (s) of the disclosure can be speech data (e.g., a text output which is mapped to the spoken utterance). The machine-learned model(s) can process the speech data to generate an output. As an example, the machine-learned model(s) can process the speech data to generate a speech recognition output. As another example, the machine-learned model(s) can process the speech data to generate a speech translation output. As another example, the machine-learned model(s) can process the speech data to determine a query which relates aesthetically to an image (i. e. , a request in the form of speech for an image having a specified aesthetic characteristic - e.g., “show me images with a vanishing point”). The machine-learned model(s) can process the speech data to generate an output. As an example, the machine- learned model(s) can process the speech data to generate a prediction output or search output (e.g., retrieving one or more images which aesthetically correspond to the speech data which includes aesthetic information associated with an image).

[0111] FIG. 1 A illustrates one example computing system that can be used to implement the disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing system 102 can include the model trainer 160 and the training data 162. In such implementations, the machine-learned models 120 can be both trained and used locally at the user computing system 102. In some of such implementations, the user computing system 102 can implement the model trainer 160 to personalize the machine-learned models 120 based on user-specific data.

[0112] FIG. IB depicts a block diagram of an example computing device 10 that performs according to example embodiments of the disclosure. The computing device 10 can be a user computing device or a server computing device.

[0113] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0114] As illustrated in FIG. IB, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0115] FIG. 1C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the disclosure. The computing device 50 can be a user computing device or a server computing device.

[0116] The computing device 50 includes a number of applications (e g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0117] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in FIG. 1C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0118] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As illustrated in FIG. 1C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

Example Model Arrangements

[0119] FIG. 2 depicts a flow chart diagram of an example method performed by a machine-learned model, according to example embodiments of the disclosure. In some implementations, the machine-learned models described herein (e.g., VILA-P or VILA-R) are trained to receive a set of input data 210 from a dataset which includes information (e.g., text) that is aesthetically descriptive of a paired image. For example, in FIG. 2 the input data 210 includes user comments which are aesthetically descriptive of a corresponding image (e.g., “very cool paterns and curls”, “grand scene! wonderful elements of interest.”, and “litle bit on the blurry side.”). The aesthetic image-comment pairs may be obtained or received by the computing system (e.g., any of user computing system 102, server computing system 130, or training computing system 150) from existing sources, databases, and the like (e.g., aesthetic websites or photographic forums, photograph sharing websites, etc.).

[0120] As described in more detail below, in addition to being generally pretrained on image-text pairs using a first dataset that includes general text comments about corresponding images, the machine-learned model is further pretrained aesthetically at 220, where contrastive learning and generative learning are performed using the contrastive and text sequence generation objectives, to fully leverage the fine-grained knowledge from aesthetic image-comment pairs. After pretraining the machine-learned model (which may be referred to as the VILA-P model) on the image-comment pairs, the machine-learned model may be fine-tuned in a second stage using a lightweight rank-based adapter (described further with respect to FIG. 4). The fine-tuned machine-learned model may be referred to as the VILA-R model.

[0121] Various downstream tasks can be performed using the machine-learned models disclosed herein. For example, the VILA-R model may be utilized for downstream scorebased IAA tasks 230 (e.g., providing a mean opinion score (MOS)). For example, the VILA- P model may be utilized to perform various tasks including aesthetic captioning generation 240 as well as provide zero-shot learning (ZSL) aesthetic tasks 250 involving aesthetic style classification and quality analysis.

[0122] FIG. 3 depicts a block diagram of aesthetically pretraining an example machine- learned model, according to example embodiments of the disclosure.

[0123] The methods disclosed herein train a machine-learned model by pretrammg the machine-learned model generally, and then further pretraining the machine-learned model on aesthetic image-comment pairs. For example, the machine-learned models (e.g., the VILA-P model and/or VILA-R model) may be trained via training computing system 150. An example dataset source for general pre-training includes a filtered 650M subset of an openly available dataset LAION-5B-English. For aesthetic pre-training, as another example dataset source, an AVA-Captions dataset (which is the largest aesthetic comments dataset currently available) may be used. During training, image-comment pairs may be constructed by randomly sampling one comment each time from the list of available comments. [0124] As shown in FIG. 3, for pretraining the machine-learned model 300 (e.g., the VILA-P model) on aesthetic image-comment pairs, the machine-learned model 300 receives as an input an image-text pair which includes an image 310 and text 320 (particularly an aesthetic comment) about the image 310. The disclosed machine-learned model 300 may utilize abundant image-comment pairs from existing sources, databases, and the like (e.g., aesthetic websites or photographic forums) for aesthetically pretraining the machine-learned model 300. For example, some image-comment pairs can be easily crawled from the Internet. For example, raw comments are input to the machine-learned model 300 rather than atopical comment. For example, the image resolution for each image may be fixed to a set value (e.g., 224 X 224).

[0125] In the pretraining stage depicted in FIG. 3, the machine-learned model 300 (i.e., an image-text model (VILA-P)) is learned using the contrastive and text sequence generation objectives, to fully leverage the fine-grained knowledge from aesthetic image-comment pairs. As shown in FIG. 3, the machine-learned model 300 implements elements of a CoCa architecture and includes an image encoder 350, a unimodal text decoder 360, and a multimodal text decoder 390. The image representation (from the image encoder 350) and the unimodal text representation (from the appended [CLS] position 380 of the unimodal text decoder 360) are aligned using a contrastive objective. The multimodal text decoder 390 is trained to perform captioning by cross-attending 392 to the image features.

[0126] For example, the image encoder 350 may be in the form of a vision transformer and be configured to perform image encoding by splitting the image 310 into patches 330 and treating the patches 330 as tokens. For example, the patch size may be a fixed value (e.g. 16 X 16). The patches are then projected to D-dimensional features and fed to the transformer blocks to generate a sequence of visual embeddings V = {vi, ..., VK} 370, where K is the number of visual tokens.

[0127] The unimodal text decoder 360 may be configured to first tokenize the text 320 into a sequence of tokens, where each token is mapped to a D-dimensional word embedding vector 340. A [CLS] token is appended to the sequence, and the input tokens are then fed through a plurality of layers (e.g., 6-layers) of transformer blocks to generate the unimodal text representation W = {wi, ..., WL, Wds} 380, where Weis is output from the [CLS] token, and L is the number of text tokens. To enable the captioning objective, the transformer text decoder layers are trained with causally -masked self-attention, which prohibits tokens from attending to future tokens. The appended learnable token wds is used as the contrastive text embedding. [0128] To apply the contrastive learning objective, the unimodal encoding modules are jointly optimized by a contrastive target which is used to align the image-text pairs:

[0129] where xi and yi are the normalized contrastive embeddings of the i-th image and text in the batch. L^12tcon is the image-to-text contrastive loss and L^t21con is the text-to-image counterpart, T is the learnable temperature, and N is the batch size.

[0130] To apply the generative learning objective and perform a captioning task whereby the machine-learned model 300 generates a caption 394 aesthetically describing an input image, the multimodal text decoder 390 may be configured to maximize the likelihood of generating the paired text conditioned on visual features in an autoregressive manner:

[0131] To co-train the two targets (i.e., the contrastive and generative objectives), two task-specific attentional pooling layers are stacked on top of the image encoder 350 to generate a contrastive image representation and a generative image representation. Finally, the combined pre-training target is a weighted sum of the contrastive loss and the generative loss, using hyper-parameters a and

L ⁼ CzLcon + j3l_Gen (2).

[0132] FIG. 4 depicts a block diagram of a rank-based adapter implemented in a machine-learned model, according to example embodiments of the disclosure. To efficiently adapt the pretrained machine-learned model 300 for IAA without hindering its powerful zeroshot abilities or damaging the rich representation, a lightweight rank-based adapter module 400 may be implemented after pretraining VILA-P on the image-comment pairs. The modified machine-learned model having the rank-based adapter module is referred to as VILA-R herein. VILA-R may be used to perform IAA, for example. The rank-based adapter module 400 may be implemented by freezing an image embedding extracted from the image encoder 350. Formally,

V = E(I, Ofrozen), (3) r = F(v, y), (4)

[0133] where I is the input image, v is the image features extracted using image encoder E with its frozen pretrained weights Ofrozen. F is the IAA scoring model with parameters y, and r is the predicted aesthetic score. Given an arbitrary pair of images represented by vi and vj, and their corresponding MOS labels h and lj, IAA model output n and rj are trained to respect the order of li and lj . The effectiveness of the model F is measured by the correlation between r and 1. The rank-based adapter module 400 is configured to slightly adjust the image embedding to fit the rich textual concept of a good image. For example, as shown in FIG. 4 the frozen text embedding of “good image” 440 may be used by VILA-R as an anchor to score the images, and the image representation may be adjusted to optimize the relative ranking between two images (e.g., image 420 and image 430) according to their MOS labels. [0134] For example, v represents the unnormalized contrastive image embedding from the frozen VILA-P. A residual is added via shared learnable residual projection 410 to obtain the rank-adj usted image embedding v as where H 6 R^DXD is a learnable matrix for modeling the residuals.

[0135] The “good image” 440 is used as the text prompt, and its normalized frozen embedding w_P is extracted from the [CLS] position of the unimodal text decoder output. The cosine similarity between image v and the anchor w_P is used as the predicted IAA score for ranking, r = v^Tw_P (6)

[0136] To optimize the relative ranking between two images, the triplet ranking loss LRA is optimized using w_P as the anchor with a pair of images as input. where m is the margin hyper-parameter.

[0137] For example, m = 0.1 may be used as a default value, however other values of m may be used (e.g., 0.01, 0.05, 0. 15, 0.2). The image with a higher MOS label li will be treated as positive vi, and the image with a lower MOS lj is the corresponding negative Vj . The ranking loss ensures that the similarity between the positive sample and the anchor is greater than that of the negative sample, which effectively ranks the images according to its aesthetic ratings.

[0138] By adding feature residuals via the shared learnable residual projection 410 to the frozen image embeddings, images with good aesthetic quality are pushed closer to the textual anchor of “good image” and images with low aesthetic quality are pulled away from the same anchor. The method can effectively rank the images according to human rated preferences. [0139] For example, tunable parameters of the rank-based adapter module 400 may be limited to less than a predetermined threshold amount (e.g., less than 5%, less than 1%, less than 0.5% of the total parameters of VILA-P). For example, the only tunable parameter in the example may be H with D² parameters, which is only about 0. 1% of the total parameters of VILA-P. With 0.1% tunable parameters, the disclosed VILA-R model outperforms previous works on IAA correlation metrics over the Aesthetic Visual Analysis (AV A) dataset.

[0140] In this example, only the frozen text embedding w_P is used which can be exported for training and inference without the text backbone. As a result, the final IAA model has the same computational and storage as a single image-encoder-only architecture, with only the image as input for IAA inference. For example, by using the text embedding 440 as an anchor, and explicitly modeling the ranking concept, state-of-the-art performance with respect to the IAA task may be achieved with only a small amount of injected parameters. The disclosed rank-based adapter module 400 may be implemented to perform IAA, however the disclosed method is generally applicable in adapting largescale visual-language models to other ranking based tasks.

[0141] For example, the machine-learned models (e.g., the VILA-P model and/or VILA- R model) disclosed herein may be stored at user computing system 102 as machine-learned models 120 and/or at server computing system 130 as machine-learned models 140.

Example Methods

[0142] FIG. 5 depicts a flow chart diagram of an example method to perform according to example embodiments of the disclosure. Although FIG. 5 depicts operations performed in a particular order for purposes of illustration and discussion, the methods of the disclosure are not limited to the particularly illustrated order or arrangement. The various operations of the method 500 can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the disclosure.

[0143] At 510, a computing system receives a query relating to at least one image. For example, the query can include an image or a plurality of images. The query may include a request to aesthetically score or rate the image (or each of the plurality of image). For example, the query can include a search request (e.g., via a text input, a voice input, etc ), requesting one or more images having a specified aesthetic characteristic (e.g., having a particular style, for example, one or more styles corresponding to the style classes from the AVA-Style dataset, or having a particular aesthetic rating including a “bad image” or a “good image”). For example, the query can include a request (e.g., via a text input, a voice input, etc.) with respect to an input image or plurality of images, to provide a caption or description which aesthetically describes or rates the image (or each of the plurality of image). For example, the query can include a request (e.g., via a text input, a voice input, etc.) with respect to an input image or plurality of images, to aesthetically classify the image (or each of the plurality of image), for example according to a style class basis from the AVA-Style dataset or according to a curated prompt (e.g., “bad image,” “good image,” etc.). For example, the query can include a request (e.g., via a text input, a voice input, etc.) to aesthetically rank a plurality of input images. For example, when a plurality of images are captured by a capturing device (e g., a camera, a LIDAR, a RADAR, etc ), the computing system (e.g., the machine-learned model) may automatically aesthetically rank the plurality of images captured by the capturing device and select a highest aesthetically ranked image for display by the computing system. In some implementations, the remaining images may be deleted. For example, a computing system may receive a search query for one or more images. In determining a search result responsive to the search query, the computing system may search through a plurality of query images (the query' images corresponding to an input query for the machine-learned model) and automatically aesthetically rank the plurality of query images relating to the search query and select one or more highest aesthetically ranked images for display by the computing system in response to the search query.

[0144] At 520, the computing system processes the query, using a machine-learned model provided by the computing system. The machine-learned model may be pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset. The first dataset may include textual descriptions associated with a first plurality of images from the first dataset and the second dataset may include aesthetic textual descriptions associated with a second plurality of images from the second dataset. For example, the first dataset and the second dataset may correspond to a LAION-5B-English-Filtered dataset, AVA dataset, AVA-Captions dataset, or AVA-Style dataset. However, other datasets may also be utilized. For example, any dataset or source where images are paired with general descriptions or comments about an image may be utilized (e.g., websites or photographic forums) for generally pretraining the machine-learned model. For example, any dataset or source where images are paired with aesthetic ratings or aesthetic comments may be utilized (e.g., aesthetic websites or photographic forums) for aesthetically pretraining the machine- learned model. For example, the first dataset may be a dataset for general pretraining the machine-learned model (e.g., LAION-5B-English or LAION-5B-English-Filtered) where the paired text or comments include general statements about the image content and are not necessarily directed to or directly related to aesthetic qualities about the image content. For example, the second dataset may be a dataset for aesthetically pretraining the machine- learned model (e.g., AVA-Captions). Pretraining the machine-learned model using the second dataset may be performed after pretraining the machme-leamed model using the generally pretrained machine-learned model using the first dataset. The second dataset may include fewer images than the first dataset.

[0145] The computing system, and particularly the machine-learned model, may process the query differently according to the task to be performed based on the content of the query. For example, the machine-learned model may process a query relating to an image by determining the aesthetic score with respect to the image based on the machine-learned model utilizing a rank-based adapter which optimizes a relative aesthetic score ranking between the image and at least one other image. For example, the machine-learned model may process a query relating to a search request for an image having a specified aesthetic characteristic by determining a plurality of images having the specified aesthetic characteristic as a plurality of candidate images, aesthetically scoring each of the plurality of candidate images, and determining an aesthetic rank of each of the plurality of candidate images having the specified aesthetic characteristic. For example, the machine-learned model may process a query relating to a request to aesthetically describe an image by generating a caption for the image. For example, the machine-learned model may process a query relating to a request to aesthetically describe an image by aesthetically classifying the image (e.g., according to a style of the image). [0146] At 530, the computing system performs one or more tasks based on the processing. For example, the one or more tasks may be associated aesthetically with respect to the query. For example, the one or more tasks may include the computing system (or the machine-learned model) outputting an aesthetic score with respect to an image associated with the query. For example, the one or more tasks may include the computing system (or the machine-learned model) outputting one or more images having a specified aesthetic characteristic with respect to a query which requests one or more images having the specified aesthetic characteristic. For example, the one or more tasks may include the computing system (or the machine-learned model) outputting a caption generated by the machine- learned model for an image associated with the query. For example, the one or more tasks may include the computing system (or the machine-learned model) outputting an aesthetic classification for an image associated with the query, the aesthetic classification being selected from a plurality of predetermined aesthetic classifications. For example, the one or more tasks may include the computing system (or the machine-learned model) outputting or selecting (e.g., for presentation on a display of a computing device) a highest aesthetically ranked captured image from among a plurality of aesthetically ranked captured images, with respect to a query associated with a plurality of captured images.

[0147] As an example implementation, the computer-implemented method 500 may include receiving, by the computing system (e.g., computing systems 102, 130), a query relating to at least one image, processing the query, by a machine-learned model (e.g., machine-learned models 120, 140) provided by the computing system (e g., computing systems 102, 130), the machine-learned model (e.g., machine-learned models 120, 140) being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing, by the computing system (e.g., computing systems 102, 130). The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting an aesthetic score with respect to the at least one image when the query includes the at least one image. The first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset. The machine-learned model (e g., machine- learned models 120, 140) may process the query by determining the aesthetic score with respect to the at least one image based on the machine-learned model (e.g., machine-learned models 120, 140) utilizing a rank-based adapter which optimizes a relative aesthetic score ranking between the at least one image and at least one other image. [0148] As an example implementation, the computer-implemented method 500 may include receiving, by the computing system (e.g., computing systems 102, 130), a query relating to at least one image, processing the query, by a machine-learned model (e.g., machine-learned models 120, 140) provided by the computing system (e.g., computing systems 102, 130), the machine-learned model (e.g., machine-learned models 120, 140) being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing, by the computing system (e g., computing systems 102, 130). The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting one or more images having an aesthetic characteristic specified by the query when the query includes a search request for one or more images having the aesthetic characteristic. The first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset. The machine-learned model (e.g., machine-learned models 120, 140) may process the query by determining an aesthetic rank of a plurality of candidate images having the aesthetic characteristic.

[0149] As an example implementation, the computer-implemented method 500 may include receiving, by the computing system (e.g., computing systems 102, 130), a query relating to at least one image, processing the query, by a machine-learned model (e.g., machine-learned models 120, 140) provided by the computing system (e.g., computing systems 102, 130), the machine-learned model (e.g., machine-learned models 120, 140) being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing, by the computing system (e.g., computing systems 102, 130). The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting a caption generated by the machine-learned model (e.g., machine-learned models 120, 140) for the at least one image when the query includes the at least one image. The first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset. The machine-learned model (e g., machine-learned models 120, 140) may process the query by generating the caption for the at least one image.

[0150] As an example implementation, the computer-implemented method 500 may include receiving, by the computing system (e.g., computing systems 102, 130), a query relating to at least one image, processing the query, by a machine-learned model (e.g., machine-learned models 120, 140) provided by the computing system (e.g., computing systems 102, 130), the machine-learned model (e.g., machine-learned models 120, 140) being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing, by the computing system (e.g., computing systems 102, 130). The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting an aesthetic classification for the at least one image when the query includes the at least one image. The first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset. The aesthetic classification may be selected from a plurality of predetermined aesthetic classifications.

[0151] As an example implementation, the computer-implemented method 500 may include receiving, by the computing system (e.g., computing systems 102, 130), a query relating to at least one image, processing the query, by a machine-learned model (e.g., machine-learned models 120, 140) provided by the computing system (e.g., computing systems 102, 130), the machine-learned model (e.g., machine-learned models 120, 140) being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing, by the computing system (e.g., computing systems 102, 130). The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting a selection of ahighest aesthetically ranked query image when a plurality of query images are included in the query. The first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset. The machine-learned model (e.g., machine-learned models 120, 140) may process the query by aesthetically ranking each of the plurality of query images.

[0152] FIG. 6 depicts a flow chart diagram of an example method to perform according to example embodiments of the disclosure. Although FIG. 6 depicts operations performed in a particular order for purposes of illustration and discussion, the methods of the disclosure are not limited to the particularly illustrated order or arrangement. The various operations of the method 600 can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the disclosure. [0153] At 610, a computing system pretrains an image aesthetic model in a first manner using a first dataset, the first dataset including textual descriptions associated with a first plurality of images from the first dataset.

[0154] At 620, the computing system pretrains the image aesthetic model in a second manner using a second dataset, the second dataset including aesthetic textual descriptions associated with a second plurality of images from the second dataset. As discussed above, the first dataset and the second dataset may correspond to a LAION-5B-English-Filtered dataset, AVA dataset, AVA-Captions dataset, or AVA-Style dataset. However, other datasets may also be utilized. For example, any dataset or source where images are paired with general descriptions or comments about an image may be utilized (e.g., websites or photographic forums) for generally pretraining the image aesthetic model (i.e., machine- learned model). For example, any dataset or source where images are paired with aesthetic ratings or aesthetic comments may be utilized (e.g., aesthetic websites or photographic forums) for aesthetically pretraining the image aesthetic model. Pretraining the image aesthetic model using the second dataset may be performed after pretraining the image aesthetic model using the generally pretrained image aesthetic model using the first dataset. The second dataset may include fewer images than the first dataset.

[0155] At 630, the computing system implements the image aesthetic model based on the pretraining in the first manner and the pretraining in the second manner, to perform one or more tasks with respect to an input query relating to at least one image. For example, the image aesthetic model may process the input query and perform one or more tasks based on the processing, similar to the operations and one or more tasks described above with respect to FIG. 5 and operations 520 and 530.

[0156] As an example implementation, the computer-implemented method 600 may include pretraining an image aesthetic model (e.g., machine-learned models 120, 140) in a first manner using a first dataset, the first dataset including textual descriptions associated with a first plurality of images from the first dataset, pretraining the image aesthetic model (e.g., machine-learned models 120, 140) in a second manner using a second dataset, the second dataset including aesthetic textual descriptions associated with a second plurality of images from the second dataset, and implementing the image aesthetic model (e.g., machine- learned models 120, 140) based on the pretraining in the first manner and the pretraining in the second manner, to perform one or more tasks with respect to an input query relating to at least one image. The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting an aesthetic score with respect to the at least one image when the input query includes the at least one image. The machine-learned model (e.g., machine- learned models 120, 140) may process the input query by determining the aesthetic score with respect to the at least one image based on the machine-learned model (e.g., machine- learned models 120, 140) utilizing a rank-based adapter which optimizes a relative aesthetic score ranking between the at least one image and at least one other image.

[0157] As an example implementation, the computer-implemented method 600 may include pretraining an image aesthetic model (e.g., machine-learned models 120, 140) in a first manner using a first dataset, the first dataset including textual descriptions associated with a first plurality of images from the first dataset, pretraining the image aesthetic model (e.g., machine-learned models 120, 140) in a second manner using a second dataset, the second dataset including aesthetic textual descriptions associated with a second plurality of images from the second dataset, and implementing the image aesthetic model (e.g., machine- learned models 120, 140) based on the pretraining in the first manner and the pretraining in the second manner, to perform one or more tasks with respect to an input query relating to at least one image. The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting one or more images having an aesthetic charactenstic specified by the input query when the input query includes a search request for one or more images having the aesthetic characteristic. The machine-learned model (e.g., machine-learned models 120, 140) may process the input query by determining an aesthetic rank of a plurality of candidate images having the aesthetic characteristic.

[0158] As an example implementation, the computer-implemented method 600 may include pretraining an image aesthetic model (e.g., machine-learned models 120, 140) in a first manner using a first dataset, the first dataset including textual descriptions associated with a first plurality of images from the first dataset, pretraining the image aesthetic model (e.g., machine-learned models 120, 140) in a second manner using a second dataset, the second dataset including aesthetic textual descriptions associated with a second plurality of images from the second dataset, and implementing the image aesthetic model (e.g., machine- learned models 120, 140) based on the pretraining in the first manner and the pretraining in the second manner, to perform one or more tasks with respect to an input query relating to at least one image. The one or more tasks may include the computing system (e g., computing systems 102, 130) outputting a caption generated by the machine-learned model (e.g., machine-learned models 120, 140) for the at least one image when the input query includes the at least one image. The machine-learned model (e.g., machine-learned models 120, 140) may process the query by generating the caption for the at least one image. [0159] As an example implementation, the computer-implemented method 600 may include pretraining an image aesthetic model (e.g., machine-learned models 120, 140) in a first manner using a first dataset, the first dataset including textual descriptions associated with a first plurality of images from the first dataset, pretraining the image aesthetic model (e.g., machine-learned models 120, 140) in a second manner using a second dataset, the second dataset including aesthetic textual descriptions associated with a second plurality of images from the second dataset, and implementing the image aesthetic model (e.g., machine- learned models 120, 140) based on the pretraining in the first manner and the pretraining in the second manner, to perform one or more tasks with respect to an input query relating to at least one image. The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting an aesthetic classification for the at least one image when the input query includes the at least one image. The aesthetic classification may be selected from a plurality of predetermined aesthetic classifications.

[0160] As an example implementation, the computer-implemented method 600 may include pretraining an image aesthetic model (e.g., machine-learned models 120, 140) in a first manner using a first dataset, the first dataset including textual descriptions associated with a first plurality of images from the first dataset, pretraining the image aesthetic model (e.g., machine-learned models 120, 140) in a second manner using a second dataset, the second dataset including aesthetic textual descriptions associated with a second plurality of images from the second dataset, and implementing the image aesthetic model (e.g., machine- learned models 120, 140) based on the pretraining in the first manner and the pretraining in the second manner, to perform one or more tasks with respect to an input query relating to at least one image. The one or more tasks may include the computing system (e.g., computing systems 102, 130) outputting a selection of a highest aesthetically ranked query image when a plurality of query images are included in the input query. The machine-learned model (e.g., machine-learned models 120, 140) may process the input query by aesthetically ranking each of the plurality of query images.

Experimental Results

[01 1] The disclosed image aesthetic models (e g., VILA-P and VILA-R) were implemented under various conditions and compared with results achieved by other existing models.

[0162] Source datasets utilized for obtaining the experimental results included LAION- 5B-English-Filtered, AVA Dataset, AVA-Captions, and AVA-Style. [0163] LAION-5B-English-Filtered is a 650M filtered subset from the English split in LAION-5B. LAION-5B is the largest publicly available dataset with 5B CLIP-filtered imagetext pairs. Basic filtering is applied to obtain the subset, removing data that are bad or non- informative, e.g., bad image size, bad aspect ratio, poor image content, and badly formatted text. For example, according to example embodiments the LAION-5B-English-Filtered maybe used for general image-text pretraining.

[0164] AVA Dataset is a large-scale data source for aesthetic visual analysis originated from the DPChallenge website. There are more than 250k images with user voting scores (1 to 10). It is the largest and most commonly used IAA benchmark. For IAA evaluation, the model performance on the available 19,928 AVA test images may be measured. For example, Spearman rank order correlation coefficient (SRCC) and Pearson linear correlation coefficient (PLCC) values may be obtained.

[0165] AVA-Captions contains the user comments for those AVA images crawled from the DPChallenge website, with some basic text filtering applied. It contains 230k images and 1.5M captions with an average of 5 comments per image. The training dataset includes 212,585 images paired with 1.2M captions. For evaluating the aesthetic comment generation quality, model performance is measured on 9,361 AVACaptions test images, and utilize commonly used captioning metrics BLEU, ROUGE, and CIDEr scores.

[0166] AVA-Style contains images with 14 photographic style labels: complementary colors, duo tones, hdr, image grain, light on white, long exposure, macro, motion blur, negative image, rule of thirds, shallow dof, silhouettes, soft focus, and vanishing point. In experiments described below, 2,809 testing images were used for quantifying the pretrained model’s capability on zero-shot aesthetic sty le classification tasks.

[0167] In implementing the disclosed image aesthetic models, a variant of CoCa, i.e., CoCa-Base, was utilized. An example image encoder included a ViT-B/16 containing 12 transformer layers with hidden dimension D = 768 and MLP size 3072. The image resolution was set to 224 X 224 with patch size 16 X 16, resulting in K = 196 image tokens. During training, random horizontal flipping and random cropping from 272 X 272 for data augmentation was utilized. An example unimodal text decoder includes 6 transformer layers with the same hidden dimension and MLP size. An example multimodal text decoder includes another 6 transformer layers.

[0168] For LAION pretraining, a batch size of 4096 was used and the model was trained for 500k steps using a learning rate 5e-4 with linear decay to zero, and 0.01 weight decay. For image aesthetic pretraining on AVA-Captions, a batch size of 128 was used and the model was trained for 500k steps using learning rate le-5 with linear decay to zero, and 0.04 weight decay. Example contrastive loss weight values include a value of a = 1 and example generative loss weight values include a value of P = 2. A trainable temperature T was set with an initial value of 0.07 for the contrastive loss. To finetune the rank-based adapter on AV A, a batch size of 128 was used and the model was trained for 30k steps using a learning rate le- 5 with linear decay to zero, and 0.01 weight decay. The Adafactor optimizer was implemented with Pi = 0.9; 2 = 0.999 for all experiments, and the models were trained using TPUv3 hardware.

[0169] Table 1 below illustrates results of the image aesthetic models disclosed herein (e.g., VILA-P and VILA-R) compared to various other methods. The upper section of Table 1 shows the baselines including the ranking method (Kong), distribution matching based approaches (AMP, NIMA, Zeng), customized neural networks (AFDC+SPP, GATxs-GATP, MLSP, MUSIQ, MaxViT), and semantic-aware methods (TANet, Niu). Overall, VILA-R achieves the best performance and surpasses the current methods (e.g., SOTA GATxi-GATP) by 1.6% and 1.3% in terms of SRCC (0.774 vs 0.762) and PLCC (0.774 vs 0.764), respectively. Moreover, VILA-R uses a resolution of 224X224 while others may need larger inputs. For example, MUSIQ uses the full-size image and two additional resolutions, yet it underperforms VILA-R.

Table 1

[0170] The Zero-shot Learning (ZSL) for IAA section in Table 1 shows the results of using the image-text pretrained model VILA-P for zero-shot IAA. For the single prompt setting, the similarity between the image and a single pair of prompt (“good image”, “bad image”) was computed, and the softmax normalized output was used for “good image” as the model ZSL score for IAA. For example, p_g and pb correspond to the normalized unimodal text embedding for the “good” and “bad” prompts respectively, and v corresponds to the normalized image contrastive embedding. The cosine similarity was computed and the softmax normalized score was used for “good image” as the final score r for IAA:

[0171] For an ensemble of prompts, an average ensemble of six pairs of prompts was used, composed of “good” or “bad” plus “image”, “lighting”, “composition”, “foreground”, “background”, and “content” (i.e., good/bad image, good/bad lighting, good/bad content, good/bad background, good/bad foreground, and good/bad composition,). For each pair, a score n, i = 1, . . . 6 was obtained and the average ensemble of the scores was used to get the final score r for IAA. Without any human label supervision, the ZSL model implemented via VILA-P outperforms several supervised baselines such as Kong, NIMA, and AFDC+SPP, with results of SRCC 0.657 and PLCC 0.663. Therefore, the disclosed image aesthetic model (e.g., VILA-P) can greatly alleviate human labeling costs when utilized in connection with ZSL.

[0172] Table 2 below illustrates the effects of image-text pretraining. For example, Table 2 validates the effects of the proposed image-text pretraining in which general pretraining followed by aesthetic pretraining achieves superior results. For example, the general pretraining and aesthetic pretraining are proceeded on the LAI ON subset and AVACaptions, respectively. With only the general pretraining, the model has suboptimal performance on the IAA task which verifies the assumption that image aesthetic information gets diluted by the vast amount of unrelated data from the web. Adding aesthetic pretraining greatly improves model performance in both zero-shot and finetuned settings. Both general and aesthetic pretraining have significant positive impacts on the final IAA task predictions. Regardless of the pretraining schema, adding the disclosed rank-based adapter allows the model to perform better on the IAA task using minimally tuned parameters.

Table 2

[0173] Table 3 illustrates the effectiveness of the rank-based adapter module implemented under various conditions. For example, different options of adapting the frozen VILA-P to downstream score-based IAA are compared. The first group shows regression baselines that predict either the single MOS score using a L2 loss, or the distribution of MOS scores using EMD loss. VILA-R outperforms both of them, showing the effectiveness of a rank-based target. In the second group different implementations of components in the rankbased adapter module are compared, “w/o Text Anchor” denotes using a learnable projection to replace the frozen text prompt embedding w_P. VILA-P performs better and it demonstrates the benefit of using the rich text embedding as a ranking anchor. For “w/o Residual”, a simple learnable projection without the residual is used, i.e., v = normalize(v^TH). Its performance confirms the intuition that slightly adjusting the image embedding may improve performance, thus learning the residual is easier. The final line shows that VILA-R can be further improved with finetuning the image encoder. Yet its performance gain comes at the cost of disturbing the generic pretrained weights, e.g. its ZSL performance on AVA-Style drops from 69.0% to 26.3% mAP. VILA-R enables effective IAA adaptation while inheriting the pretrained weights.

Table 3

[0174] Table 4 illustrates the effectiveness of the rank-based adapter module implemented in the VILA-R model when different values for the margin hyper-parameter m are used (for example, with respect to Eq. (7)) when tuning VILA-R. Overall, a value of m = 0. 1 provided the best performance.

Margin SRCC PLCC

•m = 0.01 0.769 0.769 0.770 0.770

0,774 (1.774

0.772 0.771 0.772 0.770

Table 4

[0175] Experimental results also confirm that the VILA-P model learns meaningful representations and is generalizable to other tasks. Performance of the VILA-P model was measured on zero-shot style classification and the quality of its generated aesthete comments.

[0176] In an example, ZSL perfomrance was tested on the AVA-Style test set to demonstrate that the image-text model trained on the noisy image-comment pairs captures abundant information for recognizing the diverse aspects such as composition, color, and style. Natural language prompts were manually curated based on the 14 class names and cosine similarities were used to approximate the probability that an image involves particular styles. For example, for a single prompt the 14 photographic style names were used as the language prompts: (“complementary colors”, “duo tones”, “hdr”, “image gram”, “light on white”, “long exposure”, “macro”, “morion blur”, “negative image”, “rule of thirds”, “shallow dof ’, “silhouettes”, “soft focus”, “vanishing point”). The cosine similarity between the prompt text embedding and the image embedding was used as the prediction score. For example, for an ensemble of prompts five sentences/phrases were manually curated that were frequently mentioned in the AVA-Caption user comments, for each of the sty les. These prompts either use synonyms (e.g. “color” and “colors”) of the styles or add more text contexts (e g., “i like the lines and fading or vanishing”). FIG. 7 depicts the example ensemble prompts 700 utilized for training the VILA model, according to example embodiments of the disclosure.

[0177] Table 5 below shows the results. For example, results on a “per-class” basis for each of the 14 style classes from AVA-Style are provided in Table 5. [0178] Methods in the first group use supervised learning and are trained on 1 Ik images with style annotations. Without such supervision, VILA-P achieves 69.0% ZSL mAP and it outperforms many supervised methods such as MNet (65.5%) and Lu et al. (64.1%). This demonstrates the power of the disclosed framework which enables open-set aesthetic information to be learned without any human label.

[0179] Table 5 also shows the general pretraining results are much lower than the ones with aesthetic pretraining. This again verifies that that aesthetic pretraining is advantageous for capturing rich aesthetic information.

Table 5

[0180] Table 6 below shows the captioning metrics of VILA-P on AVA-Captions test set. The disclosed method outperforms CWS and Yeo et al. in terms of BLEU-2, BLEU-3, BLEU-4, ROUGE, and CIDEr. Although it produces slightly lower BLEU-1 than CWS, BLEU-1 only measures precision of a unigram, and higher order BLEU-2, BLEU-3, BLEU-4 emphasize more on the fluency of generated sentences hence are better metrics. Besides, the supenonty of the disclosed method in terms of ROUGE and CIDEr denotes that the VILA model generates semantically more similar sentences to the real user comments.

VILA O.5O3 9.288 0.170 04 U 0.242 A974

Table 6

[0181] Qualitative examples obtained through implementation of the disclosed image aesthetic models (e.g., VILA-P and VILA-R) were also obtained.

[0182] FIG. 8 depicts images retrieved by a machine-learned model with respect to a prompt describing an image aesthetic style, according to example embodiments of the disclosure. For example, FIG. 8 shows the top-5 images 800 retrieved by queries “Bad photo” and “Good photo” from the AVA testing examples respectively. For “Bad photo”, the retrieved results involve bad cropping, indecent lighting, poor composition, distortion, and motion blur. In contrast, the “Good photo” group has visibly better aesthetic quality. These examples qualitatively demonstrate that the disclosed pretrained model captures the aesthetic related knowledge.

[0183] FIG. 9 depicts images retrieved by a machine-learned model with respect to various inputs describing a desired image aesthetic style, according to example embodiments of the disclosure. For example, FIG. 9 visualizes the AVA-Style predictions to qualitatively understand what the VILA model learned. Specifically, style names were used as queries to retrieve the top-5 images 900. Results show that the contrastive learning on noisy imagecomment pairs help the model understand low-level aesthetic atributes. For example, the learned model understands that “Macro” is a visual concept capturing finer details, regardless of the semantic objects such as lenses or insects. Another example is “HDR”, for which all retrieved photos have high dynamic range while portraying different semantic objects such as trees and cars.

[0184] FIG. 10 depicts images retrieved by a machine-learned model with respect to various inputs describing a desired image aesthetic style, according to example embodiments of the disclosure. For example, FIG. 10 is another example of style retrieval results 1000, in this case for all 14 style classes from AVA-Style, using style names as the query. VILA captures the color or intensity related information precisely. For example, the retrieved images for “Light on white”, “HDR”, and “Complementary colors” accurately depict the concept. Moreover, VILA also performs well at recognizing the photographic composition such as “Vanishing point”.

[0185] FIG. 11 depicts image captions generated by a machine-learned model with respect to various images, according to example embodiments of the disclosure. For example, FIG. 11 shows comments 1100 generated by the disclosed VILA model for various images. The VILA model generates diverse captions conditioned on the images, providing aesthetic comments including “color”, “sharpness”, “refraction”, “macro”, “details”, and “texture”. In addition, the disclosed VILA model is able to criticize the “refraction of light” in the top-right example, which makes sense from an aesthetic perspective.

Additional Disclosure

[0186] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0187] While the subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the disclosure cover such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method, comprising: receiving, by a computing system, a query relating to at least one image; processing the query, by a machine-learned model provided by the computing system, the machine-learned model being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset; and performing one or more tasks based on the processing, by the computing system, the one or more tasks comprising one or more of: outputting an aesthetic score with respect to the at least one image, outputting one or more images having an aesthetic characteristic specified by the query, outputting a caption generated by the machine-learned model for the at least one image, outputting an aesthetic classification for the at least one image, or outputting a selection of a highest aesthetically ranked query image when a plurality of query images are included in the query, wherein the first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset.

2. The computer-implemented method of claim 1, wherein a number of the first plurality of images from the first dataset is greater than a number of the second plurality of images from the second dataset.

3. The computer-implemented method of claim 1, wherein the query includes the at least one image, and the one or more tasks include outputting the aesthetic score with respect to the at least one image.

4. The computer-implemented method of claim 3, wherein processing the query includes the machine-learned model determining the aesthetic score with respect to the at least one image based on the machine-learned model utilizing a rank-based adapter which optimizes a relative aesthetic score ranking between the at least one image and at least one other image.

5. The computer-implemented method of claim 1, wherein the query includes a search request for one or more images having the aesthetic characteristic, and the one or more tasks include outputting the one or more images having the aesthetic characteristic.

6. The computer-implemented method of claim 5, wherein processing the query includes the machine-learned model determining an aesthetic rank of a plurality of candidate images having the aesthetic characteristic, and the one or more images included in the output are ranked highest aesthetically among the plurality of candidate images according to an aesthetic score generated by the machine- learned model.

7. The computer-implemented method of claim 6, wherein the aesthetic characteristic includes a style of an image.

8. The computer-implemented method of claim 1 , wherein the query includes the at least one image, processing the query includes the machine-learned model generating the caption for the at least one image, and the one or more tasks include outputting the caption generated by the machine-learned model for the at least one image.

9. The computer-implemented method of claim 1 , wherein the query includes the at least one image, the one or more tasks include outputting the aesthetic classification for the at least one image, the aesthetic classification being selected from a plurality of predetermined aesthetic classifications.

10. The computer-implemented method of claim 1, wherein the query includes the plurality of query images, processing the query includes the machine-learned model aesthetically ranking each of the plurality of query images, and the one or more tasks include outputting the selection of the highest aesthetically ranked query image from among the plurality of query images.

11. A computer-implemented method for training an image aesthetic model, comprising: pretraining the image aesthetic model in a first manner using a first dataset, the first dataset including textual descriptions associated with a first plurality of images from the first dataset; pretraining the image aesthetic model in a second manner using a second dataset, the second dataset including aesthetic textual descriptions associated with a second plurality of images from the second dataset; and implementing the image aesthetic model based on the pretraining in the first manner and the pretraining in the second manner, to perform one or more tasks with respect to an input query relating to at least one image, the one or more tasks comprising one or more of: outputting an aesthetic score with respect to the at least one image, outputting one or more images having an aesthetic characteristic specified by the input query, outputting a caption generated by the machine-learned model for the at least one image, outputting a respective aesthetic classification for the at least one image, or outputting a selection of a highest aesthetically ranked query image when a plurality of query images are included in the input query.

12. The computer-implemented method of claim 11, wherein a number of the first plurality of images from the first dataset is greater than a number of the second plurality of images from the second dataset.

13. The computer-implemented method of claim 11, wherein the input query includes the at least one image, and the one or more tasks include outputting the aesthetic score with respect to the at least one image.

14. The computer-implemented method of claim 13, wherein implementing the image aesthetic model includes the image aesthetic model determining the aesthetic score with respect to the at least one image based on the image aesthetic model utilizing a rank-based adapter which optimizes a relative aesthetic score ranking between the at least one image and at least one other image.

15. The computer-implemented method of claim 11, wherein the input query includes a search request for one or more images having the aesthetic characteristic, and the one or more tasks include outputting a predetermined number of candidate images having the aesthetic characteristic by the image aesthetic model determining an aesthetic rank of a plurality of candidate images having the aesthetic characteristic and selecting the predetermined number of candidate images which are ranked highest aesthetically among the plurality of candidate images according to an aesthetic score generated by the image aesthetic model for each of the plurality of candidate images.

16. The computer-implemented method of claim 15, wherein the aesthetic characteristic includes a style of an image.

17. The computer-implemented method of claim 11, wherein the input query includes the at least one image, implementing the image aesthetic model includes the image aesthetic model generating the caption for the at least one image, and the one or more tasks include outputting the caption generated by the image aesthetic model for the at least one image.

18. The computer-implemented method of claim 1 1 , wherein the input query includes the at least one image, and the one or more tasks include outputting the aesthetic classification for the at least one image, the aesthetic classification being selected from a plurality of predetermined aesthetic classifications.

19. The computer-implemented method of claim 11, wherein the input query includes the plurality of query images, implementing the image aesthetic model includes the image aesthetic model aesthetically ranking each of the plurality of query images, and the one or more tasks include outputting the selection of the highest aesthetically ranked query image from among the plurality of query images.

20. A computing system, comprising: one or more processors; and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: receiving a query relating to at least one image, processing the query, by a machine-learned model provided by the computing system, the machine-learned model being pretrained in a first manner using a first dataset and pretrained in a second manner using a second dataset, and performing one or more tasks based on the processing, the one or more tasks comprising one or more of: outputting an aesthetic score with respect to the at least one image, outputting one or more images having an aesthetic characteristic specified by the query, outputting a caption generated by the machine-learned model for the at least one image, outputting an aesthetic classification for the at least one image, or outputting a selection of a highest aesthetically ranked query image when a plurality of query images are included in the query, wherein the first dataset includes textual descriptions associated with a first plurality of images from the first dataset and the second dataset includes aesthetic textual descriptions associated with a second plurality of images from the second dataset.