RU2742602C1 - Recognition of events on photographs with automatic selection of albums - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to image processing and analysis methods and can be used for organizing photo galleries in mobile systems. Disclosed is a method of recognizing events on photographs with automatic selection of albums. According to method automatically assign album boundaries, estimating similarity between degrees of confidence for each pair of consecutive photos in gallery, wherein similarities between photographs are calculated as distance between degrees of confidence for each type of event, estimated by classification of each photograph, or similarity between photographs is calculated as sum of similarities between degrees of confidence for each type of event, estimated on the basis of classification of each photo, and similarities between their locations, if the EXIF (Exchangeable Photo File Format) data of said photos contain location information.

EFFECT: technical result is to improve the accuracy of identifying events by combining successively taken photographs into albums with similar content with subsequent classification of each album based on the neuronal attention mechanism.

11 cl, 5 dwg, 7 tbl

Description

The technical field to which the invention relates

The invention relates to the field of computer technology, in particular, to methods for processing and analyzing images, and can be used to organize a photo gallery in mobile systems.

Description of the prior art

Recently, thanks to the rapid growth of social networks, cloud services and mobile technologies, people have taken far more photos than ever before [13]. To organize a personal collection, photographs are usually assigned to albums in accordance with some event. Photo organization systems (Apple iPhoto, Google Photos, etc.) allow the user to quickly find the desired photos, and also increase the efficiency of working with the gallery [27]. Today, these systems usually involve content analysis of photos and automatic association of each photo with different tags (description of the scene, people, objects, places, etc.). Such analysis can be used not only to selectively extract photographs by a specific tag in order to preserve pleasant memories of certain episodes of a user's life [32], but also to provide personalized recommendations that help users find relevant objects in large collections. The development of such systems requires a careful analysis of the user approach to modeling [26]. A large gallery of photos on a mobile device can be used to understand user interests such as sports, tech, fitness, clothing, cars, food, travel, pets, and more. [12, 20].

The present invention focuses on one of the most difficult parts of the photo organizing mechanism, namely the task of recognizing events in photographs [1] to extract events such as holidays, sporting events, weddings, various activities, etc. The term "event" can be defined as a category that captures "the complex behavior of a group of people interacting with a variety of objects, taking place in a certain setting" [32]. There are two different tasks in the field of event recognition. The first task is focused on processing individual photographs, i.e. the event is viewed as a complex scene with large variations in appearance and structure [32]. The second task is aimed at predicting the categories of events in a group of photographs (album) [4]. In the latter case, it is assumed that all photographs in the album have weak markup [2], while each photograph may have a different importance [33]. In practice, however, only the photo gallery is available, so the second approach requires the user to manually select albums. Another option uses location-based album creation if GPS tags are enabled. In both cases, the use of album-based event recognition is limited or even impossible.

The essence of the invention

In the present invention, a new formulation of the event recognition problem is studied: it is required to predict the categories of events in the gallery of photos for which albums (groups of photos corresponding to one event) are unknown. A new two-stage approach is proposed. First, vector representations (features) are extracted from each photo using a pretrained convolutional neural network. These vector representations are classified individually. Classifier scores are used to group consecutive photographs into multiple clusters. Finally, the vector representations of the photographs in each group are aggregated into a single descriptor using the neural mechanism of attention. This mechanism is a weighting scheme for the linear union of all vector representations in the input set. The weights are calculated adaptively using a feedforward neural network. Due to the attention mechanism, aggregation is invariant with respect to the order of images and does not depend on the number of images in the input set. This algorithm can be extended as needed to improve the classification accuracy of each photo in the group. In contrast to the usual fine-tuning (additional training) of convolutional neural networks (CNN), it is proposed to use the compilation of an image caption, that is, a generative model that converts photographs into textual descriptions. They are unitary encoded and converted into a sparse vector representation suitable for training an arbitrary classifier. Experimental work with the Photo Event Collection and the Multi-Label Curation Flickr Events dataset showed that the accuracy of the proposed approach is 9-20% higher than recognizing events in individual photographs. In addition, the proposed method has a 13-16% lower error probability than the classification of groups of photographs obtained using hierarchical clustering. It has been experimentally shown that photo captions trained on the Conceptual Captions dataset can be classified more accurately than vector representations from an object detector, although both appear not to be as meaningful as vector representations based on CNN. However, it is possible to combine the proposed approach with conventional CNNs into an ensemble to obtain the highest accuracy for multiple event datasets .

Thus, the present invention is based on a novel event recognition problem in which a gallery of photos is given and it is known that this gallery contains ordered albums with unknown boundaries. It is proposed to automatically assign these boundaries based on the analysis of the visual content of successive photos in the gallery. Then sequential photographs are grouped and the descriptor of each group is calculated using the attention mechanism from the neural aggregation module [37]. Finally, this approach is expanded as follows. Despite the traditional use of CNN as discriminatory models in the program structure of the classifier, it is proposed to borrow generative models to represent the input photograph in another area. In particular, the well-known methods of automatic caption to an image [14] are used, which generate textual descriptions of photographs. The main contribution of the invention is to demonstrate that the generated descriptions can be fed to the input of a classifier in an ensemble in order to increase the accuracy of event recognition compared to traditional methods. Although the proposed visual representation is not as meaningful as the vector representations extracted using additional trained CNNs, it is better than the outputs of object detectors [20].

The task of recognizing events in personal collections of photographs is to recognize an event not in a single photograph, but in the entire album [29]. Events and sub-events in the photographs of the subsequence are identified in [8] by integrating the optimized linear programming with the caption image color descriptor. In [5], the hidden Markov models (Stopwatch Hidden Markov) were used. by treating the photos in the album as sequential data. Event-relevant object detectors were trained on the holiday dataset [29]. The holidays were then classified based on the object detector outputs. Article [2] examines the presence of irrelevant images in an album using multi-instance learning methods. In [33], an iterative update procedure is presented for predicting the type of event and assessing the importance of the image in the Siamese network. The authors of this article used CNN, which recognizes the type of event, and a long short term memory (LSTM) sequence event classifier throughout the album. In addition, they successfully applied this method to train representative deep features for analysis of a set of images [34]. The latter approach focuses on the assessment of co-occurrences and frequency of features, so that the temporal coherence of photographs in the album is not required. In [13], a model of event recognition from large to small hierarchical levels using multi-granular features [24] based on the attention network that studies the representations of photo albums is proposed. The efficiency of re-detecting expected photographs in mobile phones has been improved by using a method for classifying personal photographs based on the relationship of time and location of the photograph to specific events [10].

Album boundary information is not always available, as the gallery contains an unstructured list of photos sorted by the time they were taken. In this case, you can use the existing methods of recognizing events in individual photographs [1]. As with other areas of computer vision, the most common approach predominantly uses CNN-based architectures . For example, four different layers of retrained CNN were used to extract features and perform linear discriminant analysis to win the ChaLearn LAP 20 cultural event recognition competition [9]. To improve the accuracy of event recognition, bounding boxes of detected objects are projected onto multiscale spatial maps [35]. In [32], a new iterative selection method was introduced to identify a subset of the classes most relevant for the transmission of deep representations trained on datasets of objects (ImageNet) and scenes (Places2).

A method is proposed for recognizing events in photographs with automatic selection of albums, which consists in the following: automatically assign boundaries of albums based on the visual content of consecutive photographs in the gallery; group consecutive photos from the gallery into albums; calculating the descriptor of each album using the attention mechanism from the neural aggregation module; recognize the event type tag of each album by supplying its descriptor to the input of the classifier; recognize the event in the photos in the gallery by assigning an appropriate event type tag to all the photos in each album. At the same time, when automatically assigning the boundaries of albums: assess the similarity between the degrees of confidence for each pair of consecutive photos in the gallery; if the similarity does not exceed the specified threshold, then both photos are included in the same album. In this case, the similarity between the photographs is calculated as the distance between the degrees of confidence for each type of event, assessed by the classification of each photograph. In this case, the similarity between photographs is calculated as the sum of the similarities between the degrees of certainty for each type of event, assessed based on the classification of each photograph, and the similarity between their locations, if the EXIF (Exchangeable Photo File Format) data of these photographs contains location information. In this case, the predetermined threshold is calculated automatically during the preliminary training procedure using the provided training set of albums.

Also proposed is a method for recognizing an event in a photograph with its representation in a text area, which consists in the following: calculating vector representations of a photograph by feeding its RGB (red-green-blue) representation to the input of a convolutional neural network; use the method of automatically composing an image caption to generate textual descriptions of a photograph based on its vector representations; encode the generated signature; feed the generated text descriptions to the input of the event classifier; recognize the event by assigning the classifier output to the event tag of the corresponding photograph. In this case, the generated signature is encoded as a sparse vector using unitary coding of the textual description of the photograph: the v-th component of the vector is equal to 1, only if the generated signature contains at least one v-th word from the dictionary. According to the proposed method, the outputs of the generated text description classifier and the traditional vector representation classifier are additionally combined into an ensemble to improve the accuracy of event recognition. In this case, the outputs of the classifiers are obtained as follows: the estimates of the degree of confidence for all types of events for the photograph are calculated by feeding its vector representations into an arbitrary classifier; calculating confidence scores for all types of events for the photograph by supplying a sparse vector of the unitary encoded text description to an arbitrary classifier; combine outputs of classifiers of vector representations and texts in an ensemble based on simple voting with soft aggregation. The textual description of the photograph is generated as follows: the extracted vector representations and the sequence of previously generated words are fed to a recurrent neural network (RNN) to predict the next word in the textual description of the photograph; map numbers generated by RNN to real words from the dictionary; select a subset of the vocabulary by selecting the most common words in the training data with the optional exclusion of stop words. In the proposed method, a predicted word is additionally added to this sequence of previously generated words, and the extracted visual vector representations and this sequence are fed to the same RNN. In this case, the training sample is a set of photographs with a known type of event. In this case, the traditional classifier should be preliminarily trained to recognize events in the training sample, in which each photograph is presented along with its own extracted visual vector representations. In this case, in particular, the aggregated degrees of confidence are calculated as a weighted sum of the estimates of the degrees of confidence, and the decision is made in favor of the class with the maximum aggregated degree of confidence.

BRIEF DESCRIPTION OF DRAWINGS

The above and / or other aspects will become more apparent from the description of exemplary embodiments of the invention with reference to the accompanying drawings.

FIG. 1 depicts a proposed approach for gallery-based event recognition.

FIG. 2 depicts an attention-based neural network for vector representations from MobileNet v2.

FIG. 3 depicts a demo graphical interface for mobile devices.

FIG. 4 depicts a proposed pipeline for event recognition based on captioning an image.

FIG. 5 depicts exemplary event recognition results.

DETAILED DESCRIPTION

The invention can be used in photo management software that can optionally be implemented on, for example, a computer readable medium to automatically select albums from an unlabeled set of photographs and associate each album with specific tags (types of events or scenes). The present invention can be implemented in software and / or hardware in any device such as a smartphone, mobile phone, tablet PC, laptop, computer, and the like.

The technical effect is to significantly improve the accuracy of event recognition by combining sequentially taken photographs into albums (i.e., groups of sequential photographs with similar content), followed by the classification of each album based on the neural mechanism of attention. To find similar photographs, instead of the usual comparison of visual vector representations, an assessment of the degrees of confidence of the event classifiers is used. In addition to traditional visual vector representations, their automatically generated text descriptions (signatures) are used to improve the accuracy of the ensemble of classifiers by increasing the diversification of visual vector representations.

The suggested solution is as follows:

For each photo in the gallery, the following processing is performed. Vector representations of a photo are calculated by feeding its RGB (red-green-blue) representation to the input of a convolutional neural network. It is proposed to use an additional representation of a photograph using methods of automatic caption to an image to increase the diversification of visual vector representations. In particular, a signature (textual description) of a photo is created by feeding its vector representations into a specially trained recurrent neural network. Next, the generated signatures are unitary encoded. The degree of confidence for each type of event is assessed by classifying a photograph using an ensemble consisting of a traditional classifier of vector representations and a classifier of unitary coded captions to photographs. The ensemble of the traditional classifier of vector representations and the classifier of unitarily encoded captions to photographs is as follows: in the traditional approach, vector representations of photographs are classified, extracted using convolutional neural networks; In the proposed approach, a textual description of the photograph is first generated, then this text is converted into numerical form using the unitary coding method, and the resulting vector is classified by known methods.

The similarity between each pair of consecutive photos in the gallery is evaluated. Instead of the traditional computation of the similarity between vector representations extracted by the convolutional neural network, it is proposed to compute the similarity between the normalized confidence scores described above. To normalize the estimates of the degrees of confidence, their L2-norm is calculated as the square root of the sum of the squares of all the estimates of the degrees of confidence, and each estimate is divided by this L2-norm. If the EXIF (Exchangeable Photo File Format) data of these photos contains location information, then similarities between their locations can be added to the similarities between the normalized confidence levels. Successive photos in the gallery are automatically grouped into albums according to the following rule: if the similarity does not exceed a specified threshold, then it is assumed that both photos are included in the same album. This threshold is calculated automatically during the pre-training procedure using the provided training album set. The training albums are taken from the available training sample, for example, from the Photo Event Collection (PEC) or from the Multi-Label Curation of Flickr Events (ML-CUFED) dataset, they are required to train the classifier.

Unlike existing methods for detecting events in photo collections, an event type tag is assigned to all photos in each album using a special attention-engine neural network trained to classify events in a set of photos from one album. The invention improves accuracy over the conventional photo tagging technique, because events are recognized simultaneously for all photos in selected albums and errors for individual photos can be corrected automatically.

A new trend in photo organization services is the annotation of personal photo albums [8]. In [17], a method for the hierarchical organization of photos on a smartphone according to topics and thematic categories is proposed, based on the integration of a convolutional neural network (CNN) and topic modeling for the classification of photos. In [16], automatic hierarchical clustering and a solution for choosing the best photo are proposed for simulating user decisions when organizing or clustering similar photos in albums. In [24], the organization of photo albums for predicting user preferences on a mobile device is considered.

Unfortunately, the accuracy of classification of events in static photographs [32] is usually much lower than the accuracy of recognition from albums [33]. Therefore, the present invention proposes to concentrate on other suitable visual vector representations extracted using generative models, in particular on methods for automatically composing an image caption. There is a wide range of signature compilation applications, from automatically generating descriptions for photos posted on social networks to extracting photos from databases using generated text descriptions [30]. Image captioning techniques are typically based on a codec neural network that first encodes a photo into a fixed-length vector representation using a pretrained CNN, and then decodes that representation into a caption (natural language description). During the training of the decoder (generator), the input photograph and its true textual description are fed into the neural network as input, and the desired network output is a single unitary coded description. The description is encoded using text vector representations in the "Vector representation (lookup table)" layer [11]. The generated vector representations of the photo and the text are combined by concatenation or addition and they form the input for the decoding part of the network. Typically a Recurrent Neural Network (RNN) layer is provided, followed by a fully connected layer with a Soft-max output layer.

One of the first successful models, Show and Tell [31], won the first MS COCO Image Captioning Challenge in 2015. In the decoder part, it uses RNN with Long Short Term Memory (LSTM) blocks. Its improved Show, Attend and Tell model [36] includes a soft attention mechanism to improve the quality of signature generation. The Neural Baby Talk image caption model [18] is based on the creation of a template with slot locations explicitly tied to specific areas of the image. These slots are then populated with visual concepts identified in the object detectors. The foreground areas are obtained using the Faster-RCNN [21], and the decoder is an attention mechanism LSTM. A multimodal recurrent neural network (mRNN) [19] is based on the Inception network to extract vector representations of photographs and deep RNN to generate sentences. One of the best models at present is the Auto-Reconstructor Network (ARNet) [6], which uses the Inception-V4 [28] encoder and the LSTM-based decoder. There are two pretrained models with greedy search (ARNet-g) and ray search (ARNet-b) with size 3 designed to generate the final caption for each input photo.

MATERIALS AND METHODS

Formulation of the problem

As noted above, the problem of recognizing events in each photo in the gallery is solved. At the same time, at the first stage, albums (groups of consecutive photographs of the same event taken by the user) are automatically selected based on the analysis of the content of the photograph. Then the events corresponding to each selected album are recognized. After that, the event recognized for this album is displayed in each photo of this album. This makes it possible, as a result, to recognize events more accurately than to simply classify the events in each individual photograph.

из галереи входного пользователя к одной из С>1 категорий событий (классов). В данном случае T≥1 - общее количество фотографий в галерее. Имеется обучающий набор N≥1 альбомов для обучения классификатора событий. n-й обучающий (эталонный) альбом образован коллекцией Ln фотографий

. Предполагается, что предоставлена метка класса

каждого n-го альбома, т.е. что альбом связан с одним конкретным типом события.This subsection describes a technological mechanism that allows to solve this problem by sequential processing of photographs similar to cluster analysis using the Basic Sequential Algorithmic Scheme (BSAS) [3]. The main task can be formulated as follows. it is required to assign each photo

from the gallery of the input user to one of the C> 1 categories of events (classes). In this case, T≥1 is the total number of photos in the gallery. There is a training set of N≥1 albums for training the event classifier. The n-th training (reference) album is formed by a collection of Ln photos

... Assumes class label provided

every nth album, i.e. that the album is associated with one particular type of event.

The usual recognition of events in individual photographs [32] is a special case of the problem formulated above, if T = l. Consequently, this problem can be solved by classifying events separately in each t-th photograph (t = 1,2, ..., T). However, you can take into account that the gallery {Xt} in this problem is not a random collection of photographs, but can be represented as a sequence of disconnected albums. Each photo in the album is associated with the same event. Unlike recognizing events in an album, we do not know the boundaries of each album, i.e. number of the first t1 and last t2 photos from the gallery, for which it is guaranteed that all photos X _t , t = t ₁ + 1,…, t ₂ correspond to the given album. This problem has a number of characteristics that make it extremely difficult compared to previously studied problems. One of these characteristics is the presence of irrelevant or irrelevant photographs, which, in principle, can be associated with any event [1]. These photographs are easily detected in attention-based models [13, 37], but can significantly affect the accuracy of automatic album delineation.

. Метки c_n - это номера события, соответствующие всему альбому (коллекции фотографий). Затем можно обучить произвольный известный классификатор, который способен возвращать вектор оценок степени уверенности для каждого класса, вместо того, чтобы предсказывать только метку класса. Если L достаточно мало для обучения глубокой CNN (сверточной нейронной сети) с нуля, то можно применить перенос обучения или адаптацию домена [11]. В этих методах используется большой внешний набор данных, например, ImageNet-1000 или Places2 [38], для предварительного обучения глубокой CNN. Поскольку особое внимание уделяется автономному распознаванию на мобильных устройствах, целесообразно использовать такие CNN, как MobileNet vl/v2 [15, 22]. Заключительным этапом в переносе обучении является дообучение этой нейронной сети на наборе данных X. Данный этап включает в себя замену последнего слоя предварительно обученной CNN новым слоем с активациями Softmax и выводами С. Во время процесса классификации каждая входная фотография X_t подается в дообученную CNN для вычисления оценок (прогнозов на последнем слое:In this case, the basic approach is to classify all T photographs independently of each other so that the decision for each photograph does not affect the decision for any other photograph. In this case, training albums are usually expanded into a set X = {X ₁ (1),…, X ₁ (L ₁ ), X ₂ (1),…, X ₂ (L ₂ ),…, X _N (1), …, X _N (L _N )} from L = L ₁ +… + L _N photos in such a way that the label c _{n of} the collection level of the n-th album is assigned to the labels of each 1st photo

... Tags c _n are event numbers corresponding to the entire album (collection of photos). An arbitrary known classifier can then be trained that is capable of returning a vector of confidence scores for each class, rather than predicting only the class label. If L is small enough to train deep CNN (Convolutional Neural Network) from scratch, then transfer learning or domain adaptation can be applied [11]. These methods use a large external dataset such as ImageNet-1000 or Places2 [38] to pre-train deep CNN. Since special attention is paid to autonomous recognition on mobile devices, it is advisable to use such CNNs as MobileNet vl / v2 [15, 22]. The final stage in the transfer learning is the additional training of this neural network on dataset X. This stage includes replacing the last layer of the pretrained CNN with a new layer with Softmax activations and C outputs. During the classification process, each input photo X _t is fed to the retrained CNN for computation estimates (forecasts on the last layer:

.This procedure can be modified by replacing the C logistic regressions in the last layer with a more complex classifier, such as a random forest (RF), support vector machine (SVM), or gradient boosting. In this case, vector representations [25] are extracted using the outputs of one of the last layers of the pretrained CNN. In particular, photographs X _t and X _n (l) are fed to CNN, and the outputs of the penultimate layer are used as D-dimensional vectors of vector representations x _t = [x _{1; t} , ..., x _{D ;; t} ] and x _n ( l) = [x _{n; 1} (l),…, x _{n; D} (l)], respectively. Such deep learning-based vector representation extraction blocks allow training of an arbitrary classifier. The t-th photo is fed into this classifier to obtain C-dimensional estimates of the degree of confidence

...

, вычисленные любым из вышеперечисленных способов, используются для принятия решения в пользу наиболее вероятного класса:In conclusion, degrees of confidence

calculated by any of the above methods are used to decide in favor of the most likely class:

Recognition of events in the photo gallery.

FIG. 1 shows a suggested approach for gallery-based event recognition. It should be noted that all described blocks can be implemented in software and / or hardware.

The CNN Extraction block does the following: extracts visual vector representations (vector representations) by feeding an RGB representation of each photo from the gallery to the input of a convolutional neural network to compute the outputs of one of its last layers (usually the penultimate one).

In the "Classifier" block, the following is performed: estimates of the degree of confidence for all types of events for each photo are calculated by feeding their vector representations into an arbitrary classifier. This classifier must be pre-trained to recognize events in the training set. The confidence score vector is normalized using the L2-norm.

In the "Sequential Cluster Analysis" block, the following is performed: for each photo from the gallery, the similarity is calculated between its normalized confidence scores and the normalized confidence scores of the next photo. If the EXIF (Exchangeable Photo File Format) data of these photos contains location information, then their locations can be added to the similarity between the normalized confidence ratings. If this similarity exceeds the specified threshold, the photos will be included in different albums, and a border will be established between the albums of these two consecutive photos.

In the "Neural model of attention" block, the following is done: for each sequential pair of album boundaries extracted at the previous stage, visual vector representations of all photos between these boundaries are obtained. This set of vector representations is fed into a neural model of attention, which predicts one class of events for a set of photographs. The predicted event class is assigned to all photos between these boundaries.

классификатора. Затем используется последовательный анализ по аналогии с кластеризацией BSAS [3] в блоке "Последовательный кластерный анализ" для определения последовательности степеней уверенностей {

} для получения границ альбомов. В частности, вычисляются сходства между степенями уверенностями всех последующих фотографий p(

,

). При этом можно использовать любое подходящее сходство, например, эвклидово, Минковского, хи-квадрат, расхождения Кульбака-Лейблера и Дженсена-Шеннона и т.п. Если сходство не превышает заданный порог

, то предполагается, что обе фотографии включены в один и тот же альбом. Если в данных EXIF (Exchangeable Photo File Format) этих фотографий содержится информация о местоположении, то к p(

,

) можно добавить сходство между их местоположениями, чтобы получить окончательное сходство, сопоставимое с порогом. В противном случае граница между двумя альбомами устанавливается в t-й позиции. В результате получаются границы

,

, так что k-й альбом содержит фотографии

, где t₀₌0. См. фиг. 2.In this case, the "Vector Representation Extractor" first calculates the vector representations x _{t of} each individual t-th photograph, as described above. The "Classifier" block evaluates the degree of confidence

classifier. Then sequential analysis is used by analogy with BSAS clustering [3] in the "Sequential cluster analysis" block to determine the sequence of degrees of confidence {

} to get the boundaries of albums. In particular, the similarities between the degrees of confidence of all subsequent photographs p (

,

). In this case, you can use any suitable similarity, for example, Euclidean, Minkowski, chi-square, Kullback-Leibler and Jensen-Shannon divergences, etc. If the similarity does not exceed the specified threshold

, it is assumed that both photographs are included in the same album. If the EXIF (Exchangeable Photo File Format) data of these photos contains location information, then p (

,

) you can add similarities between their locations to get a final similarity comparable to the threshold. Otherwise, the border between the two albums is set at the t-th position. The result is boundaries

,

so that the kth album contains photos

, where t _{0 =} 0. See Fig. 2.

At the second stage, the final descriptor x (k) of the kth album is created as a weighted sum of individual vector representations x _t :

where the weights w can depend on the vector representations x _t . In this case, the average pooling (grouping) (AvgPool) with equal weights is usually used, so that the usual calculation of the average vector representation is implemented.

Algorithm 1 Proposed gallery-based event recognition

}, классификатор C, порог

o. Input: input gallery

}, classifier C, threshold

o.

Output: event labels c * (t) of all input images.

1: Assign K: = 0, initialize boundary list B: = □

повторить:2: for each input image

repeat:

3: Feed t-th image to CNN and compute vector representations x _t

4: Calculate degrees of confidence p _t using classifier C

, то5: if t = 1 or

then

6: Assign K: = K + 1, add t-1 to list B

7: finish if

8: finish for

9: Add T to list B

повторить:10: for each extracted album

repeat:

в11: Submit input images

in

attention net (2) - (3) and get event class c *

12: Assign c * {t): = c * for all

13: finish for

{1,…,T}14: return labels c * (t),

{1, ..., T}

However, the present invention proposes to train the weights w (x _t ), in particular, using the attention mechanism from the neural aggregation module, which was previously used only for video recognition [37]:

}.Here q is the trained D-dimensional vector of weights. A dense (fully connected) layer is attached to the resulting descriptor x (k), and the entire neural network (Fig. 2) is trained end-to-end using the provided training set of N≥1 albums. The class of the event predicted by this network in the "Neural model of attention" block (Fig. 1) is assigned to all photos

}.

The complete classification and training procedures are presented in Algorithm 1 and Algorithm 2, respectively. For simplicity, it should be noted that in the second algorithm, at step 17, the event classification algorithm 1 is called. However, to speed up the computations, it is recommended to pre-compute the pairwise similarity matrix between the confidence estimates of all training photographs so that during model training it is not required to extract vector representations (steps 3-4 in Algorithm 1) and calculate the similarity.

The present invention implements the entire pipeline (Fig. 1) in the publicly available demo Android application (https: drive.google.comopen? Id = laYN0ZwU90T8ZruacvND01hbIaJS4EZLI) (Fig. 3), developed earlier to extract user preferences by processing all photos from the gallery in the background stream [26]. Similar events found in photographs taken on the same day are combined into high-level recordings for the most important events. Only those scenes / events are displayed for which at least two photos exist and the average score of the scene / event forecasts for all photos of that day exceeds a certain threshold. This threshold is set automatically during the learning procedure (steps 16-22 in Algorithm 2). FIG. 3a shows an example screenshot of the main user interface. You can click on any column in this histogram to display a new form with detailed categories (Fig. 3c). If you click on a specific category, a "display" form will appear containing a list of all photos from the gallery of that category (Fig. 3c). In this list, events are grouped by date and it is possible to select a specific day.

Algorithm 2. Training procedure according to the proposed approach

1: for each album n∈ {1,…, N} repeat

2: for each image l∈ {1,…, L _n } repeat

3: Feed image X _n (l) to CNN and compute vector representations x _n (l)

4: finish for

5: finish for

6: Train classifier C using expanded training set X vector representations

7: Train the attention network (2) - {3) using the fixed-size subsets S of all training feature sets {x _n (l)}

8: for each album n∈ {1,…, N} repeat

9: for each image l∈ {1,…, L _n } repeat

10: Predict estimates of the degree of confidence p _n (l) for vector representations x _n (l) using classifier C

11: finish for

12: finish for

13: Perform random permutation of all indices {1,…, N} to obtain the sequence (n _i ,…, n _N )

={X_n1(1),…,X₁(L_n1),…,X_nN(1),…,X_nN(L_nN)}14: Expand all training vector representations using this permutation:

= {X _n1 (1),…, X ₁ (L _n1 ),…, X _nN (1),…, X _nN (L _nN )}

15: Assign ρ: = 0, α *: = 0

16: for each potential threshold with repeat

, С и порогом с17: Call algorithm 1 with parameters

, С and threshold с

18: Calculate precision b using predictions for all training images

19: if α * <α, then

20: Assign α *: = α, ρ ₀ : = ρ

21: finish if

22: finish for

23: return classifier C, attention net, threshold from ₀

Recognition of events in individual photos

The problem of recognizing an event in individual photographs can be formulated as a common problem of image recognition. It is necessary to assign each photo X from the gallery to one of the C> 1 categories of events (classes). To train the classifier, a training sample of N≥1 photos X = {X _n | n∈ {1,…, N}} with known event labels with _n ∈ {l,…, C} is available. Sometimes educational photographs of the same event are associated with an album [5, 33]. In this case, the training albums are expanded into set X so that an album collection level tag is assigned to the tags of each photo in that album. This task has several characteristics that make it extremely difficult compared to album-based event recognition. One of these characteristics is the presence of irrelevant or irrelevant photographs that can be associated with any event [1]. These photographs can be identified using attention-based models when the entire album is available [13], however, the recognition of events in individual photographs can be significantly affected.

Since N is usually quite small, learning transfer can be applied [11]. First, deep CNN is trained on a large dataset, for example, ImageNet or Places [38]. Then this CNN is fine-tuned (retrained) to the training set X, that is, the last layer is replaced with a new one with Softmax activations and C outputs. Each input X photo is classified by feeding it to the retrained CNN to calculate the C outputs of the output layer, i.e. estimates of posterior probabilities for all categories of events. This procedure can be modified by extracting deep vector representations of photographs using the outputs of one of the last layers of the pretrained CNN. Photos X and X _n are fed to the CNN input, and the outputs of the penultimate layer are used as D-dimensional vectors vector representations X = [x ₁ , ..., x _D ] and X _n = [x _{n; 1} ,…, x _{n; D} ], respectively. Such deep learning-based vector representation extractors train a generic C _emb classifier such as k-nearest neighbor, random forest (RF), support vector machine (SVM), or gradient boosting. The C-dimensional vector p _emb = C _emb (x) of the confidence level estimates is predicted taking into account the input photo in both cases of the retrained network with the last Softmax layer as the C _emb classifier and extracting vector representations using a general classifier. See fig. 4. The final decision can be made in favor of the class with maximum confidence.

The present invention uses a different approach to event recognition based on generative models and image captioning. The proposed approach for recognizing events based on composing an image caption is shown in FIG. 4. It should be noted that all the blocks described can be implemented in software and / or hardware.

The CNN Extraction block does the following: extracts visual vector representations (vector representations) by feeding an RGB representation of a photo from a gallery to the input of a pretrained convolutional neural network to compute the outputs of one of its last layers (usually the penultimate layer).

In the Signature Generation block, the following is done: the extracted visual vector representations and the sequence of previously generated words are fed to an RNN (recurrent neural network) to predict the next word in the textual description of the photo. This sequence of previously generated words initially contains only one special word <START>. As long as the predicted word is not equal to the special word <END>, it is added to this sequence of previously generated words and the extracted visual vector representations, and this sequence is fed to the input of the same RNN.

In the "Dictionary" block, the following is done: since the aforementioned RNN operates with words represented by numbers, it is necessary to map the numbers generated by the RNN into real words from the dictionary.

In the block "Signature preprocessing", the following is performed: a subset of the dictionary is selected by selecting the most frequent words in the training set with the optional exclusion of stop words. Further, each photo is represented as a sparse vector using unitary coding: the v-th component of the vector is 1 only if at least one word in the generated signature is equal to the v-th word from the dictionary.

The "Training sample" block contains a set of training photos with a known type of event.

In the "Classification of vector representations" block, the following is done: estimates of the degree of confidence for all types of events for a photograph are calculated by feeding its vector representations to any classifier. This classifier must be pre-trained to recognize events in the training set, where each photo from the training set is represented by its own visual vector representations, extracted using the procedure from the Extraction of vector representations (CNN) block.

In the "Text classification" block, the following is performed: estimates of the degree of confidence for all types of events for a photograph are calculated by supplying a sparse vector of a unitary encoded text description to any classifier. This classifier must be pretrained to recognize events in the training set, where each photograph from the training set is represented by its own visual vector representations, extracted using the procedure from the Signature Preprocessing block.

In the "Ensemble" block: the outputs of the classifiers for vector representations and texts are combined in an ensemble based on a simple voting with soft aggregation. In particular, the aggregated confidence levels are calculated as the weighted sum of the confidence scores calculated in the Vector Representation Classification and Text Classification boxes. The decision is made in favor of the class with the highest aggregate degree of confidence.

First, the usual extraction of vector representations of x is performed using a pretrained CNN. These visual vector representations and vocabulary V are then fed into a special neural network (generator) based on the RNN, which creates a signature describing each input photo. The signature is represented as a sequence L> 0 of words from the dictionary (t _l ∈V, l∈ {0,…, L}). It is generated sequentially, word by word, starting with the special word t ₀ = <START>, until the special word t _L = <END> [6] is created. See fig. 4.

путем выбора наиболее часто встречающихся слов в обучающих данных {t_n} с необязательным исключением стоп-слов. Затем каждую входную фотографию представляют как |

|- мерный разреженный вектор

, где |

| - размер сокращенного словаря

, а v-й компонент вектора

равен 1, только если хотя бы одно из L слов в подписи t равно v-му слову из словаря

. Это будет означать, например, превращение последовательности {1,5,10,2} в

-мерный разреженный вектор, состоящий из всех нулей, за исключением индексов 1, 2, 5 и 10, значения которых будут единицами [7]. Такая же процедура используется для описания каждой n-й обучающей фотографии с

-мерным разреженным вектором

. После этого можно использовать произвольный классификатор C_txt таких текстовых представлений, подходящий для разреженных данных, чтобы предсказать С оценок степени уверенности P_txt=C_txt(

). В [7] было продемонстрировано, что такой подход даже более точен, чем традиционные классификаторы на основе RNN (включающие в себя один слой LSTM) для набора данных IMDB.The generated signature t is fed to the event classifier. To train its parameters, each n-th photograph from the training sample is fed to the same image captioning network to create a signature t _n = {t _{n; 0} , t _{n ;; 1} ..., t _n; L _n +1}. Since the number of words Ln is not the same for all photographs, it is necessary either to train a sequential classifier based on RNN, or to convert all signatures into vectors of vector representation with the same dimension. Since the number of training instances N is not very large, it was determined experimentally that the latter approach is as accurate as the first, with significantly less training time. Therefore, it was decided to use unitary encoding of sequences t and {t _n } into vectors 0 and 1, as described in [7]. In particular, a subset of the dictionary is formed

by selecting the most frequent words in the training data {t _n } with the optional exclusion of stop words. Then each input photo is presented as |

| - dimensional sparse vector

where |

| - the size of the abbreviated dictionary

, and the vth component of the vector

is equal to 1 only if at least one of the L words in the signature t is equal to the vth word from the dictionary

... This will mean, for example, the transformation of the sequence {1,5,10,2} into

-dimensional sparse vector, consisting of all zeros, except for indices 1, 2, 5 and 10, whose values will be ones [7]. The same procedure is used to describe each nth training photo with

-dimensional sparse vector

... Thereafter, an arbitrary classifier C _{txt of} such text representations, suitable for sparse data, can be used to predict the C confidence scores P _txt = C _txt (

). It was demonstrated in [7] that this approach is even more accurate than traditional RNN classifiers (including one LSTM layer) for the IMDB dataset.

In general, the classification of short text descriptions is not expected to be more accurate than traditional image recognition methods. However, there is confidence that the presence of captions to photographs in the ensemble of classifiers can significantly improve its diversification. In addition, since the signatures are generated based on the extracted vector x of vector representations, only one CNN output is required if the combined common common vector representation classifier from separate classifiers is combined in a simple soft aggregation vote.

In particular, the aggregated confidences are calculated as the weighted sum of the outputs of the individual classifier:

The decision is made in favor of the class with maximum confidence:

The weight w∈ [0,1] in (4) can be chosen using a special control subset to achieve maximum accuracy of criterion (5).

Here are some qualitative examples for using the described approach (Fig. 4). FIG. 5 shows the results of (correct) event recognition using an ensemble. The first line of the title contains the generated photo caption. In addition, the header displays the results of event recognition using t captions (second line), vector representations of x _emb (third line), and the entire ensemble (last line). As you can see, a unified classification of signatures is not always correct. However, this ensemble is capable of delivering a reliable solution, even if individual classifiers make the wrong decisions.

EXPERIMENTAL STUDY

Recognition of events in the photo gallery

Only a limited number of data sets are available for recognizing events in personal photo collections [1]. Therefore, this area deals with two main datasets, namely:

1. PEC [5] containing 61,364 photographs from 807 collections of 14 classes of social events (birthday, wedding, graduation, etc.). The division made by its authors was used: a training sample containing 667 albums (50,279 photographs) and a test sample containing 140 albums (11,085 photographs).

2. ML-CUFED [33] containing 23 common types of events. Each album is associated with several events, i.e. the problem of multiclass classification is being solved. The usual division into a training set (75,377 photos, 1507 albums) and a test set (376 albums with 19,420 photographs) was used.

Vector representations were retrieved using scene recognition models (Inception v3 and MobileNet v2 with a = 1 and a = 1.4) pretrained on the Places2 dataset [38]. To obtain the final descriptor of a set of photographs, two methods were used, namely: (1) simple averaging of vector representations of individual photographs in the set (AvgPool), and (2) implementation of the neural mechanism of attention (2) - (3) for L ₂ -normalized vector representations ... In the first case, the linear SVM classifier from the scikit-learn library was used as C, since it has a higher accuracy than RF, k-NN, and RBF SVM. In the second case, the weights of the attention-based network (Fig. 2) were trained using sets of S = 10 randomly selected photographs from all albums to create an identical shape of the input tensors. As a result, 667 training subsets and 1507 subsets with S = 10 photos were obtained for PEC and ML-CUFED, respectively. Since ML-CUFED contains multiple labels per album, sigmoid activations and binary cross-entropy loss are used. For PEC, the usual Softmax activations and categorical cross-entropy apply. This model was trained using the ADAM optimizer (learning rate 0.001) for 10 early-stop epochs in a Keras 2.3 environment with a TensorFlow 1.15 backend.

Table 1: Accuracy (%) of event recognition in a set of images (album)

CNN Aggregation PEC ML-CUFED MobileNet2, b = 1.0 AvgPool
Attention 86.42
89.29 81.38
84.04 MobileNet2, b = 1.4 AvgPool
Attention 87.14
87.36 81.91
84.31 Inception v3 AvgPool
Attention 86.43
87.86 82.45
84.84 Ales net CNN-LSTM-Iterative [33]
Aggregation of Representative Features | 34] 84.5
87.9 79.3
84.5 ResNet-101 CNN-LSTM-Iterative [33]
Aggregation of Representative Features | 34] 84.5
89.1 71.7
83.4

Table 1 shows the indicators of the recognition accuracy of the pretrained CNN. In this case, the multi-class precision is calculated for ML-CUFED, so it is assumed that the predicted event is correct if it matches any label associated with the album. The table shows the most famous results for the indicated data sets [33, 34].

Moreover, in all cases, attention-based aggregation is 1-3% more accurate than the classification of mean vector representations. It can be seen that the proposed implementation of the attention mechanism achieves the highest results to date, although much faster convolutional networks are used (MobileNet and Inception, not AlexNet and ResNet-101) and the sequential nature of photos into an album on the attention-based network is not taken into account ( Fig. 2). The most notable fact in this case is that the best results for PEC are achieved for the simplest model (MobileNet v2, a = 1.0), which can be explained by the lack of training data for this particular dataset.

As noted above, there is no information about albums in the gallery at all. Therefore, the event must be assigned to all photos individually. In the following experiment, each photo contained in both datasets is directly assigned the first collection-level label and simply uses the photo itself to recognize events without any meta-information. In addition to the basic approach (subsection 3.1), hierarchical agglomeration clustering of the entire test gallery is used. Only the results achieved by the mean connected clustering of the vector representations x _t extracted by the pretrained CNN and the confidence score pt are shown. In the first case, both the Euclidean distance (L2) and the chi-square distance (X2) are used. Since the confidence scores returned by the decision function for LinearSVC are not always non-negative, only Euclidean distance is used for them. The results are shown in Table 2.

Table 2: Accuracy (%) of event recognition in one image.

Data set CNN Basic approach Vector views Estimates L ₂ X ² L ₂ PEC MobiLeNet2, b = 1.0
MobileNet2, b = 1.4
Inception v3 58.32
60.34
61.82 60.42
61.25
64.19 60.69
61.92
64.22 58.44
60.58
61.97 ML-CUFED MobileNet2, b = 1.0
MobiLeNet2, b = 1.4
Inception v3 54.41
53.54
57.26 57.03
54.97
59.19 57.45
55.98
60.12 54.56
54.03
57.87

At the same time, firstly, the accuracy of event recognition in individual photographs is 25-30% lower than the accuracy of classification by albums (Table 1). Second, the clustering of confidence estimates at the output of the best classifier does not significantly affect the overall accuracy. Third, hierarchical clustering with chi-squared distance leads to slightly more accurate results than the usual Euclidean metric. And finally, preliminary clustering of vector representations reduces the error rate of the basic approach by only 1.2-2%, even if the similarity threshold was carefully selected during clustering.

Let's demonstrate how the assumption of sequentially ordered photos in an album can improve the accuracy of event recognition. To complicate the task, the following transformation of the order of test photos was carried out 10 times. The sequence of albums was randomly shuffled, and the photos in each album were also shuffled. In addition to comparing confidence scores from the linearSVC decisive function, L2-normalization was performed. In addition, CNN was retrained using the expanded training set X as follows. First, the weights in the base part of the CNN were frozen and a new head element (fully connected layer with C outputs and Soft-max activation) was studied for 10 epochs. The weights were then studied across the entire CNN over 3 epochs with a 10-fold decrease in learning rate.

Tables 3 and 4 present the results (mean accuracy ± its standard deviation) of the proposed algorithms 1, 2 for PEC and ML-CUFED, respectively. At the same time, the attention mechanism in most cases reduces the error rate by 8%. Notably, matching the similarity between L2-normalized certainties significantly improves the overall accuracy of the attention model for PEC (Table 3), although the experiments presented did not show any improvement in traditional clustering compared to the previous experiment (Table 2). Obviously, overtrained CNNs lead to the most accurate solution, but the difference (0.1-1.6%) with the best results from pretrained models is quite small. However, the latter do not require additional output in existing models of scene recognition, so the implementation of event recognition in the album will be very fast if the scenes need to be further classified, for example, for more detailed modeling by the user [26]. Surprisingly, the calculation of the similarity between the estimates of the degrees of confidence of the classifiers (c (P _t , P _t-1 )) reduces the probability of error in traditional matching of vector representations (c (X _t , X _t-1 )) by 2-7%. It should be recalled that the usual clustering of vector representations was 1-2% more accurate when compared with the classifier's estimates (Table 2). Apparently, in this particular case, the threshold with ₀ can be estimated (Algorithm 2) more reliably when most of the photographs of the same event are compared in the prediction procedure (Algorithm 1). And finally, the most important conclusion is that the proposed approach has 9-20% higher accuracy compared to the basic approach. Moreover, this algorithm is 13-16% more accurate than the classification of groups of photographs obtained using hierarchical clustering (Table 2).

Table 3: Precision (%) of the proposed approach, PEC.

CNN Aggregation A basic level of Vector views Estimates Estimates ( L ₂ -normalized) L ₂ x ² L ₂ x ² L ₂ MobileNet2, b = 1.0 (pretrained), vector representations AvgPool
Attention 58.32
54.43 66.85 ± 0.59
68.51 ± 0.41 68.52 ± 0.89
70.65 ± 1.20 71.08 ± 0.59
74.49 ± 0.70 - 72.68 ± 0.56
80.48 ± 1.01 MobileNeC. b = 1.4 (pretrained), vector representations AvgPool
Attention 60.34
55.36 68.85 ± 0.50
70.53 ± 0.79 63.57 ± 0.57
71.16 ± 0.72 72.50 ± 1.49
78.20 ± 1.47 - 73.49 ± 0.86
81.27 ± 0.81 MobileNet2, b = 1.4 (retrained), estimates AvgPool
Attention 61.89
61.55 -
- -
- 75.66 ± 0.55
78.77 ± 0.49 76.96 ± 0.97
81.33 ± 0.69 -
- Inception v3 (pretrained), vector views AvgPool
Attention 61.82
56.94 72.29 ± 1.28
72.38 ± 1.13 72.32 ± 1.54
71.36 ± 0.67 74.54 ± 1.04
76.76 ± 0.70 -
- 76.48 ± 0.47
80.17 ± 1.14 Inception v3
(retrained), grades AvgPool
Attention 63.56
62.91 -
- -
- 78.87 ± 0.67
81.03 ± 0.77 73.92 ± 0.65
81.95 ± 1.11 -
-

Table 4: Accuracy (%) of the proposed approach, ML-CUFED.

CNN Aggregation Base
an approach Vector views Estimates Estimates (L ₂ -normalized) L ₂ x ² L ₂ x ² L ₂ MobileNet2, b = 1.0 (pretrained), vector representations AvgPool
Attention 54.41
51.05 67.54 ± 0.76
68.71 ± 0.71 67.42 ± 0.93
68.55 ± 0.61 69.83 ± 0.74
71.44 ± 0.82 -
- 70.42 ± 0.41
71.61 ± 0.69 MobileNet2, b = 1.4 (pretrained), vector representations AvgPool
Attention 53.54
51.12 66.93 ± 0.60
68.34 ± 0.68 67.21 ± 0.55
68.62 ± 0.50 68.56 ± 0.73
70.79 ± 0.75 -
- 69.47 ± 0.36
71.78 ± 0.74 MobileNet2, b = 1.4 (retrained), estimates AvgPool
Attention 56.01
56.09 -
- -
- 70.57 ± 0.48
72.90 ± 0.59 71.61 ± 0.28
73.46 ± 0.58 -
- Inception v3
(pretrained), vector representations AvgPool
Attention 57.26
50.89 69.91 ± 0.58
69.30 ± 0.47 70.01 ± 0.62
68.52 ± 0.89 72.25 ± 0.61
72.73 ± 0.72 -
- 72.78 ± 0.71
73.00 ± 0.65 Inception v3
(retrained), grades AvgPool
Attention 57.12
57.29 -
- -
- 72.18 ± 0.63
73.06 ± 0.74 73.20 ± 0.74
73.92 ± 0.81 -
-

Recognition of events in individual photos

In addition to PEC and ML-CUFED, the WIDER (Web Image Dataset for Event Recognition) [35] database of web images for event recognition was investigated, containing 50,574 photographs and С = 61 categories of events (parade, dances, meetings, press conferences, etc.) etc.). The standard training / testing division was used for all datasets suggested by their developers. In PEC and ML-CUFED, a collection level tag was assigned directly to each photograph contained in a given collection. Any metadata, such as time information, was completely ignored, with the exception of the photograph itself, just like in article [32].

To focus on the possibility of implementing autonomous event recognition on mobile devices [26] in order to compare the proposed approach with traditional classifiers, CNN MobileNet v2 with a = 1 [23] and Inception v4 [28] were used. They were first trained on the Places2 dataset [38] to extract vector representations. The linear SVM classifier from scikit-learn library was used, as it has higher accuracy than other classifiers from this library (RF, k-NN and RBF SVM). In addition, these CNNs were retrained using the provided training sample as follows. First, we fixed the weights in the base CNN part and trained a new classifier (fully connected layer with C outputs and Softmax activation) using the ADAM optimizer (learning rate 0.001) for epochs with early stopping in Keras 2.2 environment with TensorFlow 1.15 backend. Weights were then trained throughout CNN for 5 epochs using ADAM. Finally, CNN was trained using SGD for 3 epochs with a 10-fold decrease in the learning rate parameter.

In addition, vector representations were used from object detection models, which are typical for event recognition [35, 26]. Since many photographs of the same event sometimes contain identical objects (such as a soccer ball), they can be detected using modern CNN-based methods such as SS-DLite [23] or Faster R-CNN [21]. These methods determine the position of several objects in the input photograph and predict the estimates of each class from a given set of K> 1 types. For each type of object a sparse K-dimensional vector of estimates is extracted. If there are several objects of the same type, the maximum estimate is stored in this vector vector representations [20]. This vector representation vector is either classified by a linear SVM or used to train a feedforward neural network with two hidden layers containing 32 neurons each. Both classifiers were trained using a training sample from each event dataset. This study examines SSDs with MobileNet architecture and Faster R-CNN with InceptionResNet architectures. Models pretrained on the Open Photos v4 dataset (K = 601 objects) were taken from the TensorFlow Object Detection Model Zoo.

|=5000 наиболее часто встречающихся слова, за исключением специальных слов <START> и <END>. Их классифицировали либо линейным SVM, либо нейронной сетью с прямым распространением с той же архитектурой, что и в случае обнаружения объекта. Эти классификаторы также обучались с нуля на каждом предоставленной обучающей выборке событий. Этот же набор использовался для оценки веса w в ансамбле (уравнение 1).A preliminary pilot study with pretrained image caption composing models discussed in Section 2 demonstrated that the best quality for the MS COCO signature compilation dataset is achieved by the AR-Net model [6]. Therefore, the ARNet encoder / decoder model was used in this experiment. However, it can be replaced by any other method of captioning an image without changing the event recognition algorithm. ARNet was trained on the Conceptual Captions Dataset, which contains more than 3.3 million pairs of photo URLs and captions in the training set and about 15 thousand pairs in the test set. The extraction of vector representations in the encoder is implemented not only by the same CNN (Inception and MobileNet v2). Was checked out |

| = 5000 most common words, excluding the special words <START> and <END>. They were classified as either a linear SVM or a feedforward neural network with the same architecture as in the case of object detection. These classifiers were also trained from scratch on each training set of events provided. The same set was used to estimate the weight w in the ensemble (Equation 1).

Tables 5, 6, 7 present the results of lightweight mobile (MobileNet and SSD object detector) and deep models (Inception and Faster R-CNN) for PEC, WIDER and ML-CUFED, respectively. They add the most famous results for 24 years for the same experimental protocols.

Table 5: Accuracy of Event Recognition (%), PEC

Classifier Signs Lightweight models Deep models SVM Vector views
Objects
Texts
Proposed ensemble (4), (5) 59.72
42.18
43.77
60.56 61.82
47.83
47.24
62.87 Retrained CNN Vector views
Objects
Texts
Proposed ensemble (4), (5) 62.33
40.17
43.52
63.38 63.56
47.42
46.84
65.12 Aggregated SVM [5]
Multiple Sub-Events [5]
SHMM [5]
Initialization Based Learning Transfer [32]
Transfer of training data and knowledge [321 41.4
51.4
55.7
60.6
62.2

Table 6: Accuracy of event recognition (%), WIDER

Classifier Signs Lightweight models Deep models SVM Vector views
Objects
Texts
Proposed ensemble (4), (5) 48.31
19.91
26.38
48.91 50.48
28.66
31.89
51.59 Retrained
CNN Vector views
Objects
Texts
Proposed ensemble (4), (5) 49.11
12.91
23.93
49.80 50.97
21.27
30.91
51.84 Basic CNN [35]
Deep Channel Synthesis [35]
Initialization Based Learning Transfer [32]
Transfer of training data and knowledge [32] 39.7
42.4
50.8
53.0

Of course, the proposed recognition of photo captions is not as accurate as with conventional CNN-based vector representations. However, the classification of text descriptions is much better than a random guess with an accuracy of 100% / 14≈7.14%, 100% / 6l≈1.64%, and 100% / 23≈4.35% for PEC, WIDER and ML-CUFED. respectively. It is important to emphasize that in most cases this approach yields a lower error rate than the classification of vector representations based on object detection. This improvement is especially noticeable on lightweight SSD models, which are 1.5-13% less accurate than the proposed photo caption classification due to the limitations of SSD-based models when detecting small objects (food, pets, fashion accessories, etc.) etc.). Vector representations detected by Faster R-CNN can be classified more accurately, but Faster R-CNN with InceptionResNet architecture gives several times slower output than decoding in ARNet (6-10 seconds versus 0.5-2 seconds in MacBook Pro 2015 ).

Table 7. Accuracy of event recognition (%), ML-CUFED

Classifier Signs Lightweight models Deep models SVM Vector views
Objects
Texts
Proposed ensemble (4), (5) 53.54
34.21
37.24
55.26 57.27
40.94
41.52
58.86 Retrained CNN Vector views
Objects
Texts
Proposed ensemble (4), (5) 56.01
32.05
36.74
57.94 57.12
40.12
41.35
60.01

Finally, the most suitable way to use image captioning in classifying events is to merge them with regular CNNs. In this case, the previous known best accuracy for PEC increases from 62.2% [32] even for lightweight models (63.38%), if additional trained CNNs are used in the ensemble. The presented model based on Inception is even better (65.12% accuracy). The highest accuracy rate to date of 53% [32] for the WIDER dataset is still not achieved, although the best accuracy (51.84%) is 9% higher than the best results (42.4%) from the original paper [35] ... The presented experimental protocol for the ML-CUFED dataset was studied for the first time in this work, as this dataset is designed primarily for the recognition of album-based events.

In practice, it is preferable to use a pretrained CNN as a vector representation extractor in order to eliminate an additional pass through the retrained CNN when it diverges from the encoder in the image signature model. Unfortunately, SVM accuracy for vector representations of the pretrained CNN is 1.5-3% lower compared to the pretrained models for PEC and ML-CUFED. In this case, an additional conclusion can be assumed. However, the difference in error rates between pretrained and pretrained models for the WIDER dataset is not significant, so pretrained CNNs are definitely worth using in this case.

The above described embodiments of the invention are examples and should not be construed as limiting. In addition, the description of the exemplary embodiments is intended to be illustrative, and not to limit the scope of the claims, and many alternatives, modifications, and variations will be apparent to those skilled in the art.

LITERATURE

[1] Kashif Ahmad and Nicola Conci, 'How deep features have improved event recognition in multimedia: A survey', ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 15 (2), 39 (2019).

[2] Kashif Ahmad, Nicola Conci, Giulia Boato, and Francesco GB De Natale, 'Event recognition in personal photo collections via multiple instance learning-based classification of multiple images', Journal of Electronic Imaging, 26 (6), 060502, ( 2017).

[3] Wesam M Ashour, Riham Z Muqat, Alaaeddin B AlQazzaz, and Saeb R AbdElnabi, 'Improve basic sequential algorithm scheme using ant colony algorithm', in Proceedings of the 7th Palestinian International Conference on Electrical and Computer Engineering (PICECE), pp ... 1-6. IEEE, (2019).

[4] Siham Bacha, Mohand Said Allili, and Nadjia Benblidia, 'Event recognition in photo albums using probabilistic graphical models and feature relevance', Journal of Visual Communication and Image Representation, 40, 546-558, (2016).

[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool, 'Event recognition in photo collections with a stopwatch HMM', in Proceedings of the International Conference on Computer Vision (ICCV), pp. 1193-1200. IEEE, (2013).

[6] Xinpeng Chen, Lin Ma, Wenhao Jiang, Jian Yao, and Wei Liu, 'Regularizing rnns for caption generation by reconstructing the past with the present', in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2018).

[7] Francois Chollet, Deep learning with Python, Manning Publications Company, 2017.

[8] Minh-Son Dao, Due-Tien Dang-Nguyen, and Francesco GB De Natale, 'Signature-image-based event analysis for personal photo albums', in Proceedings of the 19th International Conference on Multimedia (ACM MM), pp. ... 1481-1484. ACM, (2011).

[9] Sergio Escalera, Junior Fabian, Pablo Pardo, Xavier Baro, Jordi Gonzalez, Hugo J Escalante, Dusan Misevic, Ulrich Steiner, and Isabelle Guyon, 'Chalearn looking at people 2015: Apparent age and cultural event recognition datasets and results', in Proceedings of the International Conference on Computer Vision Workshops (ICCVW), pp. 1-9, (2015).

[10] Ming Geng, Yukun Li, and Fenglian Liu, 'Classifying personal photo collections: an event-based approach', in Proceedings of the Asia- Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data, pp. 201-215. Springer, (2018).

[11] Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep learning, MIT Press, 2016.

[12] Ivan Grechikhin and Andrey V Savchenko, 'User modeling on mobile device based on facial clustering and object detection in photos and videos', in Proceedings of the Iberian Conference on Pattern Recognition and Image Analysis, pp. 429-440. Springer, (2019).

[13] Cong Guo, Xinmei Tian, and Tao Mei, 'Multigranular event recognition of personal photo albums', IEEE Transactions on Multimedia, 20 (7), 1837-1847, (2017).

[14] MD Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga, 'A comprehensive survey of deep learning for image captioning', ACM Computing Surveys (CSUR), 51 (6), 118, (2019).

[15] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam, 'MobileNets: Efficient convolutional neural networks for mobile vision applications', arXivpreprint arXiv: 1704.04861, (2017) ...

[16] Dmitry Kuzovkin, Tania Pouli, Olivier Le Meur, Remi Cozot, Jonathan Kervec, and Kadi Bouatouch, 'Context in photo albums: Understanding and modeling user behavior in clustering and selection', ACM Transactions on Applied Perception (TAP), 16 (2), 11, (2019).

[17] Stefan Lonn, Petia Radeva, and Mariella Dimiccoli, 'Smartphone picture organization: A hierarchical approach', Computer Vision and Image Understanding, 187, 102789, (2019).

[18] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh, 'Neural baby talk', in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2018).

[19] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L. Yuille, 'Deep captioning with multimodal recurrent neural networks (m-RNN)', in Proceedings of the International Conference on Learning Representations (ICLR), ( 2015).

[20] Alexandr Rassadin and Andrey Savchenko, 'Scene recognition in user preference prediction based on classification of deep embeddings and object detection', in Proceedings of the International Symposium on Neural Networks (ISNN), volume 11555, pp. 422-430. Springer, (2019).

[21] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, 'Faster R-CNN: Towards real-time object detection with region proposal networks', in Advances in Neural Information Processing Systems (NIPS), pp. 91-99, (2015).

[22] Mark Sandier, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen, 'MobilenetV2: Inverted residuals and linear bottlenecks', in Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, (2018).

[23] Mark Sandier, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen, 'Mobilenetv2: Inverted residuals and linear bottlenecks', in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4510-4520, (2018).

[24] Andrey V. Savchenko, 'Efficient facial representations for age, gender and identity recognition in organizing photo albums using multi-output ConvNet', PeerJ Computer Science, 5 (el97), (2019).

[25] Andrey V. Savchenko, 'Sequential three-way decisions in multi-category image recognition with deep features based on distance factor', Information Sciences, 489, 18-36, (2019).

[26] Andrey V Savchenko, Kirill V Demochkin, and Ivan S Grechikhin, 'User preference prediction in visual data on mobile devices', arXiv preprint arXiv: 1907.04519, (2019).

[27] Anastasiia D Sokolova, Angelina S Kharchevnikova, and Andrey V Savchenko, 'Organizing multimedia data in video surveillance systems based on face verification with convolutional neural networks', in Proceedings of the International Conference on Analysis of Images, Social Networks and Texts ( AIST), pp. 223-230. Springer, (2017).

[28] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex A. Alemi, 'Inception-v4, Inception-ResNet and the impact of residual connections on learning', in Proceedings of the International Conference on Learning Representations (ICLR) Workshop, (2016).

[29] Shen-Fu Tsai, Thomas S Huang, and Feng Tang, 'Album-based object-centric event recognition', in Proceedings of the International Conference on Multimedia and Expo, pp. 1-6. IEEE, (2011).

[30] Nivetha Vijayaraju, 'Image retrieval using image captioning', Master's Projects, 687, (2019).

[31] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, 'Show and tell: Lessons learned from the 20 mscoco image captioning challenge', IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 (4), 652-663, (2017).

[32] Limin Wang, Zhe Wang, Yu Qiao, and Luc Van Gool, 'Transferring deep object and scene representations for event recognition in still images', International Journal of Computer Vision, 126 (2-4), 390-409, ( 2018).

[33] Yufei Wang, Zhe Lin, Xiaohui Shen, Radomir Mech, Gavin Miller, and Garrison W Cottrell, 'Recognizing and curating photo albums via event-specific image importance', in Proceedings of British Conference on Machine Vision (BMVC), ( 2017).

[34] Zifeng Wu, Yongzhen Huang, and Liang Wang, 'Learning representative deep features for image set analysis', IEEE Transactions on Multimedia, 17 (11), 1960-1968, (2015).

[35] Yuanjun Xiong, Kai Zhu, Dahua Lin, and Xiaoou Tang, 'Recognize complex events from static images by fusing deep channels', in Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1600-1609, (2015).

[36] Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio, 'Show, attend and tell: Neural image caption generation with visual Attention', in Proceedings of the

International Conference on International Conference on Machine Learning (ICML), pp. 2048-2057, (2015) .30

[37] J. Yang, P. Ren, D. Zhang, D. Chen, F. Wen, H. Li, and G. Hua, 'Neural aggregation network for video face recognition', in Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5216-5225. IEEE, (2017).

[38] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba, 'Places: A million image database for scene recognition', IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 (6), 1452-1464, ( 2018).

Claims (25)

1. A method of recognizing events in photographs with automatic selection of albums, which consists in the following: automatically assign the boundaries of albums, evaluating the similarity between the degrees of confidence for each pair of consecutive photos in the gallery, while the similarity between the photos is calculated as the distance between the degrees of confidence for each type of event, assessed by the classification of each photo, or the similarity between photos is calculated as the sum of similarities between degrees of confidence for each type of event, assessed based on the classification of each photograph and the similarity between their locations, if the EXIF (Exchangeable Photo File Format) data of those photographs contains location information; group consecutive photos from the gallery into albums; calculating the descriptor of each album using the attention mechanism of the neural model of attention, which predicts one class of events for a set of photos, while the predicted class is assigned to all photos between the mentioned boundaries; recognize the event type tag of each album by supplying its descriptor to the input of the classifier; recognize the event in the photos in the gallery by assigning an appropriate event type tag to all the photos in each album. 2. The method according to claim 1, wherein it is assessed that if the similarity does not exceed a predetermined threshold, then both photographs are included in the same album. 3. The method of claim 2, wherein the predetermined threshold is calculated automatically during the pre-training procedure using the provided training album set. 4. A method for recognizing an event in a photograph with its presentation in a text area, which consists in the following: calculating vector representations of the photograph by feeding its RGB (red-green-blue) representations to the input of the convolutional neural network; automatically compose image captions to generate textual descriptions of a photo based on its vector representations, while a textual description of a photo is generated as follows: the extracted vector representations and a sequence of previously generated words are fed to a recurrent neural network (RNN) to predict the next word in the textual description of the photo; selecting a subset of the vocabulary by selecting the most common words in the training data with optional exclusion of stop words; encode the generated signature; feed the generated text descriptions to the input of the event classifier; recognize the event by assigning the event classifier output to the tag of the corresponding photo. 5. The method according to claim 4, wherein the generated signature is encoded as a sparse vector using unitary encoding of the textual description of the photograph: the vth component of the vector is 1 only if the generated signature contains at least one vth word from the dictionary. 6. The method according to claim 5, further comprising: combining outputs of the classifier for the generated text descriptions and the traditional vector representation classifier into an ensemble to improve the accuracy of event recognition. 7. The method according to claim 6, wherein the classifier outputs are obtained as follows: calculating confidence scores for all types of events for the photograph by feeding its vector representations to an arbitrary classifier; calculating confidence scores for all types of events for the photograph by supplying a sparse vector of the unitary encoded text description to an arbitrary classifier; combine outputs of classifiers for vector representations and texts in an ensemble based on simple voting with soft aggregation. 8. The method of claim 4, further comprising adding the predicted word to the sequence of previously generated words and feeding the extracted visual vector representations and the sequence to the same RNN. 9. The method of claim 4, wherein the training sample is a collection of photographs with a known type of event. 10. The method of claim 6, wherein the conventional classifier should be pre-trained to recognize events in the training sample, in which each photograph is presented along with its extracted visual vector representations. 11. A method according to claim 7, in which, in particular, the aggregated degrees of confidence are calculated as a weighted sum of the ratings of the degrees of confidence and the decision is made in favor of the class with the maximum aggregated degree of confidence.

Images