BR102019024569A2 - video recognition method capable of encoding temporal and spatial relationships of concepts using contextual information - Google Patents

Abstract

The proposed invention aims to encode contextual information for video analysis and comprehension, encoding spatial and temporal relationships of objects and the main agent in a scene. The main target application of the invention is the recognition of human activity. Encoding these spatial and temporal relationships can be crucial to distinguishing different categories of human activities and can be important in helping to discriminate different categories of video for video classification, retrieval, categorization, and other video analytics applications.

Description

VIDEO RECOGNITION METHOD CAPABLE OF CODING TEMPORAL AND SPATIAL RELATIONS OF CONCEPTS USING CONTEXTUAL INFORMATION technical field

[0001] The present invention refers to a method of recognition of human activities that considers occurrences of objects and their temporal and spatial relationships. Initially, the invention could be incorporated into cameras in order to allow understanding of video data in terms of human activities. Furthermore, vision-based monitoring systems can make use of the invention for video analysis and comprehension. A wide range of applications can benefit from the invention, including, for example, recognizing activities in sports videos, monitoring suspicious activity in video surveillance systems (eg, in airports, bus stations, subway stations and others public places), health monitoring using cameras to detect falls and other events (eg elderly care), video-based traffic monitoring systems (eg to detect traffic accidents), video library locator/organizer personal (eg finding relevant activities in personal videos), etc.

[0002] The proposed invention can effectively make use of context information encoding the spatial and temporal relationship of the elements that appear in the video scene. For example, the proposed invention can be used to recognize what activities are being performed in a video, for example on YouTube, and help with retrieval and recommendation. Activity recognition can also be used to trigger recording on surveillance cameras such as Google Clips or other personal and professional cameras.

Background of the invention

[0003] Video monitoring systems are becoming more important given the high availability of cameras in many different places and scenarios. Understanding this huge amount of video information becomes extremely important to make video data useful, to eliminate or reduce the cost of manual video analytics, and to improve the decision-making process based on video information, just to name a few benefits.

[0004] Given the recent popularity of mobile devices equipped with high-resolution cameras and the dissemination of videos and images on the Internet, understanding visual data is also crucial for extracting knowledge from human-generated videos and images.

[0005] Currently, artificial intelligence (AI) plays an important role in all organizations, enabling a new set of applications and functionalities that impact the lives of millions of people. Video comprehension, especially recognition of human activity, is one of the main AI topics for video data.

[0006] Many of the current video comprehension techniques lack the analysis of contextual information, which can contribute or even determine the correct interpretation of video content. Specifically, for the recognition of human activity, context plays an important role. Certain activities are strongly related to their surroundings, which provide important indications for the discrimination of the activity.

[0007] Objects, people, pose, scene, background information and other elements available in a video scene are here considered as context. Not only the presence of such elements, but also their spatial and temporal relationships are very important to determine context information.

[0008] For activity recognition, for example, to differentiate videos of sports played with the ball (for example, football, volleyball, basketball, etc.), the spatial and temporal relationships of the ball with the actors are very relevant. The spatial arrangement of the ball with players can help differentiate between sports played with the hand (volleyball and basketball) versus sports played with the feet (football), while temporal cues will help differentiate between volleyball and basketball (player never holds a ball for a long time in a volleyball match).

[0009] The encoding of such contextual information by the spatial and temporal relationships of the objects available in the video is the main aspect of the proposed invention. Therefore, the proposed invention can benefit a wide range of applications related to video analysis and comprehension.

[0010] Patent document WO2014146463 A1, entitled: "BEHAVIOUR RECOGNITION METHOD BASED ON HIDDEN STRUCTURE REASONING", by the INSTITUTE OF AUTOMATION, CHINESE ACADEMY OF SCIENCES, published on September 25, 2014, discloses how to model parts of the body of a human skeleton estimated using a directed graph, which uses descriptors such as HOG3D, HOG, and HOF for classification using SVM, which is not an end-to-end learning architecture. The present invention, on the other hand, can use higher-level or lower-level semantic concepts and directly explores the relationships between concepts using an end-to-end differentiable end-to-end differentiable temporal egocentric relational network. The method of patent document WO2014146463 A1 does not encode the spatial and temporal relationships of concepts and agents, as the present invention does.

[0011] Patent document US2017091537, entitled: "ACTIVITY RECOGNITION SYSTEMS AND METHODS", by NANT Holdings IP LLC, published March 30, 2016, explores a wide range of feature descriptors. Such descriptors are modeled using activity graphs, which are assigned a score using graph comparison and similarity techniques. In contrast, the present invention uses a temporal egocentric relational network to learn the spatial and temporal relationships between concepts in a fully differentiable end-to-end framework, which allows learning the relationships between objects that matter to the target activity.

[0012] Patent document CN104091167 A titled: "FEATURE EXTRACTION METHOD BASED ON HUMAN BODY ACTIVITY RECOGNITION OF MOTION SENSING CAMERA", by UNIV ELECTRONIC SCIENCE & TECH, published on October 8, 2014, proposes a method of activity recognition based on Human Body. To that end, the method in patent document CN104091167 A extracts feature descriptors such as light flux histograms, gradient histograms and local depth features. The local depth characteristics are extracted similarly to the Word Spatial Arrangement (WSA) method ("Visual word spatial arrangement for image retrieval and classification", Penatti et al., Pattern Recognition, 2014), using each point of interest within the human body as the center. In contrast to the present invention, the method described in patent document CN104091167 A works on lower level concepts, while the present invention benefits from higher level semantics. In addition, the patent lacks a module to learn semantic relationships between features, like the present invention. Finally, the patent is adapted to tasks that comprise depth images, while the present invention is broader, allowing it to be applied to datasets with/without depth images.

[0013] Document titled: "Object Bank: A HighLevel Image Representation for Scene Classification & Semantic Feature Sparsification", Li et al., NIPS, 2010) uses various object detections and, from these, extracts histograms in a pyramidal way. to encode object granularity and spatial relationships. However, the spatial pyramid used is an evenly spaced grid built in the center of the frame without pointing to any specific object, ie it is not built around an object such as the person performing the action. Thus, the method in the article only captures the global object distribution of the scene, disregarding which objects are more important. In contrast, the present invention is more general, considering various contextual clues, such as spatial and temporal information. Furthermore, the representation of the object of the present invention focuses on capturing the local context in relation to the agent that performs the activity. The present invention attaches more importance to the main objects related to an activity. Furthermore, the method in the article focuses on scene recognition, while the present invention is focused on activity recognition.

[0014] Document titled: "WSA - Visual word spatial arrangement for image retrieval and classification", by Penatti et al., Pattern Recognition, 2014 proposes the WSA method to encode the spatial relationship of visual words for image retrieval and classification. The egocentric pyramids present in this invention can be considered as the WSA method employed to encode the spatial arrangement of object detections. However, the WSA is a low-level descriptor that models the arrangement of points of interest, while the egocentric pyramid is oriented towards higher semantic concepts. Furthermore, the egocentric representation of the present invention includes other contextual information, such as temporality, which is not encoded by the WSA method. Finally, the WSA lacks a module to learn semantic relationships between features like the present invention.

[0015] Document titled: "What do 15,000 object categories tell us about classifying and localizing actions?", Jain et al., CVPR, 2015, explores the use of objects in the task of activity recognition, showing that objects help to recognize actions. To detect objects, the authors trained an AlexNet-type architecture using 15,000 ImageNet object categories. The network was then used to detect the presence of each object, generating a bag-of-words with the probability of each class. The network uses object coding and HOG, HOF, and MBH motion descriptors calculated along improved trajectories — a technique to compensate for camera movement. The approach proposed by Jain et al. it differs from the present invention in the following respects: First, the authors focused only on information about objects. The present invention explores other sources of contextual signals in addition to objects, such as temporal relationships. Second, the method in the article does not consider the spatial and temporal relationships between objects. Finally, the method in the article does not explore how contextual information relates to the main agent performing an action, which is captured in the present invention.

[0016] Paper titled: "Harnessing Object and Scene Semantics for Large-Scale Video Understanding", Wu et al., CVPR, 2016, takes advantage of object and scene semantics for large-scale video comprehension. Instead of summing the average object response for each action, the authors train a discriminative neural network, the Object-Scene Semantic Fusion (OSF) network. OSF is a multilayer perceptron (MLP) that learns to fuse scene, object, and feature vectors. It has three layers, one for each input stream, namely object, scene and feature vectors. Although the approach in the article also uses other sources for contextual signals in addition to objects, it does not explore the egocentric hypothesis and its temporal and spatial relationships of objects like the present invention.

[0017] Document titled: "Spatio-temporal humanobject interactions for action recognition in videos", Escorcia and Niebles, ICCVW, 2013, builds concentric ellipses around an agent performing an action, in addition to other descriptors, such as relative size and occupation. The method in the article has limitations, such as using dataset annotations to provide the agent's position, as well as assuming that there is only a single object in the scene (ie, it does not consider multiple objects like the present invention). The present invention also builds a pyramid, but as a grid, and more importantly, a data-centric approach is employed to learn the semantic relationships between features, which can include objects, the agent, temporal features, and others.

[0018] Document titled: "Two-Stream SR-CNNs for Action Recognition in Videos", Wang et al., BMVC, 2016, explores contextual information based on objects. The input data for this network are the same as for conventional networks two-way (conventional two-stream networks), ie RGB and optical flow modalities. However, the SR-CNN proposes to add two lanes in addition to the traditional ones after the last convolutional layer. The first lane remains the same, having as input the last convolutional layer triggers, capturing the overall appearance of the entire frame. The second and third way process contextual information by receiving bindings from the main agent bounding box and from object bounding box triggers. SR-CNN is different from the present invention in that SR-CNN takes advantage of object detections together with the two-way network in a d framework. and end to end. On the other hand, the present invention seeks to incorporate additional context from other contextual sources and its representations of objects are based on egocentric pyramids and object pairings. In addition, SR-CNN uses bounding box activations, preventing it from capturing the spatial arrangement of objects. SR-CNN also does not capture the spatial relationships of objects in relation to the main agent, as it is only able to explore correlations between object occurrences and the main agent in the fusion layer. Finally, the present invention also employs an egocentric temporal relational network for arbitrary numbers of objects, while SRCNN uses multi-instance learning to select relevant objects.

[0019] Document titled: "Going deeper into first-person activity recognition", Ma et al., CVPR, 2016, proposes an approach that uses object and action labels for the recognition of egocentric activities. In egocentric activities, important objects are usually close to the agent's hands. Thus, the authors argue that hand detection can be useful to accurately recognize the main object and its location. Ma et al. leverage handheld information for object detection by tuning an FCN32-s architecture trained in hand segmentation to generate object coordinate proposals. The object proposals are then fed to another convolutional neural network (CNN), called ObjectNet, to predict the object label. In later stages, the method proposed in the article combines the predictions generated by an action network (ActionNet) and ObjectNet to generate the activity label. The approach in the article is different from the present invention as it focuses on accurately predicting object labels using hand segmentation information. The method in the article then uses the resources extracted from the object bounding box for classification globally, that is, it does not address how the spatial and temporal arrangement of contextual information and its relationships with the principal agent can help in activity recognition, as does the present invention.

[0020] Document titled: "A simple neural network module for relational reasoning", Santoro et al., Corr, 2017, has a similar concept to the present invention, in terms of encoding the relationship between objects. However, the article presents several differences in the present invention. First, it was not originally designed for activity recognition, so it does not encode temporal information as the present invention allows it to do. Second, the input to the present invention considers objects and agent pairings using egocentric pyramids.

[0021] Document titled: "Egocentric activity recognition with multimodal fisher vector", Song et al., ICASSP, 2016, proposes a technique for the fusion of feature descriptors extracted from egocentric videos and sensor data using a Fisher kernel. Unlike the present invention, the approach in the article does not target a principal agent to capture spatial/temporal relationships, nor does it take advantage of a fully differentiable end-to-end framework to extract semantic relationships between a variable number of features/objects, like the present invention does.

[0022] Document titled: "Temporal Relational Reasoning in Videos" by Zhou et al., ECCV, 2018, proposes an architecture for understanding temporal relations based on relational networks. This approach creates a hierarchical relational network pyramid, with varying time scales, to encode temporal information given the characteristics obtained by a CNN. In contrast, the present invention models the spatial and temporal relationships between objects given by a concept detector. Spatial relationships are obtained through the coordinates of concepts and the egocentric pyramid, while temporal information is encoded through the position of the normalized frame and the structure of temporal segment networks.

summary

[0023] The proposed invention aims to encode contextual information for analysis and understanding of video, encoding spatial and temporal relationships of objects and the main agent in a scene. The main target application of the invention is the recognition of human activity. Encoding these spatial and temporal relationships can be crucial to distinguishing different categories of human activities and can be important in helping to discriminate different categories of video for video classification, retrieval, categorization, and other video analytics applications.
An advantage of the proposed invention is that it allows an end-to-end training process, which means that the machine can automatically learn the best parameters to understand contextual information in terms of spatial and temporal relationships, receiving the video itself as input. . Therefore, this eliminates the need to manually design feature descriptors that better encode the input data, making it possible to achieve higher accuracy rates and provide a better classifier for the target product.

Examples of direct applications of the proposed invention

[0024] Triggers for smartphone camera: some camera features may be started according to the scene being recorded. For example, if some specific actions are detected using the proposed method (such as a baseball swing during a baseball game), the slow-motion or super slow-motion features could start. The proposed method can be executed every time the camera is being used.

[0025] Camera Scene Optimizer: Scene optimizer can be adjusted not only based on still image properties (as is currently done), but also based on video category. Using the proposed method to analyze a set of camera frames (eg a few seconds), the video category can be determined and then camera resources can be adjusted to improve image quality during video recording. or image.

[0026] Enhanced Gallery: Smartphone video gallery may have a sorting/sorting option based on video categories. Video search can also be improved using the proposed invention to categorize videos. Users could better search for videos of specific activities or categories (eg sporting events, car racing, dancing, etc.) The proposed invention could work at the same time as the image analyzer works in order to categorize and index the content of the gallery.

[0027] Triggers for health monitoring systems: the proposed invention could work for all content being recorded/monitored in this type of system and, if some specific actions are detected (eg falls), an emergency alert could be given to system operators to request attention.

[0028] Triggers for video surveillance systems: the proposed invention could run in the background for all content being recorded in this type of system and, in case a specific action is detected (eg fight, person running among the crowd, several people running), an emergency alert could be shown to the system operators in order to call for attention.

Brief description of the drawings

[0029] The objectives and advantages of the present invention will become clearer through the following detailed description of the example and the non-limiting drawings presented at the end of this document:

[0030] Figure 1 shows a scenario of the proposed invention being used, in which a scenario or people are being recorded by cameras that generate digital videos.

[0031] Figure 2A depicts the flowchart of the proposed approach to the system.

[0032] Figure 2B shows the flowchart of the proposed approach to the method.

[0033] Figure 3 compares a common spatial pyramid with the proposed egocentric pyramid.

[0034] Figure 4 shows the experimental results of the proposed invention in comparison with reference methods based on object occurrences.

[0035] Figure 5 shows the experimental results of an embodiment of the proposed invention (TERN) combined with other state of the art architectures for action recognition, which also consider motion information.

[0036] Figure 6 shows the differences in precision between the proposed invention alone (TERN) and the proposed invention combined with the prior art approach (TERN + Two-Way (TSN)).

Detailed Description Features of the invention

[0037] The proposed invention describes an approach for recognizing human activity in videos, which can encode contextual information by spatial and temporal relationships of objects and the main agent in a scene. Spatial relationships are encoded through the Temporal Egocentric Relational Network (TERN, acronym for Temporal Egocentric Relational Network), occurrences of objects and egocentric pyramids, in which the latter is a technique proposed to encode the spatial arrangement of objects around the main agent in the scene. Temporal relations are coded by combining relational networks and temporal segment networks. All coding steps are differentiable, allowing an end-to-end learning process, which makes it possible to obtain higher recognition rates and provide a better classifier for the target product. The proposed approach, although validated for the recognition of human activity, can be used for other tasks related to video analysis and comprehension.

• 1) A codificação de informações contextuais, ou seja, as relações espaciais e temporais de objetos em relação ao agente principal na cena, melhora a precisão para sistemas de reconhecimento de atividade humana com base em dados de vídeo;
• 2) Essa codificação de informações contextuais pode beneficiar outras aplicações de análise de vídeo e compreensão de vídeo;
• 3) Sistemas de compreensão de vídeo mais precisos permitem uma melhor extração de conhecimento de conjuntos de dados de vídeo, incluindo melhor categorização de vídeo, melhor pesquisa de vídeo, etiquetagem automática de vídeo, sumarização de vídeo, entre outras aplicações;
• 4) Melhores sistemas de compreensão de vídeo exigem menos anotação humana/manual de conteúdo de vídeo;
• 5) Incorporar a invenção em dispositivos móveis permite melhores aplicações de inteligência artificial (IA) para os usuários;
• 6) Melhor categorização de vídeo e pesquisa de vídeo em aplicativos móveis (por exemplo, Galeria).

[0038] Based on the description of the existing methods and the description of the proposed invention, the following advantages for the invention can be enumerated:

• 1) The encoding of contextual information, that is, the spatial and temporal relationships of objects in relation to the main actor in the scene, improves the accuracy for human activity recognition systems based on video data;
• 2) This encoding of contextual information can benefit other video analytics and video comprehension applications;
• 3) More accurate video comprehension systems allow better knowledge extraction from video datasets, including better video categorization, better video search, automatic video tagging, video summarization, among other applications;
• 4) Better video comprehension systems require less human/manual annotation of video content;
• 5) Embedding the invention in mobile devices allows better artificial intelligence (AI) applications for users;
• 6) Better video categorization and video search in mobile apps (eg Gallery).

[0039] The main objective of the proposed invention is the recognition of human activities based on videos. However, the invention can also be used for any other video analytics application. Incoming videos are recorded by cameras and may be available from different sources such as YouTube, surveillance cameras, smartphones, etc. Recognition algorithms can understand activities performed on video, such as horse racing, kayaking, applying lipstick, walking a dog, playing the cello, and others.

[0040] As shown in Figure 1, a user or scenario (101) is recorded by a camera (102), generating a video (103). This video (103) is then processed by the present invention (104), which can run on a computer, remote server, mobile device or other device including cloud servers. These videos are then processed by the proposed invention in order to produce an output, which are the recognized human activity categories. The recognized activities are the output (105) of the proposed system. The camera device (102) can be an IP camera, a smartphone, a surveillance camera, or any other device that comprises a camera. The proposed invention has the advantage of allowing recognition systems to learn contextual relationships to improve accuracy.

[0041] Figure 2A illustrates the system where the proposed invention is performed. Figure 2A depicts the system flowchart, in which the input video data (201A) goes through the concept detection module (202A) in order to obtain the concepts (for example, objects), then the contextual characteristics are captured by the Egocentric Pyramid (203A) and by the Concept Pairing Module (204A), whose outputs are processed by the temporal egocentric relational network (205A), considering the spatial and temporal relations of the concepts, generating the output (206A) of the system. The relationships are computed for both object-agent and object-object pairings and used as input to a neural network, which learns the best combinations of concepts and parameters.

• i. computar, pela pirâmide egocêntrica, relações espaciais de conceitos detectados em relação ao principal agente da cena (203B) (pareamentos conceito-agente);
• II. computar pareamentos entre conceitos (204B) (pareamentos conceito-conceito);
• III. fazer uso de pareamentos de conceito e pareamentos egocêntricos para aprender suas relações temporais, pela Rede Relacional Egocêntrica Temporal, para gerar vetores de características (205B);

d. emitir as predições obtidas pela Rede Relacional Egocêntrica Temporal (206B).[0042] Figure 2B illustrates the steps of the method of the proposed invention. Figure 2B shows the video recognition flowchart using contextual descriptors capable of encoding spatial and temporal relationships of concepts comprising the steps of:
The. acquiring input video data (201B);
B. processing the input video data to detect concepts in the video (202B);
ç. compute contextual descriptors of the detected concepts, including the following sub-steps:

• i. compute, through the egocentric pyramid, spatial relations of detected concepts in relation to the main agent of the scene (203B) (concept-agent pairings);
• II. compute pairings between concepts (204B) (concept-concept pairings);
• III. make use of concept pairings and egocentric pairings to learn their temporal relationships, through the Temporal Egocentric Relational Network, to generate feature vectors (205B);

d. issue the predictions obtained by the Temporal Egocentric Relational Network (206B).

[0043] The purpose of the invention is to recognize human activities based on video, which is the data input (201A) of the system. Input video data (201A) is processed in order to detect concepts (202B). Concepts can be objects, people, object parts, etc. The concepts are then passed to the module to generate the contextual characteristics, the egocentric pyramid (203A). This module is divided into two sub-modules. The egocentric pyramid (203A) obtains information about the spatial relationships of objects and the main actor in the scene. The concept pairing module (204A) obtains spatial relationships from object pairings. Spatial relationships are used as input to a Temporal Egocentric Relational Network (TERN) (205a), which not only learns the best object pairings, but also their temporal relationships. The output (206A) of the method is the predictions in terms of human activities, considering human activity recognition, or any other video classification task.

[0044] Input video data (201A) can be obtained from, including, but not limited to, video cameras, smart phones, wearable cameras, surveillance cameras, websites such as YouTube, and others. The input video data (201A) is initially divided into t segments of size T. From each segment, a random chunk is sampled with length |Si| such that |Si| ≤ T. Video segments can be used as input to the concept detection module (202A).

[0045] The detection of concepts, which can be objects, people, object parts and others, can be based on object detectors, including, but not limited to YOLO ("YOLO9000: Better, Faster, Stronger", Redmon and Farhadi , CVPR, 2017), SSD ("SSD: Single shot multibox detector", Liu et al., ECCV, 2016), Faster-RCNN ("Faster R-CNN: Towards real-time object detection with region proposal networks", Ren et al., NIPS, 2015), etc. The concept detection module (202A) produces all concepts detected in the input video data (201A).

[0046] The egocentric pyramid (203A) is responsible for encoding the spatial relationships between the concepts and the main agent in the scene (concept-agent pairings). Figure 3 shows an egocentric pyramid compared to an ordinary spatial pyramid. The egocentric pyramid divides the image space according to the main actor in the scene. An egocentric pyramid takes an agent as a reference, and builds a spatial pyramid centered on it. An agent is chosen as the central concept that performs an activity. This can be determined in a number of ways, including, but not limited to, choosing the concept with the highest probability assigned by the concept detector; tracking the concept with the greatest probabilities, among others.

[0047] An advantage of the egocentric pyramid (203A) over the common spatial pyramids is that the elements surrounding a given agent are position-invariant. Since a traditional spatial pyramid takes the center of the frame as a reference, it can be susceptible to problems because it assumes that all activities are always performed in the center of the video, which is not necessarily true. For example, if the dog walking activity is being addressed and the person walking the dog starts in the upper left corner of the frame and then moves to the lower right corner, the bin corresponding to 'dog' will be assigned to the histograms corresponding to the second and fourth quadrants. This will generate a different histogram signature for the same activity, as in a case where the person with the dog starts in the lower left corner and moves to the lower right corner. However, this is prevented by the egocentric pyramid, as it takes the agent's position as a reference instead of the center of the frame, since the relevant elements move around the one performing the action.

[0048] In the egocentric pyramid (203A), in case a concept is in the limit of more than one quadrant or a concept is divided into several quadrants, there are different options to update the corresponding quadrant histograms. One option is to update only the histogram where the concept occupies the most part. Another option is to use the concept dimensions (determined by the bounding box calculated by the concept detector) to update all quadrant histograms weighted by the portion of the concept that belongs to each quadrant.

[0049] The egocentric pyramid (203A) can also be used in case there is more than one prominent agent in the scene. This can happen when concept detector probabilities are similar for more than one concept (eg three concepts with scores around 0.3). In this case, all concepts with similar high probabilities are used as agents and a separate egocentric pyramid is computed using each concept as the main agent. All of these egocentric pyramids can then be used as input to the Temporal Egocentric Relational Network (TERN) (205A).

[0050] The concept pairing module (204A) obtains the spatial relationships of all pairings of a concept with other concepts (concept-concept pairings).

[0051] The concept-agent pairings and the concept-concept pairings obtained respectively by egocentric pyramids (203A) and concept pairing modules (204A), can be used as input to a temporal egocentric relational network (TERN). That is, contextual characteristics can be computed by TERN considering only concept-agent information, only concept-concept information, or both types of pairings.

[0052] Due to having a machine learning system, it is necessary to first train the method. This learning phase can be based on a certain set of video data, in which the system will learn the parameters and generate a classification model. This can happen separately from where the system is used, that is, from the inference phase. For example, the classifier can be trained on a computer/server and then the learned model can be used on a mobile device. It is also possible to have both phases in the same location. In addition, it is possible to update or retrain the classifier at certain time periods using new data, which may come from user datasets. The proposed invention has no restriction on where the training and inference phases occur.

[0053] The temporal egocentric relational network (TERN) (205A) makes use of the pairings in order to learn characteristics and perform the classification in a unified way. TERN is designed to relate over concept information over time, which means that TERN will learn the spatial and temporal relationships to contextual feature descriptors. Given a sequence of video clips S = {S1,S2,...,St} comprising t clips sampled uniformly or randomly, the temporal egocentric relational network is defined as
TERN(S) = G(RΦ(S1), RΦ(S2), ..., RΦ(St)),
where St is a video clip, RΦ is a relational network with parameters Φ, and G is an aggregation/pooling operation. In particular, a relational network RΦ, given parameters Φ = [Φ1,Φ2] , is defined as

[0054] Here, 0 = {0i}ni=1 represents a set of n inputs of detected concepts (for example, objects), where 0i is the i-th concept such that 0i ∈ ℝf; and functions fΦ1 and gΦ2 are multilayer perceptrons (MLP) parameterized by parameters Φ1 and Φ2, respectively.

[0055] The learning procedure generates a model that will be used while using the system for feature extraction and classification. In this training configuration, sampling random chunks is a data augmentation technique where at each iteration a different chunk is seen by the network. At the same time, it is guaranteed that the video is seen as a whole, according to the number of segments and the length of the excerpt. For example, if three segments are chosen, it is ensured that the network will see the data for the beginning, middle and end of the video. The consensus layer then makes the network learn weights that favor consistency between them. TERN benefits from the efficient reuse of weights between concept pairings and time segments. This imposes restrictions that act as regularizers, while reducing the number of parameters, as pointed out in the literature.

[0056] The entire process of obtaining contextual resources in the proposed invention (203B-204B-205B) is differentiable, which means that the system can be trained end-to-end, from concept detections to activity predictions. This allows the system to obtain the best parameters automatically, without requiring human intervention or specialized knowledge for the problem domain.

[0057] Experiments on the UCF101 human activity recognition dataset demonstrate the improvements in accuracy over existing reference methods when using the proposed invention. Initially, preliminary experiments are conducted in the 1st division of the UCF101 dataset to evaluate the egocentric pyramid alone and reference methods based on object occurrences, namely, spatial pyramid, object probabilities, as reported by Jain et al., a implementation using an object detector, and extension based on occurrences. Figure 4 shows the experimental results of the proposed invention in comparison with reference methods based on occurrences of objects. First, it can be seen that using the SSD detector to reproduce the reference method produces a gain of 5 percentage points (p.p.) when compared to the original report by Jain et al. (“What do 15,000 object categories tell us about classifying and localizing actions?”, Jain et al., CVPR, 2015). Furthermore, by evaluating the representation of the number of occurrences alone, a result similar to the baseline (65%) is achieved. It shows that occurrences alone are not as representative as object probabilities. Finally, the combination of probabilities and occurrences obtains an accuracy of 72%, which shows that they are complementary. Subsequently, the deep learning architecture of the temporal egocentric relational network (TERN) is evaluated. Comparing the TERN results with the egocentric pyramid alone, there is an improvement of 1.92 pp, which suggests that there are other non-explicit contextual features that can be explored in addition to spatial arrangements, such as temporal signals of relative frame positions and multiple stretches, and other spatial indications such as size and precise location.

[0058] Figure 5 shows the experimental results of an implementation of the proposed invention (TERN) merged with other latest generation architectures for action recognition, which also consider motion information. The results show the competitiveness of the proposed invention with other approaches.

[0059] Figure 5 shows the TERN evaluated in the three divisions of the UCF101, where the last column is the mean of the accuracy. These results are compared with the known two-way networks, with modifications by Wang et al. (“Towards good practices for very deep Two-Stream convnets”, Jain et al., arxiv, 2015). The results of Wang et al. were obtained by executing the code provided by the authors. For a fair comparison, Temporal Segment Networks (TSN) are also included. Compared to the spaceway, TERN obtains a similar result, which may suggest that it is encoding part of the necessary spatial information encoded by the spaceway. However, when the spaceway is also embedded in the TSN framework, the difference between the two approaches increases. The reason could be that TSN allows the spatial convolutional network to learn temporally consistent visual patterns that are not available from object detections alone, such as scene/background and pose suggestions. The merger of TERN forecasts with the temporal lane produces an improvement close to the two-way network, suggesting complementarity between the two modalities. Comparing with the two-way network alone, it should be noted that the TERN + Two-Way fusion is able to slightly improve the recognition (1.34 p.p.). However, this gain is smaller when merged with TSN (0.04 p.p.), demonstrating a lower complementarity between the two approaches.

[0060] To better understand how TERN and TSN affect each other, the difference in accuracy between the proposed invention alone (TERN) and the proposed invention merged with the prior art approach (TERN + Two-Way (TSN)) is analyzed. Figure 6 shows a summary of the scenarios that the TERN + Two-Way (TSN) performed better (positive bars) and worse (negative bars) than the Two-Way network (TSN) alone. The graph shows that activities that have objects are easier to recognize, such as archery, punching bag, shotput, and typing, while TERN performs poorly for classes where objects are missing or difficult. detection (applying eye makeup), appearance and/or movement plays an important role (long jump, punch), or objects are absent among the detector categories (stick throw). Comparing TERN alone with TSN, TERN performed better in situations where objects played an important role, such as playing guitar or riding. Still, looks and movement perform better in most classes, as expected. However, as shown in Figures 5 and 6, there are activity categories that benefit from merging with TERN, suggesting that there are contextual clues that can be explored by action recognition architectures beyond appearance/movement.

[0061] Although the present invention has been described in connection with certain preferred embodiments, it is to be understood that it is not intended to limit the invention to those particular embodiments. Rather, the present invention is intended to cover all possible alternatives, modifications and equivalents within the spirit and scope of this disclosure as defined by the appended claims.

Claims (11)

Video recognition method using contextual feature descriptors capable of encoding temporal and spatial relationships of concepts characterized by the fact that it comprises the steps of:
The. acquiring input video data (201B);
B. processing the input video data to detect concepts in the video (202B);
ç. compute contextual characteristics descriptors of the detected concepts, in addition, comprising the following sub-steps:

• i. compute, through the egocentric pyramid, the spatial relationships of detected concepts in relation to the main agent of the scene (concept-agent pairings) (203B);
• ii. compute pairings between concepts (concept-concept pairings) (204B);
• iii. make use of pairings to learn their temporal relations, through the Temporal Egocentric Relational Network (TERN), to generate feature vectors (205B);

d. issue the predictions generated by the Temporal Egocentric Relational Network (206B). Method according to claim 1, characterized in that the input video data acquisition step (201B) comprises dividing the video into t video segments of equal size T and then from each segment , a random stretch Si is sampled with length |Si| such that |Si| ≤ T. Method, according to claim 1, characterized in that the step of computing the detected concepts assigns probabilities to the captured context in order to determine the concepts and the agent(s) in the video excerpts. Method, according to claim 1, characterized by the fact that the egocentric pyramid considers as the main agent in the scene the concept with the highest detection score obtained by the concept detector. Method, according to claim 1, characterized by the fact that when more than one agent is in the scene, the number of agents results in the same number of egocentric pyramids, each of them considering a separate concept as the agent in the scene. Method, according to claim 1, characterized by the fact that the Temporal Egocentric Relational Network learns temporal relationships of both egocentric pairings and concept pairings. Method, according to claim 1, characterized by the fact that the Temporal Egocentric Relational Network learns temporal relationships from egocentric pairings. Method, according to claim 1, characterized by the fact that the Temporal Egocentric Relational Network learns temporal relationships from concept pairings. Method, according to claim 1, characterized by the fact that the Temporal Egocentric Relational Network makes use of pairings in order to learn characteristics and classify in a unified way. Method, according to claim 1, characterized by the fact that the Temporal Egocentric Relational Network was designed to relate concept information over time. Method, according to claim 1, characterized in that the Temporal Egocentric Relational Network is defined as:
TERN(S) = G(RΦ(S1), RΦ(S2), ..., RΦ(St)),
where St is a video clip, RΦ is a relational network with parameters Φ, G is a grouping/pooling operation. In particular, a relational network RΦ, given parameters Φ = [Φ1,Φ2] , is defined as

where, 0 = {0i}ni=1 represents a set of n inputs of detected concepts (for example, objects), where 0i is the i-th concept such that 0i ∈ ℝ3; and functions fΦ1 and gΦ2 are multilayer perceptrons (MLP) parameterized by parameters Φ1 and Φ2 respectively.

Images