WO2020243116A1

WO2020243116A1 - Self-learning knowledge graph

Info

Publication number: WO2020243116A1
Application number: PCT/US2020/034598
Authority: WO
Inventors: Delia FERNANDEZ CAÑELLAS; Joan Espadaler Rodes; David RODRIGUEZ NAVARRO; Blai GAROLERA HUGUET; Gemma CANET TARRÉS; Aleix COLOM SERRA; Joan MARCO RIMMEK; Elisenda Bou Balust; Juan Carlos Riveiro Insua
Original assignee: Vilynx, Inc.
Priority date: 2019-05-26
Filing date: 2020-05-26
Publication date: 2020-12-03

Abstract

Analysis of content such as multimedia and videos are facilitated by the generation and use of a knowledge graph. The knowledge graph represents entities, such as people, places and organizations, and relationships between these entities. In some embodiments, a knowledge graph is used to generate summaries of video content, the summaries being based on relevance of the entities and relationships found within the video content. Knowledge graphs are optionally used to identify content relationships, disambiguate entities, generation of content summary, automatic video editing, and/or personalization or recommendations. The knowledge graph optionally includes short-term and long-term memories.

Description

Self-Learning Knowledge Graph

CROSS-REFERENCE TO RELATED APPLICATIONS

[01] This application is a Continuation-in part of US patent application Ser. No. 16/566,635 filed September 10, 2019;

this application is a Continuation-in-part of US application Ser. No. 14/486,015 filed December 22, 2016 (now US Pat. No. 10,410,679), which is a Continuation of US application Ser. No. 13/247,771 filed September 28, 2011 (now US Pat. No. 8,869,198);

this application is a Continuation-in-part of US application Ser. No. 16/254,445 filed January 22, 2019, which is a Continuation of US application Ser. No. 14/833,036 filed August 21, 2015;

this application is a Continuation-in-part of US application Ser. No. 16/346,102 filed April 29, 2019, which is a national stage of PCT/US2017/059119, which claims benefit of US provisional patent applications 62/414,308 filed October 28, 2016 and 62/552,369 filed August 30, 2017;

this application claims benefit and priority of US provisional patent application Ser. No.

62/853,154 filed May 27, 2019; and

62/853,047 filed May 26, 2019. The disclosures of all the above patent applications are hereby incorporated herein by reference.

BACKGROUND

[02] Field of the Invention: [03] The invention is in the field of automated learning systems and optionally in the field of content processing to identify contents of a video, audio recording, document or webpage.

[04] Related art:

[05] The internet represents a vast store of knowledge. Systems have been developed to gather this information. However, as the internet is comprised primarily of unstructured data, some of which is of questionable reliability, it is difficult to make use of a mere "data dump" from the internet. For example, systems have been developed to identify objects within images. However, mere identification of objects is typically not sufficient to determine what an image or a video is "about." This makes it difficult to classify and summarize videos and other internet content.

SUMMARY

[06] Development and maintenance of a knowledge tree is achieved by collecting data from multiple sources, and identifying entities and relationships within the collected data. The knowledge graph includes data records having various time horizons, representing at least a "short-term memory" and a "long-term memory." Classifying data between short-term and long-term memory provides a variety of significant advantages. For example, short-term memory may be used for data that is changing in real-time, such as standings in a sports tournament or a game score. While long-term memory may be used for less time variant information or information about entities and relationships that have happened in the past.

[07] The use of short-term and long-term memory also allows for the distinction between information that is more certain or less certain to be correct. Information may be stored in short-term memory until the information satisfies confidence criteria, wherein it is added to long-term memory. For example, identification of confirming or contradictory information may be used to change a confidence in a correctness of data stored in short-term memory.

Confirming and contradicting information may be collected from a variety of source types, e.g., blogs, news video, news articles, live feeds, etc.

[08] Information stored in the knowledge graph is subject to an ontology including predicates and constraints. Predicates define deductions that can be made from information. For example, if Sabrina is the daughter of Simone then Simone can be deduced to be a parent of Sabrina. Constraints limit allowed characteristics and relationships. For example, if Sabrina is a daughter of Simone, then Sabrina cannot be a parent of Simone. Some embodiments of the invention include systems and methods of discovering new predicates and constraints.

Information stored in short-term and long-term memory are optionally subject to different ontologies.

[09] The knowledge graph includes entities, represented by nodes, and relationships, represented by connections between the nodes. Content such as images videos, text and audio are processed based on navigation of the knowledge graph. The entities can be, for example, people, places, events, activities, actions, topics scenes, camera shots, settings, buildings, places, locations, organizations, and/or the like. Entities have properties and relationships with each other. Properties can include a name, location, time, size, or any other characteristic. Edges of the knowledge graph (connections between nodes) represent relationships between entities. Such relationships can include, for example, membership, supervisor, size difference, compatibility, type, owner, proximity, sibling, leader, teacher, interviewer, judge, and/or the like. The knowledge graph is updated over time using information available on the Internet and other sources. These updates include addition of new entities, addition of new relationships, and confirmation of prior entities and relationships. As is discussed further herein, an update to the knowledge graph typically include storing information (e.g., entity characteristics and relationships) is short-term memory and then moving the information to long-term memory after a required level of confidence in the information is achieved.

[010] Various embodiments of the invention include a multilingual system for event detection and comprehension from media feeds. The system retrieves information from news sites and social networks, aggregates them into events (event detection), and summarizes them by extracting semantic labels of its most relevant entities (event representation) in order to answer the journalism W's: who, what, when and where. The generated events populate a Knowledge Base (KB)- transforming unstructured text data to a structured knowledge base representation. An external entity Knowledge Base (VLX-KG) is exploited to help populate VLX- Stories. At the same time, this external knowledge base can also be extended with a Dynamic Entity Linking (DEL) module, which detects Emerging Entities (EE) on unstructured data and adds them to VLX-KG. The system is currently used in production, detecting over 6000 monthly events from over 3500 news feeds from seven different countries and in three different languages. At the same time, it detects over 1300 EE per month, which populate VLX-KG.

[Oil] Various embodiments of the invention include a video processing system comprising: knowledge storage configured to store a plurality of entity data records, each of the entity data records including characteristics of a respective entity, the characteristics including

relationships between the respective entity and other entities represented by other members of the entity records; a resource crawler configured to crawl a computer network in search of content and to retrieve found content, the content including both text and image content; content analysis logic configured to identify entities within the retrieved content; discovery logic configured to compare the identified entities to the plurality of entity data records, to assign the identified entities to corresponding data records, and to identify new entities within the identified entities; relationship logic configured to identify relationships between entities identified by the discovery logic, and to add the relationships to the corresponding entity data records; and a microprocessor configured to execute at least the content analysis logic, the discovery logic, or the relationship logic.

[012] Various embodiments of the invention include a method of generating a knowledge graph, the method comprising: crawling a source of video content to identify and retrieve content; identifying one or more entities within the retrieved content; identifying one or more relationship between the identified entities; updating a knowledge graph based on the identified one or more entities, the updating including creating new entity data records of the knowledge graph for entities not previously identified and including adding to existing entity data records of the knowledge graph for previously identified entities, wherein the knowledge graph comprises nodes representing identified entities and connections representing relationships between the identified entities; and validating the knowledge graph by checking for inconsistencies between the entities and their connections.

[013] Various embodiments of the invention include a method of processing video, the method comprising: receiving a knowledge graph comprising entity data records, the entity data records optionally including short-term and long-term data records, each of the entity data records including characteristics of a respective entity, the characteristics including relationships between the respective entity and other entities represented by other members of the entity records; receiving the video; identifying entities within the video; extracting knowledge regarding the identified entities from the knowledge graph, the knowledge including relationships of the identified entities; identifying representative content within the video based on the identified entities and their relationships; and generating a summary of the video including the representative content.

[014] Various embodiments of the invention include a self-learning video processing system comprising: knowledge storage configured to store a plurality of entity data records, each of the entity data records including characteristics of a respective entity, the characteristics including an identify of the entity and relationships between the entity and other entities represented by other members of the entity data records, the entity data records including short-term data records and long-term data records; a resource crawler configured to crawl a computer network in search of content and to retrieve found content, the content including both text and image content; content analysis logic configured to identify entities within the retrieved content; discovery logic configured to compare the identified entities to the plurality of entity data records, to assign the identified entities to corresponding data records, and to identify new entities within the identified entities, the entities and corresponding entity data records being represented by nodes of a knowledge graph; relationship logic configured to identify relationships between entities identified by the discovery logic, and to add the relationships to the corresponding entity data records, the relationships between entities being represented by connections between the nodes of the knowledge graph, each node

corresponding to at least one of the entity data records, the entity data records comprising the short-term and long-term data records; short-term memory logic configured manage a first subset of knowledge graph in the short-term data records, the first subset including new time- variant content; long term memory logic configured to manage a second subset of the knowledge graph in the long-term data records, the second subset including entity identities and relationships between entities, the identities and relationships being transferred from the short-term data records; transfer logic configured to transfer the found content from the short term data records to the long-term data records, the transfer being dependent on a measure of confidence in correct identification of the entities and/or correct identification of the relationships; and a microprocessor configured to execute at least the content analysis logic, the discovery logic, or the relationship logic.

[015] Various embodiments of the invention include a method of generating a knowledge graph, the method comprising: crawling a source of video content to identify and retrieve content; identifying a first entity within the retrieved content, the first entity being one of a plurality of entities and being associated with entity characteristics; identifying a relationship between the identified first entity and a second entity; updating the knowledge graph based on the identified first entity and the relationship, the updating including storing the relationship or characteristic of the first entity in one of a plurality of short-term data records, wherein the knowledge graph comprises nodes each representing members of the plurality of entities and connections representing relationships between the plurality of entities; and validating the knowledge graph by increasing a confidence level of the relationship or of the characteristic stored in the short-term data record; and transferring the relationship or characteristic of the first entity from the short-term data record to a long-term data record in response to the increased confidence level.

[016] The systems and methods disclosed herein may be applied to a wide variety of content including video, still images, audio, text, unstructured data, and/or the like. For example, processing methods may be applied to RSS feeds including audio, text, metadata, and images.

BRIEF DESCRIPTION OF THE DRAWINGS

[017] FIG. 1 illustrates a video processing system, according to various embodiments of the invention.

[018] FIG. 2 illustrates entity data records stored in a knowledge storage, according to various embodiments of the invention.

[019] FIG. 3 illustrates methods of automatically creating a knowledge graph, according to various embodiments of the invention.

[020] FIG. 4 illustrates methods of processing video, according to various embodiments of the invention.

[021] FIG. 5 illustrates an example of the resulting event information displayed on Vilynx Dashboard, according to various embodiments. On the top is displayed the article category, the title summarizing what happens, and the other properties: when, topic, where and who. Titles from articles clustered give context and additional information on the story. In the bottom the entities in the event semantic pattern are displayed, as related tags. Notice these entities are sorted according to its relevance describing the event and the bar behind the entity box represents the current trendiness of the entity. [022] FIG. 6 illustrates a schema of the News Event Detection pipeline, according to various embodiments of the invention.

[023] FIG. 7 illustrates a pipeline schema of the Event Semantic Pattern extraction module, composed by the Event Pattern Extraction and the Dynamic Entity Linking modules, according to various embodiments of the invention.

[024] FIG. 8 illustrates an event ontology schema, according to various embodiments of the invention.

DETAILED DESCRIPTION

[025] Embodiments of the invention include a knowledge graph, systems and methods of developing and maintaining the knowledge graph, and/or systems and methods of using the knowledge graph to process video, images, audio, text, documents, or any combination thereof. Different embodiments may be employed by different parties. For example, one party may generate and maintain the knowledge graph while another uses the knowledge graph to generate summaries of videos or otherwise understand received content. The processing of video may result in textual, audio, and/or video summaries of the video. For example, in some embodiments the knowledge graph is used to generate a summary of a video that includes selected images and audio from the video, as well as textual tags configured for indexing and searching the video.

[026] FIG. 1 illustrates a Video Processing System 100, according to various embodiments of the invention. Video Processing System 100 is configured to communicate via a Network 115, such as the internet. Network 115 may be used to communicate with optional Content Sources

195 and/or an optional Replay System 185. Contact Sources 195 can include libraries of video, websites, blogs, multi-media publications, news stories, press releases, conversations, and/or the like. Materials found in Contact Sources 195 typically change as new content becomes available via the Internet. For example, as time passes, new news stories are produced about new topics and entities within these news stories can develop new relationships. In another example, new video is continuously being uploaded to video serving websites. As used herein, the term "content" is used to refer to both unprocessed content received from Content Sources 195 and information derived therefrom. For example, content includes entity characteristics and relationships derived from raw content. Video Processing System 100 can include multiple computing devices distributed over numerous locations.

[027] Replay System 185 can include a library of searchable video, a video server, a website, a news delivery services, a video delivery service, and/or the like. For example, in some embodiments Replay System 185 includes a video delivery services configured for a user to search for video by keywords or category, and configured to provide video summaries of video to users.

[028] Video Processing System 100 includes a Knowledge Storage 110. Knowledge Storage 110 is configured to store a plurality of entity data records which form a knowledge graph.

Each of the entity data records is associated with a specific entity. Together the entity data records of the knowledge graph represent information known about things in the real world. The things represented by entities can include, for example, people, places, organizations, positions, events, movies, musical performances, art, actions, activities, work product, products, services, and/or other "things." Preferably, each entity data record represents a different entity, although errors may exist in which two entity data records represent the same thing or an entity data record is inadvertently used to represent two different things.

Relationships (between entities) represented by the plurality of entity data records include, for example, a family relationship, membership in an organization, ownership, a ranking, a spatial relationship, a structural relationship, an employment relationship, a social relationship, a sports record, a location, a value, and/or the like. A particular entity may have a large number of characteristics and/or relationships with other entities. In some embodiments, the attributes of an entity are determined by a predetermined ontology, including rules such as "a person has characteristic type 'physical attribute,' 'name' and/or 'birthdate'" or "two people may be related by "marriage."

[029] In some specific examples:

a) An entity can represent a person having professions of "politician," "developer" and "actor", a position of "President of the United States", an age of "71," etc. This entity can have a relationship "in" with a plurality of video, audio and print sources. This entity can have a relationship of "supervisor" with a member of his staff, relationships of "father" to his children, relationships of "spouse" or "ex-spouse" to his wives, a relationship of "close" to one TV station and a relationship of "distant" to a particular newspaper.

b) An entity can be a sports team associated with a particular sport, a location, a

competition history, an opponent, a mascot, a value, etc. This entity can have relationships with other entities representing players, fans, owners, employees, other teams, competitive titles, broadcasters, etc. c) An entity can be a person having characteristics of birthdate, gender, hair color, living, wealth, profession = "journalist," etc. This person may have relationships with a co anchor, a television network, a college, a political party, political positions, a location, a colleague, a family member, etc.

d) An entity can be a company having assets, locations, etc. and have relationships with customers, suppliers, shareholders, investors, employees, etc. Note that an entity may be characterized by something and also have a relationship with that "something" if the something is another entity. For example, a company can be both characterized by a founder and have a relationship with the founder.

e) An entity can include an event such as opening of a movie, arrest of a celebrity,

congressional hearing, concert, death, election, etc. Such entities can be characterized by a broad set of criteria, such as a date/dime, a value, a location, etc. They may have relationships with events that occurred in the past or will occur in the future. For example, a prior event may have a "causal" relationship with a current event, and a current event may at some future time be designated as having a "causal" relationship with a future event.

[030] In various embodiments, entity data records are divided into at least "short-term data records" and "long-term data records." Particular nodes of the knowledge graph may each be associated with one or more short-term data records and/or one or more long-term data records. Short-term and long-term data records are distinguished by having different levels of confidence in the information stored therein. Typically, short-term and long-term data records are also distinguished in that short-term data records are subject to a shorter "time horizon" relative to long-term data records. For example, short-term data records may represent a subset of the knowledge graph in which new time-variant information/content is stored. While long-term data records may represent a subset of the knowledge graph including, for example, historical events and/or past relationships. Typically, newly received information derived from unprocessed content is first stored in the short-term data records and then transferred to long term data records once a predetermine level of confidence in the correctness of the

information is established. For example, entity characteristics (e.g., identities) and relationships between entities may be transferred from short-term to long-data records based on a measure of confidence in correct identification of the entities or correct identification of the

relationships.

[031] Knowledge Storage 110 typically includes digital memory such as ROM, RAM, an optical drive, a hard drive, and/or the like. Knowledge Storage 110 may also include a database system configured to store entity data records. Knowledge Storage 110 may be configured to store these records using corresponding file and/or data structures. Knowledge Storage 110 optionally includes cloud-based storage and/or a cache for storing Entity Data Records 210 on Video Processing System 100. Long-term and short-term data records are optionally designated using a simple data flag.

[032] FIG. 2 illustrates Entity Data Records 210 stored in Knowledge Storage 110, according to various embodiments of the invention. Each of Entity Data Records 210, individually designated 210A, 210B, 210C, etc., includes characteristics of a respective entity. These characteristics can include features such as a unique identifier, a name, a location, a date of birth, publication date, occurrence frequency, occurrence sources, current state, history, size, and/or any other characteristic that may be assigned to an entity. Each of Entity Data Records 210 further includes relationships between the respective entity and other entities. Some relationships are represented by unique entity identifiers, a relationship type, a relationship state and a relationship history. Relationships may be one-to-one, one-to-many or many-to-many. The Entity Data Records 210 are optionally arranged and/or traversed as a knowledge graph, wherein each node of the graph represents an entity and relationships represents edges of the graph between nodes. In this architecture, once a particular entity is identified related entities may be found by traversing the edges connecting the node associated with the identified entity.

[033] The knowledge graph may be built on structured records defined in part by the ontology of entities. For example, the ontology may specify specific types of relationships that can be identified between persons. Typically, at least part of this ontology is predetermined.

However, in some embodiments, the logic discussed herein is configured to add rules to the ontology. For example, a new type of relationship between entities may be discovered and added to the ontology.

[034] Each of Entity Data Records 210 may include a short-term data record part and/or a short-term data record part. These parts are optionally subject to different ontologies.

Knowledge Storage 110 is optionally further configured to store the ontologies. For example, the ontology of the long-term data records may require additional constraints relative to the ontology of the short-term data records. The additional constraints may include, for example, a requirement that information be confirmed or that the relationship has been stable (e.g., unchanged) over a predetermined time horizon. For example, a constraint in the long-term data record ontology may require that information be confirmed by an independent source optionally of a different source type (e.g., a relationship determined from a video be confirmed by a video from a different source or a newspaper article). Longevity of information (entity characteristic or relationship) stored in in a short-term data record may indicate a lack of contradictory information. Stability may, therefore, also be used as a constraint in the ontology of the subset of the knowledge graph including long-term data records.

[035] The knowledge graph stored in Knowledge Storage 110 is expected to grow over time. For example, new entities, types of entities, relationships and types of relationships can be identified as new Contact Sources 195 become available. In some embodiments, entities (and their relationships) which are found to be irrelevant, misleading, and/or infrequently used are removed or archived. New entities and relationships are optionally first stored in short-term data records. Likewise, new entity characteristics and new relationship types are optionally first included in the ontology of the short-term data records. The age of a relationship and/or entity characteristics in long-term or short-term data records is optionally representative of a confidence in the relationship. Relationships and entity characteristics that "stand the test of time" may be assigned a greater level of confidence.

[036] In some embodiments, Entity Data Records 210 include data regarding a history of use of a particular Entity Data Record 210, or data regarding origin of information within a particular Entity Data Record 210, e.g., what content was used to establish or infer data within the data record. For example, Entity Data Record 210 may include data indicating sources of evidence supporting or contradictory to characteristics of entities or relationships represented by Entity Data Record 210A. This data is optionally used to determine confidence levels regarding content of Entity Data Record 210A.

[037] Video Processing System 100 further includes a Resource Crawler 120. Resource Crawler 120 is configured to crawl a computer network in search of content and to retrieve found content. Specifically, Resource Crawler 120 is configured to crawl Network 115 and identify Content Sources 195 connected thereto. In some embodiments, content may be pushed or pulled from known sources. For example, Resource Crawler 120 may be configured to automatically pull content from established news networks such as BBC, CNN or the New York Times.

[038] The received content is optionally stored in Video Processing System 100, and may include text, images, audio, video, multimedia, executable computing instructions, and/or any other types of content discussed herein. In some embodiments, the received content is stored along with records of its source, date information, confidence ratings (e.g., content from one source may be considered more reliable than content from a different source. Resource Crawler 120 is optionally configured to identify when particular content received from multiple sources.

[039] Video Processing System 100 further includes Content Analysis Logic 130. Content Analysis Logic 130 is configured to identify entities and/or relationships within the content retrieved using Resource Crawler 120. Content Analysis Logic 130 optionally includes one or more rule-based systems and/or one or more machine learning systems such as neural network. Content Analysis Logic 130 may use a wide variety of subsystems to identify entities and relationships. For example, Content Analysis Logic 130 may include a natural language processor (NLP) configured to interpret meaning of text or subtitles within video; an object tagging system configured to identify objects and generate textual tags characterizing objects found with video; a facial recognition system configured to identify specific people within a video; a speech to text converter configured to convert audio to text; a voice recognition system configured to identify which person in a video is speaking, or if audio includes a voice overlay; an audio recognition system configured to identify audio (e.g., specific music, gunfire, sirens, etc.) within content; and/or the like.

[040] Typically, Content Analysis Logic 130 is configured to identify entities and relationships within content using a combination of text, images and/or audio within the retrieved content. Other information, such as a date or identities of one or more sources from which content was received may also be used by Content Analysis Logic 130 to determine entities and/or relationships. For example, that content was retrieved from a particular network show on New Year's Eve may be used to identify a particular person (host) in the video of that content.

[041] Further, Content Analysis Logic 130 is optionally configured to identify entities within content using relationships, and to identify relationships using entities. In the first case, if a first entity is identified, Content Analysis Logic 130 may navigate between nodes (Entity Data Records 210) of the knowledge graph to identify related entities, the edges between nodes being represented by the relationships of each entity. The related entities may then be used as candidates during further analysis of the content. For example, considering a video having two people, where a first person (of the two) is identified and is known to have a relationship "co host" with an entity "The Tomorrow Show." By navigating the knowledge graph, it may be found that another person has a relationship "co-host" or "guest" with The Tomorrow Show. In various embodiments, such navigation may include traversing the knowledge graph between short-term and/or long-term data records. Based on this result, Content Analysis Logic 130 may consider the person found by graph navigation as a candidate for identification of second of the two people in the video. As such, the relationships stored in Entity Relationship Records 210 can be used for identification of additional entities and/or relationships within content.

[042] The result of using Content Analysis Logic 130 to analyze content optionally includes modification and/or creation of new Entity Data Records 210. For example, a newly born royal baby or a new model care may be assigned to a new Entity Data Record 210. Likewise, a state of Entity Data Record 210B may change when a person it represents dies, or an event it represents starts or ends. Updating Entity Data Records 210 using Content Analysis Logic 130 can include addition of new Entity Data Records 210, addition of data to existing Entity Data Records 210, and/or modification or deletion of data previously stored in Entity Data Records 210. When one of Entity Data Records 210 is changed or is new, the related information is optionally initially stored in a short-term data record. Thus, when content/information stored in a short-term data record is changed it may not be transferred to a long-term data record until a confidence in the change meets a required threshold. Alternatively, in some embodiments, content/information stored in a long-term data records is not changed until a threshold confidence in the change is achieved.

[043] Entity Data Records 210 and the entities they represent optionally have temporal characteristics. For example, the relationships and characteristics of an entity may change over time. Specifically, an entity representing an event may have different states before, during or after the event. An entity representing a sports team and an entity representing a person may have relationships over time that change from "plays the sport," "opposing team member," "team member," "former team member," "former player," "coach," etc.

[044] Typically, content/information having relatively shorter time horizons is stored in short term data records while content/information having relatively longer time horizons is stored in long-term data records. As used herein, the term "time horizon" is meant to indicate a length of time in which an entity characteristic or relationship is expected to be true or to last. For example, current polling numbers in an election are expected to change on a regular basis while it is expected that the Abraham Lincoln will always have been president during the US Civil War. In another example, a current season win-loss record of a football team may be expected to change on a weekly basis, but their historical statistics for the 2012-2013 season are expected to stay constant. This while information stored in long-term data structures may include a temporal component (e.g., the First World War occurred before the Second World War), information that changes more rapidly is more likely to be stored in short-term data records. In another example, entities and relationships around the election of 1800 are essentially static. Such information may be stored in long-term memory. In contrast, information regarding the current election, which may change on a daily or hourly basis, may be stored in short-term memory.

[045] Entities and/or relationships can have a "relevancy" characteristic. For example, an entity more likely to be important to and/or representative of a particular piece of content, relative to another entity, is considered more relevant. Relevancy can vary in scope. For example, relevancy of an entity and/or relationship can be global, applicable to a particular subject, and/or applicable to s particular piece of content. As the characteristics of entities and their relationships change over time, relevance of the entity and its relationships can also change. For example, a current political leader may be less relevant than a past political leader. An entity (person) who has a relationship of "Director" with an entity that represents the Federal Emergency Management Agency may have more relevancy during a disaster relative to when a disaster event is not occurring. Relevancy may depend on any characteristic of an entity and/or on the content of any particular piece of content. Relevancy of one entity may depend on what other entities are found within the same content. For example, the "make and model" characteristics of a vehicle entity may be more relevant if all the vehicles in a video clip are of the same type (a car commercial) relative to a video clip in which a statistically random distribution of vehicle types is found. In some embodiments, information stored in long-term data records is considered to be weighted as more relevant relative to information stored in short-term data records, because there can be more confidence in the correctness of information stored in long-term data records.

[046] Video Processing System 100 optionally further includes Discovery Logic 140. Discovery Logic 140 is configured to compare the entities represented by Entity Data Records 210 with entities identified in content using Content Analysis Logic ISO. Discovery Logic 140 is further configured to assign entities within the content to matching members of Entity Data Records 210, and to identify entities found in the content that do not match any member of the Entity Data Records 210, e.g., "new" entities. In various embodiments, if a new entity is found using Discovery Logic 140, a new Entity Data Record 210 is created and populated with available data, to represent the new entity. A new entity is optionally initially represented by a short-term data record. Discovery Logic 140 is optionally included as part of Content Analysis Logic 130. [047] Video Processing System 100 optionally further includes Relationship Logic 150.

Relationship Logic 150 may identify relationships between entities by noting statistical, physical and/or sequential associations between entities. For example, the appearance of a statistical correlation between the audio "Bond, James Bond" and a male actor identified using facial recognition may be used by Relationship Logic 150 to determine a new relationship of "actor plays" between the actor and the character James Bond. In another example, appearance of a person in an image including the Eiffel Tower may be used by Relationship Logic 150 to determine a relationship between an entity representing the person and an entity representing part of Paris of "is in." In another example, a parent/child relationship between two people may be used by Relationship Logic 150 to infer that one of the two people was born before the other or that one of the two people has responsibility for the other. In this last example, one relationship and a set of self-consistent rules (e.g., predicates) are used to infer or deduce another relationship. Relationship Logic 150 is optionally included as part of Content Analysis Logic 130.

[048] Relationship Logic 150 may be configured to identify relationships between entities, to identify new types of relationships, and/or to discover possible inferences between

relationships or entity characteristics. For example, if a person represented by a first entity changes their "profession" from "actor" to "politician," then they may establish new

relationships with political parties, campaigns, elections, reporters and political offices. A new relationship may be one that, in fact, existed in the past but had not previously been used to connect Entity Data Records 210 within the knowledge graph. For example, a previously unidentified relationship between a congressional hearing and a position on the US Supreme court may be identified by Relationship Logic 150. Once a new relationship is identified, it may be considered for addition to existing Entity Data Records 210.

[049] Relationship Logic 150 may include logic configured to infer or deduce a relationship based on entity characteristics and/or one or more already identified relationship. Further, in some embodiments, Relationship Logic 150 is configured to identify new predicates or constraints of the short-term or long-term ontologies. These may be represented by a set of rules, a neural network, and/or statistical rules. For example, statistical discovery in a multi dimensional correlation map may be used to establish a rule that political candidates of a particular political party all have a new type of relationship with an elephant or donkey.

[050] In some embodiments, Relationship Logic 150 is configured to update relationships or entity attributes in short-term data records at a time period shorter than a time period at which Relationship Logic 150 is configured to update relationships or entity attributes in the long-term data records. For example, relationships in the short-term data records may be updated on a minute by minute basis while long-term data records are updated on an hourly or daily basis.

[051] Discovery Logic 140 and/or Relationship Logic 150 optionally include one or more statistical system, machine learning system, and/or rule-based system. For example,

Relationship Logic 150 may include a recurrent neural network trained to identify temporal relationships between entities or trained using a statistical analysis of date discovered using Resource Crawler 120.

[052] Video Processing System 100 optionally further includes STM (Short-Term Memory) Logic 154 configured to manage the subset of the knowledge graph including short-term data records. As discussed elsewhere herein, the short-term data records include content/information such as new time-variant content and/or content for which confidence is not yet sufficient to store in the long-term data records. STM Logic 154 may, for example, monitor the stability and age of information stored in short-term data records, monitor the ontology (predicates and constraints) associated with the information stored in short-term data records, monitor reliability criteria satisfied by information stored in short-term data records, and/or the like. STM Logic 154 is optionally configured to detect conflicts and/or agreement between information stored in short-term data records and long-term data records. STM Logic 154 is optionally included in Content Analysis Logic 130, Discovery Logic 140, and/or

Relationship Logic 150.

[053] In some embodiments, STM Logic 154 is configured to cause Resource Crawler 120 to search for information to confirm (or contradict) a relationship or entity characteristic stored in short term data records. If such information is found, it can be used to verify or discount the information stored in short-term data records. For example, in some embodiments, STM Logic 154 is configured to search for information in written material (e.g., a new article or blog) to confirm a relationship identified in a video. The written material being of a different source type relative to the video.

[054] Video Processing System 100 optionally further includes LTM (Long-Term Memory) Logicl55 configured to manage the subset of the knowledge graph including long-term data records. Typically, the entities and relationships managed by LTM Logic 155 have been transferred from the short-term data records. LTM Logic 155 is optionally configured to monitor the compliance of the information stored in long-term data records with the ontology

(predicates and constraints) associated with the long-term data records. For example, if a new constraint is identified, LTM Logic 155 may be used to confirm that existing information stored in long-term data records complies with this new constraint.

[055] STM Logic 154 and/or LTM Logic 155 are optionally configured to monitor for conflicts between information stored in short-term data records and long-term data records. For example, STM Logic 154 may be configured to monitor for a conflict between information stored in short-term data record and information stored in long-term data records. In a specific example, STM Logic 154 and/or LTM Logic 155 may be configured to detect that a relationship stored in long-term data records that Ernest Hemingway is married to Jane Austin conflicts with a relationship stored in short-term data records that Jane Austin is married to Harry Potter.

Such a conflict may be used to reduce a measure of confidence in the relationship stored in short-term memory. However, LTM Logic 155 may detect that a particular conflict occurs at a higher than expected rate, in which case a confidence in the correctness of the information stored in long-term data records may be lowered. The information stored in long-term data records may then be corrected as discussed elsewhere herein. LTM Logic 155 is optionally included in Content Analysis Logic 130, Discovery Logic 140, and/or Relationship Logic 150.

[056] Video Processing System 100 optionally further includes Transfer Logic 157. Transfer Logic 157 is configured to transfer information stored in short-term data records to long-term data records. (In some embodiments, this "transfer" merely includes changing the status of a data record from short-term to long-term rather than moving the information between storage locations.) In some embodiments, the transfer is dependent on a measure of confidence in correct identification of entities, of entity characteristics, and/or of relationships. [057] In some embodiments, the transfer is dependent on the information satisfying an ontology of the long-term data records, e.g., when additional constraints are satisfied. For example, the ontology may require a specific level of confidence, confirmation from a different type of content source, confirmation from an independent source, confirmation from a source having a relatively high degree of reliability, stability over a predetermined length of time or a predetermined time horizon, and/or the like. In various embodiments, information may be transferred from short to long-term data records when one or more of these requirements are met.

[058] In a specific example, Transfer Logic 157 may be configured to transfer information from short-term data records to long-term data records based on longevity/stability of the relationships or entity characteristics (e.g., identification), a reliability of a data source from which the associated content was received, satisfaction of a long-term data record ontology, and/or independent verification of the relationships or entity characteristics.

[059] Video Processing System 100 optionally further includes Correction Logic 165.

Correction Logic 165 is configured to detect and correct errors in Entity Data Records 210.

These corrections are typically based on an ontology of rules established by an Administrator and/or contrary evidence found in received content. For example, an incorrect relationship that the politician Mitt Romney is a source of wool might occur because of a headline including "Romney" and "Fleece" and a series of articles on breading of Romney sheep. Other examples of rules can include one cannot get married after having died, a person cannot be older than their parent, etc. [060] Types of errors that Correction Logic 165 may be configured to correct include:

Combining or splitting of entities, correction of entity characteristics, and/or correction of relationships. For example, Correction Logic 165 may be configured to determine that Entity Data Records 210B and 210C each represent the same entity. Specifically, if (based on content analysis) Data Record 210B is related to a body builder called Arnold Schwarzenegger and if Data Record 210C is related to a politician called Arnold Schwarzenegger, Correction Logic 165 may determine that these two Arnold Schwarzeneggers are the same. In this case, the two Entity Data Records 210 are preferably combined into one entity data record. The realization that the two Arnold Schwarzeneggers are the same could be based on facial recognition, similarity in the name, identical birthdates/places, having same wife or parents, and/or the like. Likewise, Correction Logic 165 may be configured to determine that Entity Data Record 210A actually represents two different entities. In this case Correction Logic 165 is preferably configured to split a single Entity Data Record 210 into two, or more, separate Entity Data Records 210.

[061] Correction Logic 165 is optionally configured to maintain a relevance and/or a confidence measure for a particular Entity Data Record 210 and/or characteristic thereof. For example, Correction Logic 165 may be configured to calculate a confidence measure for the correctness of a particular relationship between two entities, and to adjust that confidence measure based on supporting or contradicting evidence found in received content. Confidence measures relating to the reliability of specific content sources are optionally used for the maintenance of relevance and/or confidence measure for particular Entity Data Records. [062] In one example, a video determined to include Donald Trump may be shown to 10,000 fans of Donald Trump. If less than 10 users click on the video, then the reliability of the determination that the video includes Donald Trump is reduced and is optionally flagged by Correction Logic 165 as a potential error to be corrected. In another example, a video is determined to include a cure for autism based on a first source, and later the reliability of this determination is reduced (and possible corrected) based on contradictory data from a second source, the second source optionally being rated as a "more reliable" source relative to the first source.

[063] In some embodiments, Correction Logic 165 is configured to detect errors in Entity Data Records 210 by identifying contradictions between different Entity Data Records 210. For example, if two entities are indicated as representing people who are children of each other, this may break a rule stablished by an Administrator as to allowed family relationships. For example, a rule that a parent must be born before a child may be violated. A record indicating that person A is a child of person B contradicts an indication that person B is a child of person A. This indicates an error, which may be in the relationships, in identities of the people, etc.

[064] Correction Logic 165 may be configured to operate on either data stored in long-term or short-term data records. For example, in various embodiments, Correction Logic 165 is configured to correct relationships and entity characteristics in long-term data records based on 1) information stored in the short-term data records, 2) violation of (optionally new) constraints, 3) discovery of new relationships or predicates, and/or 4) discovery of new entities. In a specific example, if LTM Logic 155 detects that a particular conflict between information stored in short-term and long-term data records occurs at a higher than expected, the confidence of the correctness of the information stored in the long-term data records may be reduced. In this case, Correction Logic 165 may be configured to correct the information stored in the long-term data records. Optionally Resource Crawler 120 may be tasked with finding additional confirming or contrary information prior to correcting long-term data records.

[065] In some embodiments, a reliability measure for specific content sources is determined based on an amount of incorrect or misleading information received from that content source. The standard of what is "incorrect" and/or "misleading" may be based on a selected source of benchmark content sources selected by an Administrator, or by a "consciences" of contact sources regarding certain facts. Correction Logic 165 is optionally configured to select a benchmark for correctness on a fact by fact basis. Further, as the "correctness" of a fact may change overtime, Correction Logic 165 is optionally configured to change what is considered correct (and adjust reliability measures accordingly) over time.

[066] In some embodiments, Correction Logic 165 is configured for an administrator to manually flag/correct incorrect data.

[067] Video Processing System 100 optionally further includes Video Analysis Logic 170. Video Analysis Logic 170 is configured to generate a summary of a video, or other content, using Entity Data Records 210. For example, in some embodiments, Analysis Logic 170 is configured to generate a short (e.g., 10-30 sec) video summary of video content and/or a set of searchable tags characterizing the video. The video summary is optionally configured to be used as a preview to the video. In addition to (or as an alternative to) video, Video Analysis Logic 170 may be configured to generate summaries, or otherwise analyze, webpages, multi-media, still images, printed (and scanned) matter, audio, and/or any other type of content. [068] In various embodiments, generation of a video summary using Video Analysis Logic 170 can include identification of entities within the video and providing the Entity Data Records 210 representing those entities to a machine learning system. The output of the machine learning system can include specification of what should be in the video summary. For example, in some embodiments, the output includes identification of which frames of the video should be included in the summary, searchable tags to be associated with the summary, and/or identification of audio clips to be included in the summary. The video frames and audio clips are typically extracted from the content being summarized.

[069] For example, once a video is tagged (characteristics/entities identified) the resulting metadata includes who is in the video and/or what is happening in the video. Based on this, content summarization includes selecting the most relevant people and actions. In a specific example, if a video is about a trip to North Korea by Donald Trump, then the summary would include segments of Donald Trump and important features of North Korea, e.g., video segments showing Trump and Kim Jong Un together. In some embodiments, the contents of a video summary are based on external factors. For example, if a large number of searches are received about Donald Trump stepping on North Korean territory, video segments showing this occurrence may be more likely to be included in a video summary. The contents of a summary are optionally changed overtime as external factors change. In some embodiments, Video Analysis Logic 170 is configured to produce video summaries using a machine learning system trained to generate a summary that results in the most engagement (e.g., clicks), optionally from a specific audience and/or with a long video engagement time. In some embodiments,

Video Analysis Logic 170 is configured to produce video summaries targeted at viewers having different characteristics. For example, one summary of a video might be targeted at men while another summary of the same video might be targeted at women; or different summaries of the same video might be targeted according to age, language, education level, income, shopping history, and/or any other criteria used to target content delivery. In some

embodiments, Video Analysis Logic 170 is configured to generate several summaries of the same video, which can then be A/B tested to product a desired response.

[070] In various embodiments, generation of the video summary using Video Analysis Logic 170 includes identification of entities within the video, determination of relevance of the identified entities, determination of relation relevance, and/or the like. Frames of the video including the most relevant entities and/or relationships are optionally selected for inclusion in the summary. Relevance of entities or relationships used, can be based on global or content specific relevance. For example, some entities may be considered relevant because of a currently trending political event, while some entities in the same item of content may be considered relevant because they all related to the same historical event. In practice, such entities may be found in video coverage of a political party convention and a historical video concerning the US civil war. For the entities related to the political convention, their relevance may be increased because of a noted upward trend in the frequency of related entities in multiple received content, as many news stories are produced relating to the convention. For entities related to the historical event, their relevance may be increased due to the presence of so many entities having a relationship to the historical event in a particular piece of content.

[071] In some embodiments, entities and/or relationships between entities are classified by a duration. For example, information may be considered "short," and "long-term." Short-term data represents entities and/or relationships that are new (their longevity not yet established) or expected to have a limited duration. For example, a relationship been a person and a location may be expected to be limited when the person is attending a concert at the location. However, if this short-term relationship is repeated, e.g., occurs several times a week, it may be inferred that the person has a different (long term) relationship with the location, such as a workplace relationship. Classification by duration may be into more than two groups, and/or may be classified has being periodic or time/date specific. For example, holidays, school years, television seasons, elections, church sermons and the like can be expected to be characterized by time-variant relationships. This time-variance is optionally used by Video Analysis Logic 170 to process content and/or otherwise disambiguate entities and relationships.

[072] Data storage, e.g., physical memory, is optionally divided into short-term and long-term storage. In such embodiments, entity and/or relationship data that is considered to be short term may be stored in short term data storage initially and then transferred to long term storage once the data is determined to represent long-term entities and/or relationships.

Reclassification to longer term may be based on repetition, source reputation, new

relationships, etc.

[073] In some embodiments, classification by duration is valuable in content processing. For example, time-variant

[074] Optionally, Video Analysis Logic 170 is configured to generate a video summary that includes clusters of video frames (or other video parts) representing particular actions occurring in video content. For example, a cluster of video frames may show a baseball player hitting a ball and running bases. These video frames may be selected based on relationships of between the entities representing the player and the baseball game, the ball, the bases and the score.

As such, relationships and/or changes in relationships can represent actions that are relevant and useful for inclusion in a video summary. Video Analysis Logic 170 is, thus, configured to generate the video summary by selecting parts of content video based on actions associated with (or changes in) relationships stored in Entity Data Records 210.

[075] Video Analysis Logic 170 optionally includes a machine learning network trained using received content and/or the knowledge graph including Entity Data Records 210. For example, Video Analysis Logic 170 may be configured to recognize things (e.g., faces) in specific content or contexts. The relevance of certain content, context, and/or object features may be customer/application dependent.

[076] For example, for face recognition, a neural network may be trained on a large number of images using a loss function configured to separate (as measured by a distance metric or Cosine) faces that are of a particular person from faces that are not of the same person. Once trained thus, embeddings can be generated, these embeddings include feature vectors that can be used to cluster and describe faces. New faces can then be identified using these

embeddings. A new face that does not cluster with a known face may be assigned a new identity and/or new Entity Data Record 210. The face may be assigned a name by correlation within multimodal content, e.g., images of the face are correlated with audio or text including a name

[077] Further, Video Analysis Logic 170 can include subsystems configured to identify entities, such as any of those discussed herein with respect to Content Analysis Logic 130. For example,

Video Analysis Logic 170 may include a Natural Language Processor, logic configured to identify text in video, voice or facial recognition logic, and/or the like These subsystems are configured to, at least, identify entities in received content.

[078] In some embodiments, Correction Logic 165 is configured to receive data regarding use of a video summary produced by Video Analysis Logic 170. For example, Correction Logic 165 may receive information regarding how often a video summary was clicked on, how much of a video summary was viewed, what fraction of the time a full video was watched following viewing of a summary of the full video, and/or how much of the full video is watched by a user who watched the summary. Generally, it is assumed that a video summary that correctly represents a full video will result in a user being more likely to watch more of the full video, relative to video summary that poorly represents the full video. These statistics may be dependent on where a video summary was provided. For example, if a summary video was provided to a blog associated with automobiles based on a "Honda" tag of the video summary, then the number of times the summary is clicked on may be indicative of a correctness of the "Honda" tag.

[079] The feedback received regarding use of a video summary may be used by Correction Logic 165 to determine a representation of how well the video summary represents the corresponding full video. This representation may be used for training of Video Analysis Logic 170, for A/B testing, and/or to determine of whether the video summary is inadequate as a representation of the full video. If a video summary is determined to be inadequate, Correction Logic 165 may be configured to identify entities and/or relationships whose reliability is less than expected. For example, if a video summary performs badly, Correction Logic 165 may be configured to consider which entities and/or relationships used to generate the video summary may be incorrect. As sufficient statistics regarding the use of numerous video summaries are corrected, errors in particular members of Entity Data Records 210 can be identified. Output of Correction Logic 165 is optionally further used to train a machine learning system of Video Analysis Logic 170. For example, the output may be used as part of a feedback look indicative of a performance of the machine learning system.

[080] Video Processing System 100 optionally further includes Training Logic 175 configured to train the Relationship Logic 150, Discovery Logic 140, STM Logic 154, LTM Logic 155, Transfer Logic 157, and/or Video Analysis Logic 170. Training Logic 175 is optionally configured to use feedback generated by Correction Logic 165. The training performed by Training Logic 175 can include, for example, training of a neural network or other machine learning system.

[081] In some embodiments, Training Logic 175 is configured to train STM Logic 154 to apply the ontology of the short-term data records and/or to train LTM Logic 155 to apply the ontology of the long-term data records. In various embodiments, Training Logic 175 is configured to change existing predicates and constraints, and/or to identify new predicates and constraints in either of the ontologies. Training Logic 175 is optionally configured to use statistical analysis, to be triggered by a conflict between information stored in long-term and short-term data records, to be triggered by errors found by Correction Logic 165, and/or the like. Typically, it is more likely that Training Logic 175 will change predicates and constraints of the short-term ontology relative to the long-term ontology. For example, in some

embodiments, changes to predicates and constraints are first made in the short-term ontology and made in the long-term ontology if overall results are improved. Training Logic 175 may use any of the techniques know in the art of machine learning. [082] Video Processing System 100 optionally further includes Video Storage 180. Video Storage 180 is configured to store video summaries created using Video Analysis Logic 170. Video Storage 180 is optionally configured for providing the stored video summaries in requests from advertising and/or promotional systems. For example, the stored video summaries may be stored in a data based and be searchable using tags associated with each of the video summaries.

[083] Video Processing System 100 further includes at least one Processor 190. Processor 190 typically includes an electronic circuit configured to execute computing instructions. Processor 190 may be configured as a specific purpose processor by loading any of the various types of logic disclosed herein into registers of Processor 190. For example, Processor 190 may be configured to execute Video Analysis Logic 170, Correction Logic 165, Content Analysis Logic 130, Discovery Logic 140, STM Logic 154, LTM Logic 155, Transfer Logic 157, and/or

Relationship Logic 150. In various embodiments, specific knowledge graphs can be specialized to a specific topic and/or application. For example, a knowledge graph could be dedicated to identification of insects, to sports, to finance, to news, and/or the like. A knowledge graph may be confidential and/or dedicated to use for a specific customer. Node of a generalized knowledge graph may include a pointer to a specialized knowledge graph. For example, a documentary about insects may lead to Entity Data Records 210 regarding insects in general, which include (navigable) pointers to a specialized knowledge graph about insects. Likewise, in some embodiments, a generalized knowledge graph includes Entity Data Records 210 associated with "news." These Entity Data Records 210 may include pointers which may be navigated to specialized knowledge graphs related to political news, sports news or entertainment news. Knowledge graphs may be isolated or may include pointers to each other.

[084] FIG. 3 illustrates methods of creating a knowledge graph, according to various embodiments of the invention. Creation of the knowledge graph includes receiving content from one or more sources, identifying entities and relationships within the received content, and generating or modifying Entity Data Records 210 to represent the identified entities and relationships. The generated knowledge graph is optionally used to process video as discussed herein, e.g., in relation to FIG. 4.

[085] In a Crawl Step 310, a source of content is accessed to receive content, such as video content or text documents. Crawl Step 310 optionally includes using Resource Crawler 120 to navigate a computer network, e.g., the internet. The content retrieved can include any of the types of content discussed herein. In some embodiments, Crawl Step 310 includes receiving material from a large number of content sources. For example, Resource Crawler 120 may be configured to retrieve content from news websites, sports channels, television stations, blogs, social networking sites, radio stations, video or audio streaming services, and/or the like.

Different content sources may be associated with different reliability ratings. The content retrieved in Crawl Step 310, can include any of the content types discussed herein.

[086] In alternative embodiments, content may be pushed to Video Processing System 100 from a content source. For example, a publisher of news or sports videos may send the content to Video Processing System 100 in order to contribute to the knowledge graph and/or to as part of a request for a summary of the video content. Content received in Crawl Step 310 optionally further includes video sent to Video Processing System 100 for the purpose of generating a summary of the sent video.

[087] In an Identify Entity Step 320, one or more entities are identified in the content received in Crawl Step 310. Identify Entity Step 320 is optionally performed using Content Analysis Logic 130. The entities identified may be identified visually, from audio, and/or from text. Each identified entity is optionally associated with specific frames of multimedia content in which that entity is found. Identify Entity Step 320 can include using any of the subsystems of Content Analysis Logic 130 discussed herein.

[088] In an Identify Characteristics/Relationships Step 330 information about the entities identified in Identify Entity Step 320 is identified. Identify Characteristics/Relationships Step 330 is optionally performed using Relationship Logic 150. This information can include relationships between identified entities and each other and/or previously identified entities. This information can further include identifying any of the other characteristics regarding entities discussed herein. For example, the types of characteristics identified in Identify Characteristics/Relationships Step 330 can include any of those taught to be stored in Entity Data Records 210. In some embodiments, Identify Characteristics/Relationships Step 330 includes using Content Analysis Logic 130 to infer relationships between entities based on existing relationship, actions that occur within the content (e.g., changes in relationships), and/or rules established by an Administrator.

[089] In some embodiments Identify Characteristics/Relationships Step 330 includes using long-term data records and/or short-term data records of the knowledge graph to retrieve relationships and/or characteristics of a particular entity. Optionally, relationships and characteristic stored in long-term data records are associated with a higher degree of confidence and are given preference or priority over relationships stored in short-term data records. In some cases, relationships are identified by navigating between long-term data records within the knowledge graph.

[090] When an identified entity corresponds to an entity represented by one of Entity Data Records 210, Identify Characteristics/Relationships Step 330 typically includes updating the corresponding member of Entity Data Records 210 based on the identified characteristics and/or relationships. These updates are optionally performed first in short-term data records and later transferred to long-term data records if criteria are met, as discussed elsewhere herein.

[091] In Identify Entity Step 320 or Identify Characteristics/Relationships Step 330, multiple clues may be used to make the identifications. For example, visual clues may be combined in any combination with temporal clues, textual clues, and/or audio clues. In a specific example, the color and/or visual look of a sports jersey may be used to identify a team and sport, and to identify relationships between players. At the same time OCR of a player's name on one of the jerseys may be used to identify that player. The identification of the team and a relationship between the team and the player can then be used to confirm identity of the player. In various embodiments, 2, 3, 4 or more clues may be used to improve the reliability of entity and/or relationship identifications. Such clues may be more useful when they are derived from different (orthogonal) features, e.g., visual, audio, textual, temporal (short- or long-term events or event timing) features of processed content. Verbal clues are an example of audio clues, to which a speech to text converter may be applied. Verbal clues may include tone of voice, rate of speech, accent, volume, etc.

[092] In an optional New Entity Step 340, previously unidentified entities and/or relationship, which were first identified in Steps 320 and/or 330, may automatically result in establishment of new Entity Data Records 210 and/or new types of relationships between entities. Optionally, such new Entity Data Records 210 are first stored in short-term data records, and only transferred to long-term data records if predetermined criteria are met as discussed elsewhere herein. For example, if a person identified in received content does not appear to be represented by any existing Entity Data Records 210, then a new member of Entity Data Records 210 may be created and associated with that person in a short-term data record. More specifically, if an image of a person is subject to facial recognition and associated text indicates a new name, then analysis of Entity Data Records 210 may indicated that none of the current Entity Data Records 210 correspond to that person, then a new Entity Data Record 210 may be created and populated with the available information about that person.

[093] In an Update Step 350 the knowledge graph is updated based on entities and entity characteristics identified in Steps 320 and/or 330. This updating can include creating new Entity Data Records 210, creating new fields within Entity Data Records 210, and/or changing, adding to or deleting entity characteristics (including relationships) stored in Entity Data Records 210. Update Step 350 can include adding any new Entity Data Records 210 created in New Entity Step 340 to the knowledge graph. Typically, the addition of a new Entity Data Record 210 to the knowledge graph includes identity relationships between the new entity and one or more previously existing entities. New Entity Step 340 and Update Step 350 are optionally performed using Discovery Logic 140. In various embodiments, Update Step 350 is performed on short-term and/or long-term data records.

[094] In an optional Validate Step 360 the knowledge graph, is validated by checking for inconsistencies between Entity Data Records 210 and/or their relationships. Validate Step 360 may be performed using Content Analysis Logic ISO, Relationship Logic 150, STM Logic 154,

LTM Logic 155, and/or Correction Logic 165. Examples of inconsistencies that may be checked for include: violations of Administrator established rules, inconsistent dates,

impossible/conflicting relationships, more than one Entity Data Record 210 representing the same entity, one Entity Data Record 210 representing more than one actual entity, inconsistent facts, and/or the like. For example, if a determination is made that John just got married to Jane, but John has already dead, it may be assumed that the marriage occurred prior to the death. In some embodiments, Validate Step 360 includes comparing data stored in short-term data records to data stored in long-term data records, or using Resource Crawler 120 to identify confirming or contrary information from independent sources. Validate Step 360 may include increasing a confidence measure of the correctness of information stored in short-term data records.

[095] In some embodiments, Validate Step 360 includes identifying incorrect relationships or entity characteristics in short-term data records or long-term data records. This identification can be based on 1) conflicts between information stored in these two types of data records, and/or 2) violation of an ontology of the short or long-term data records. Optionally, a conflict or violation results in reduction of a measure of confidence in the conflicting information. In this case, multiple conflicts or violations may be required before the information is determined to be "incorrect."

[096] In some embodiments, Validate Step 360 includes using Correction Logic 165 to make corrections to the ontology associated with short-term data records or long-term data records. For example, it may be determined that a conflict is the result of an incorrect predicate or constraint, and this incorrect predicate or constraint may be changed. Such a chance can be made automatically or with human intervention.

[097] In an optional Transfer Step 370, information is transferred from a short-term data record to a long-term data record. Specifically, a relationship or a characteristic of an entity is transferred between short and long-term data records, optionally using Transfer Logic 157. As noted else where herein, this transfer may be dependent on a wide variety of factors related to confidence in the correctness of the information transferred. For example, the transfer may be dependent on identifying confirming information from a different, independent content source; satisfaction of additional constraints; and also requiring that the information be stable for a predetermined period of time. In some embodiments, Transfer Step 370 is dependent on navigation between long-term and/or short-term data records of the knowledge graph. In some embodiments, Transfer Step 370 includes transfer of some but not all of the entity attributes and relationships in a short-term data record to a long-term data record.

[098] In an optional Train Step 380, the knowledge graph represented by Entity Data Records 210 and their connections is used to train a machine learning system. This machine learning system is optionally configured to identify entities and relationships between entities in content. For example, the trained machine learning system may be part of Video Analysis Logic 170, STM Logic 154, LTM Logic 155, Transfer Logic 157, Relationship Logic 150, Discovery Logic

140, and/or Correction Logic 165.

[099] FIG. 4 illustrates methods of processing video, according to various embodiments of the invention. These methods may be adapted to the processing of other types of content. Briefly, received content is processed by identifying entities and their relationships within the content. This identification typically makes use of a knowledge graph, such as one generated using the methods illustrated in FIG. 3. A result of the video processing can include a summary of the processed video. For example, a 10-minute video may be processed to produce a 20 second video summary intended to convey the "meaning" or "essence" of the full video. The video summary may be used as a searchable preview of the full video. Steps of the methods illustrated in FIG. 4 are optionally performed using Video Analysis Logic 170.

[0100] In a Receive Graph Step 410, a knowledge graph comprising Entity Data Records 210 is received. Each of the Entity Data Records 210 includes characteristics of a respective entity.

The characteristics optionally include relationships between the respective entity and other entities represented by other members of the Entity Data Records 210. The knowledge graph is optionally generated using the methods illustrated by FIG. 3 and may be stored in Knowledge Storage 110. The knowledge graph optionally includes short-term data records and long-term data records. The short-term data records having a time horizon shorter than a time horizon of the long-term data records, and/or the long-term data records having a level of confidence in correctness that is greater than the short-term data records.

[0101] In a Receive Video Step 420, video is received for processing. The video can include images, text, and/or audio. For example, the video can include a news program with a reporter speaking, images of the reporter, images of the subject matter of the news, a scrolling text banner on the bottom of the images, and/or the like. The video can also include metadata such as text tags, captions, source and copyright information, etc. The video may be accompanied by additional content, such as text of a webpage in which the video was embedded, or manually provided tags representing contents of the video.

[0102] In an Identify Entities Step 430 one or more entities within the video are identified. This identification may make use of any parts or characteristics of the video. For example, Video Analysis Logic 170 may be used to perform natural language processing of text, facial recognition, audio analysis, object identification, etc. on the video and associated data.

[0103] Identify Entities Step 430 is optionally performed using a machine learning system trained using the knowledge graph. Further, Receive Video Step 420 is optionally performed in real-time as the video is received in Receive Video Step 420. The analysis of the video in Identify Entities Step 430 may include analysis of every frame of the video, key frames, or some other selection of frames. As a result, different entities may be identified in different frames and/or changes in relationships between entities in the video may be identified. Some embodiments include more than one knowledge graph. For example, one knowledge graph that represents relationships between entities found in content and a second knowledge graph that represents general knowledge.

[0104] In an Extract Knowledge Step 440, information regarding the entities identified in Identify Entities Step 430 is extracted from the knowledge graph. This process typically includes accessing the Entity Data Record 210 associated with each identified entity and reading the data therein. The information retrieved in Extract Knowledge Step 440 is optionally used to facilitate the identification of entities in Identify Entities Step 4S0. As such, Extract Knowledge

Step 440 may be performed concurrently and/or as part of Identify Entities Step 4S0. For example, assuming that Entity Data Record 210A accessed in Extract Knowledge Step 440 indicates the presence of relationships between Entity Data Record 210A and Entity Data Records 210B and 210C. Then, if a first entity identified within the video is found to be represented by an Entity Data Record 210A, then one or more entities represented by Entity Data Records 210B and 210C may be hypothesized to also be in the video; based on the predetermined relationships. This hypothesis may make the identification of the entities represented by Entity Data Records 210B and 210C more reliable and/or more computationally efficient in Identify Entities Step 430. Video Analysis Logic 170 may, thus, navigate the knowledge graph in order to efficiently identify additional entities likely to be found in video content. In some embodiments, a lower confidence is assigned to identification of entities in the same content, if they were not previously related to each other.

[0105] The information extracted from Entity Data Records 210 can include measures of the reliability of information stored therein. For example, Entity Data Record 210C can include a measure of how reliable correctness of a relationship stored in this Entity Data Record 210C is believed to be. In another example, Entity Data Record 210C may include information indicating that a frequency at which a particular entity is found in a category of content is trending upward sharply. Further, a particular entity may be found many or fewer times in a particular content. Further, information extracted from Entity Data Record 210C may indicate changes in relationships of the entity occur in the content, these changes representing

"actions". [0106] In a Generate Summary Step 460, a summary of the content received in Receive Video

Step 420 is generated. For example, a video summary of video content may be produced. As noted elsewhere herein, the video summary can include clips of the video content, audio segments, tags (and other metadata) characterizing the video content. The selection of which parts of the video content to include in the video summary is optionally dependent on any of the information extracted from Entity Data Records 210, for example the information described in the preceding paragraph.

[0107] In specific examples, parts of the video content may be selected based on a frequency at which a particular entity or class of entities is found in the video content, on relevant changes in relationships of entities, trends in the identification of an entity in multiple pieces of content, on relevance of an entity and/or its relationships, on user searches made, on social network relationships, on an intended (target) audience, etc.

[0108] In an optional Post Summary Step 470, the video summary generated in Generate Summary Step 460 is posted to a library of media, a website, a content streaming service, and/or the like. For example, a video summary may be posted to a service that allows searching and browsing of video content. The video summary is optionally indexed using metadata included in the video summary. In some embodiments, video summaries are used as advertisements to encourage users to watch full video content. Video summaries are optionally configured to auto-play when viewed in a webpage.

[0109] In an optionally Determine Quality Step 480, a measure of the quality of a video summary is determined. This determination may be made using Correction Logic 165 and/or may be responsive to the use of the video summary. The use of the video used to measure quality can include click rates, viewing times, and/or any of the other analytics discussed herein.

The use of the video is optionally used as feedback to modify reliability estimates included in Entity Data Records 210.

[0110] In an optionally Regenerate Summary Step 490 a video summary is regenerated if the measure of quality determined in Determine Quality Step 480 is below a threshold. Regenerate Summary Step 490 optionally includes a repeat of the steps illustrated in FIG. 4 in an effort to produce a video summary of higher quality. This process may be repeated to generate improved video summaries, to improve reliability of information stored in Entity Data Records 210, to perform A/B testing, and/or to train machine learning systems discussed herein.

[0111] Several embodiments are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations are covered by the above teachings and within the scope of the appended claims without departing from the spirit and intended scope thereof. For example, while the systems and methods described herein are described with respect to video and other content accessible via a computer network, the systems and methods may be applied to other applications in which a knowledge base is beneficial. For example, language understanding, medical applications, forecasting, physical or network security, financial transactions, and/or the like. Further, while the examples provided herein include two memory stages (short-term and long-term) alternative embodiments may include one or more additional memory stages, e.g., intermediate-term memory and associated data records. It should be noted that while the examples herein are primarily focused on the processing of video, they are also applicable to other types of content such as audio, still images, printed matter, documents, books, and/or any other type of unstructured or structured data. For example, any example provided herein with respect to video may also be applied to still image, audio or textual content. The knowledge graph is language agnostic. In fact, the system can identify relationships be between entities in different languages. For example, the system may determine that "house" and "Casa" represent the same concept or entity.

[0112] The embodiments discussed herein are illustrative of the present invention. As these embodiments of the present invention are described with reference to illustrations, various modifications or adaptations of the methods and or specific structures described may become apparent to those skilled in the art. All such modifications, adaptations, or variations that rely upon the teachings of the present invention, and through which these teachings have advanced the art, are considered to be within the spirit and scope of the present invention. Hence, these descriptions and drawings should not be considered in a limiting sense, as it is understood that the present invention is in no way limited to only the embodiments illustrated.

[0113] Computing systems referred to herein can comprise an integrated circuit, a

microprocessor, a personal computer, a server, a distributed computing system, a

communication device, a network device, or the like, and various combinations of the same. A computing system may also comprise volatile and/or non-volatile memory (storage) such as random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), magnetic media, optical media, nano-media, a hard drive, a compact disk, a digital versatile disc (DVD), and/or other devices configured for storing analog or digital information, such as in a database. The various examples of logic noted above can comprise hardware, firmware, or software stored on a computer-readable medium, or combinations thereof. A computer-readable medium, as used herein, expressly excludes paper. Computer- implemented steps of the methods noted herein can comprise a set of instructions stored on a computer -readable medium that when executed cause the computing system to perform the steps. A computing system programmed to perform particular functions pursuant to instructions from program software is a special purpose computing system for performing those particular functions. Data that is manipulated by a special purpose computing system while performing those particular functions is at least electronically saved in buffers of the computing system, physically changing the special purpose computing system from one state to the next with each change to the stored data.

[0114] The logic discussed herein may include hardware, firmware and/or software stored on a non-transient computer readable medium. This logic may be implemented in an electronic device (e.g., circuit) to produce a special purpose computing system.

[0115] Following is an example of a particular embodiment in which the systems and methods described herein are used to generate "VLX-Stories" in an actual implementation:

[0116] The following describes VLX-Stories, a system under exploitation that alleviates issues in news consumption, KB (Knowledge Bases) construction and KB maintenance. The system comprises a unified online workflow of event detection, tracking, pattern extraction and semantic tagging, with the aim of building an event KB. At the same time, the new EEs

(Emerging Entities) detected by VLX-Stories populate an external KB, called VLX-KG (Knowledge Graph), with background knowledge. In VLX-Stories, events are represented by means of an ontology inspired on the journalist W's: what is happening, who is involved, where and when it took place, and the general topic under discussion. Once the news stories are represented with semantics, they become part of the linked space, with applications such as analytics, search, trend detection, recommendation, multilingual linking, amongst others.

[0117] These embodiments include a novel model for multilingual news aggregation, multi document tagging through pattern mining, online knowledge base population of EE and event knowledge base construction. It is believed that this is the first system that uses the

redundancy of aggregated news articles for a robust detection of EE and their relations in an online manner. These embodiments also include a large-scale deployment of the system for real-time applications.

[0118] VLX-Stories is being used in Vilynx (www.vilynx.com) , an Al company specialized in analyzing media contents. The core of Vilynx's technology is the Vilynx Knowledge Graph ( VLX- KG), which is used as a semantic KB for indexing media documents. This KB is constructed by merging different public KBs: Freebase, Wikidata and Wikipedia. It provides multilingual alias for over 3 million entities, and 5 million relations between entities based on properties and co occurrence. VLX-KG is required to be dynamic because it supports real time indexation of media documents. This means the VLX-KG includes the dynamic population of relations and detection of EE. To provide new knowledge to VLX-KG Vilynx uses structured data, which is updated periodically by querying the three mentioned public KB. However, media news often talks about anonymous people, who are not indexed on these public knowledge resources or that have not yet been updated. Indexing these novel entities requires extracting EE from non- structured data, e.g. news articles. VLX-Stories system, of these embodiments, provides the information and dynamics required for VLX-KG maintenance, while detecting events. Moreover, VLX-Stories extracts temporal relations and information on temporal trends, which are internally used for other products, e.g. recommendations, trends detection, etc. This information is also consumed externally by customers through API calls and a dashboard, providing a general view of what is happening in the world. FIG. 5 includes an example of a detected event displayed on a dashboard. Notice how the different W's are addressed and additional context on the news story is given through clustered articles and the semantic pattern.

[0119] Event Detection

[0120] The task of clustering news articles, falls in the field of Topic Detection and Tracking (TDT). Within TDT, a topic or event is defined as a set of news documents (articles, videos, tweets, etc.) that are strongly related by some real-world occurrence. This task implies the three research lines: topic modeling, document clustering or topic detection, and topic tracking. Most TDT solutions use probabilistic methods for topic modeling, which rely on BoW (Bag of Words) representations. Then density based (DBSCAN) or hierarchical agglomerative clustering methods are commonly used for the aggregation of documents. In this field, semantic technologies have proven its efficiency to combine features and provide multilingual news aggregation.

[0121] Event Representation

[0122] Representing events in a formal way is needed for comprehension and extraction of higher-level information. Depending on the final application, different models have been proposed, e.g. summarization, triplet representation, pattern extraction, ontology

representations, etc. However, all these approaches revolve around extracting the main information units: "what is happening", "who is involved", "where is it happening" and "when did it happen". Vilynx's approach answers these four questions: pattern extraction and knowledge base representation.

[0123] Event Semantic Pattern and Entity Disambiguation: Pattern mining approaches construct a high-level event schemes or dependency structures that can be used as relational patterns. Multimodal patterns may be extracted to describe some predefined event categories with semantic concepts in a way that can be used in knowledge base population. Similar to this approach, Vilynx searches for the pattern of semantic concepts using a pattern mining approach which explodes the redundancy between aggregated news articles. However, Vilynx's method does not constrain to a close set of events and the semantic concepts extracted are coming from an external KB. Thus, the Semantic Pattern extraction module includes a Dynamic Entity Linking (DEL) module for the entity disambiguation and newly emerging entities (EE) detection, which will populate VLX-KG. The entity disambiguation has been used in other event detection works, like, as it is useful to solve co-reference, link multilingual articles and it is a more sophisticated and standard knowledge representation. Vilynx also uses the news information to detect newly EE.

[0124] Event Ontology and Knowledge Base Construction: The construction of a KB from unstructured information in the web falls in the open Information Extraction (open IE) field. In open IE, the concepts, types or relations that the system is required to extract are not predefined. Vilynx's system does not know in advance which are the events and entities one may encounter in the news. The models try to generate new episodic knowledge on events and entities with information extracted from the web, combined with already existing structured knowledge from external KB. [0125] Ontologies are a common strategy to conceptualize and share knowledge when describing semantic models. Several event models or ontologies have been published over the past years. For example, Newsreader uses Simple Event Model (SEM), an ontology designed to be versatile in different event domains allowing cross-source interoperability. The ABC ontology is an event ontology that can describe event-related concepts such as the situation, action, agent and their relationships. NOEM also proposes an event modeling based on the 5W1H elements of a piece of news. Vilynx's proposed ontology is a simplified representation on the event.

[0126] Event Detection (E.g., Discovery Logic 140)

[0127] News Event Detection pipeline has three parts, outlined in FIG. 6. First, a collection of news articles is built by crawling RRS feeds and parsed. Afterwards, in the topic modeling block, articles are represented with BoW. Finally, in the topic tracking module, each article vector is compared with articles grouped in event clusters: if matching the event, it is assigned to the cluster; if not, it is added to a pool of articles that will be processed in the topic detection module in order to detect newly emerging events.

[0128] News Feeds Crawler (E.g., Resource Crawler 120)

[0129] News articles are collected by an RSS feeds crawler, which processes 500 news feeds every 30 minutes. The RSS feeds come from a manually generated list of 3,526 feeds from the main media sources of seven countries: United States, Australia, Spain, Canada, United

Kingdom, Portugal and Ireland. Feeds are also manually categorized in seven category groups: politics, sports, general news, lifestyle and hobbies, science and technology, business and finance, and entertainment. The feeds crawler visits each feed, crawls it, and stores in the DB each article URL, published date, title and description if provided. In a second step, whenever a new article is detected in a feed, Vilynx crawls the URL and parse the article using a customized HTML parser to extract all its text data contents and images.

[0130] Topic Modeling

[0131] Topic modeling (TM) consists on representing the abstract matter that occurs in a collection of documents. To do that Vilynx relies on a bag-of-words (BoW) representation of the articles. As news stories typically revolve around people, places and other name entities (NE), some works use mentions of NE instead of all words in the document. However, some news does not turn around NE, thus other information as common nouns (CN) or noun chunks (NC) is needed to distinguish this kind of events. Combining these three works, Vilynx used named entities, common nouns and noun chunks in the BoW representation, instead of all words in the text corpus. Vilynx calls this collection of mentions and nouns, article keywords. These keywords are extracted from the article text by a Name Entity Recognition (NER) module and Part of Speech Tagger (POST). Vilynx uses the Spacy's library and multilingual models for these tasks. For performance reasons, the constrain of an article is added to be represented for at least 8 keywords, and a maximum of 80 keywords. BoW keyword's frequencies are weighted by a slightly modified TF-IDF (term frequency - inverse document frequency), which reflects how important a word is to a document in a collection or corpus. TF-IDF is computed as the product of the term frequency (fk) by the inverse document frequency (idf_k). However, Vilynx biased the TF with a weight to give more relevance to those keywords appearing on the title (a), description (a), or that are NE (b). Finally, Vilynx applied a time factor with a linear function, which favors news documents to be assigned to more recent events. [0132] Topic Detection and Tracking

[0133] Once a new article is ingested by the system, Vilynx detects if it is associated to an already detected event (topic tracking) or it describes a new event (topic detection). For the topic tracking, Vilynx uses the k-Nearest Neighbours (k-NN) algorithm. Thus, being a_\ a new article, Vilynx associates the article with an event cluster if there are more than k articles in the cluster with a similarity higher than a given threshold Y. If the incoming article is not associated to any event cluster, Vilynx tries to build a new cluster with other articles not yet related to any event. This is the task of topic detection. The chosen clustering technique for topic detection is DBSCAN, which is an unsupervised density-based algorithm that provides robustness against the presence of noise. This method requires the estimation of two hyper-parameters: min samples, which is the minimum number of samples needed to generate a new cluster; and eps, the maximum distance allowed within its samples. Vilynx decided to fix the minsamples = 5, thus all events will be represented with at least five articles, and Vilynx optimized eps in order to have high precision without missing many events. Vilynx used cosine similarity as distance metric for both tasks.

[0134] Moreover, some decisions were made in order to compensate some of the problems of dealing with an online and large-scale deployment application with noisy Web data. In order to prevent wrong event detections because of non-standard website structures, Vilynx added two extra conditions on the cluster generation: the clustered articles need to be from at least three different news publishers, and one media publisher cannot own over 50% of the articles in cluster. Also, speed issues had to be considered to provide real time track on news events: as the amount of comparisons between articles grows quadratically with the number of articles, slowing the whole article comparison. Vilynx decided to cluster articles by country, and for those countries with more feeds, Vilynx uses a category-based comparison between articles. The category of the feed is used for this split, and in case the feed provides general news from any category, Vilynx trained a deep classifier based on an LSTM to predict the article category from its title.

[0135] Event Representation

[0136] Event representation tries to synthesize the agents, locations and actions involved in an event in a formal machine understandable way, but still natural for humans. This is achieved by extracting semantics from the event articles and structuring the knowledge. Vilynx's approach provides an event semantic pattern by combining pattern mining with a dynamic entity linking module. This module uses VLX-KG for the entity disambiguation, which will also be populated with EE found in the process. Finally, this semantic pattern is post-processed to structure the entities into our event ontology.

[0137] Event Semantic Pattern

[0138] The extraction of the event semantic pattern is achieved thanks to the two modules depicted in Figure 7: Event Pattern Extraction (EPE) and Dynamic Entity Linking (DEL). The EPE module finds the keywords pattern describing the event, and the DEL links these keywords to entities from VLX-KG, while detecting candidates of new entities which populate VLX-KG. The details follow. Pattern Mining: Data mining techniques search for patterns in data that are representative and discriminative. Vilynx defined our pattern mining task with an association rule approach, such that our pattern corresponds to a set of association rules, t* -> y , that optimize the support and confidence constrains for each event. Let n be the set of all keywords in the corpus and a transaction A be the set of keywords from a given

article, such that A

C. Given a set of m transactions belonging to the same event

Vilynx wanted to find the subset of C, say t*, which can accurately predict the

belonging to a target event y

E. The support of t * is an indicator of how often t * appears in T, and it is defined as the proportion of transactions in the transaction set T that contain the itemset t*:

[0139]

[0140] The goal is to find association rules that accurately predict the belonging to an event, given a set of keywords. Therefore, one wants to find a pattern such that if t* appears in a transaction, there is a high likelihood that y, which represents an event category, appears in that transaction as well. Vilynx defined the confidence as the likelihood that if t*

A then y

A, which can be expressed as:

[0141]

[0142] Vilynx uses the popular apriori algorithm to find patterns within the transactions. Vilynx only kept the association rules with confidence C_min greater than or equal to 0.8 and calculate the support threshold (S_min) that ensures at least 10 keywords in the rule. Finally, Vilynx selected the rule t* with more keywords associated. These keywords are the ones that will be linked to VLX-KG entities.

[0143] Dynamic Entity Linking:

[0144] The event keywords in the pattern are mapped to entities in VLX-KG. This task is called Entity Linking (EL) or disambiguation. The EL module gets entity candidates from VLX-KG for each incoming mention. Then, disambiguation is applied by scoring each candidate. An intra score and an inter-score are computed for each candidate. On one hand the intra-score is computed by using the information of the mention and entity itself, combining the following metrics: word similarity, entity usability and entity type. In the other hand, the inter-score exploits the contextual information between entities by using distances between entities in a concept embedding space. The combination of all these metrics gives a confidence score for a given mention to be disambiguated to an entity. If this score is higher than a predefined threshold, the mention and entities are linked.

[0145] However, news often refers to people that has never been mentioned before, and thus, are not in indexed in VLX-KG. In order to populate VLX-KG Vilynx added dynamics into the EL module, calling it a Dynamic Entity Linking (DEL). This module which maintains EE that refer to anonymous people as they appear on the news, and manually integrate it into VLX-KG with humans in the loop. This EE detector filters the unknown mentions, keeping only those that have been recognized as persons by the NER module and that are at least composed by two words (name and surname). The detection is highly robust because the EE come from the previously extracted event pattern, which means the entity has appeared in a high amount of articles from different publishers, in the same context, and is thus relevant when describing the event. Once an EE is detected, the system starts being capable of using it for tagging and linking documents, and it is already used to describe new events. However, it optionally requires a human validation in order to become a proper concept in the KB. This validation is helpful because sometimes the names detected are spelling variations of entities already in the KB, or mistakes from the NER module.

[0146] Multi-regional event matching: Before the final event modeling, the semantic pattern of the events detected for each country are compared and merged in case of match. To do that, one first ranks the entities in the Event Semantic Pattern by relevancy describing the event. The ranking is based on re-scoring entities based on its original keywords' appearance frequency and origin (title, description or text body). As Vilynx solved the entity disambiguation, Vilynx recompute the entity frequency taking into account co-references. Origins are taken into account by weighting the frequency of appearance by the origin. Afterwards, country-events are represented with a bag of concepts BoC where entity relevancies are the weights. Cosine similarity is computed between country-events and these events are merge into worldwide- events if its similarity is lower than a manually defined threshold.

[0147] Event Model

[0148] Both semantic and contextual event information extracted on previous steps are processed in order to represent the collected data in an ontological manner. This ontological information is stored in VLX-Stories KB, which keeps growing with the multiregional news information provided by the feeds.

[0149] Modeling the Event Ontology [0150] A requirement of our example event ontology is that it has to synthesize in a both machine and human readable way the unstructured semantic pattern extracted. Events are usually defined by its agents, locations and actions occurring and the moment when the event takes place. Journalistic articles are typically structured to answer four of the journalist SW- questions: e.g. what happened, who is involved, where and when did it happen. These questions should be addressed within the first few sentences of an article to quickly inform readers. The fifth W, Why it happened, is often addressed in opinion columns or post-event coverage. Moreover, news stories often fall into bigger topics which are composed by several events, like Brexit, Academy Awards, Olympic Games, etc. This information, if present, offers the possibility of tracking long story lines and to provide a complete context on the

development of an event in a point in time.

[0151] Considering the above mentioned 4Ws and the topic, Vilynx defined their ontology with the next properties or core classes for each event: Who, What, When, Where and Topic. These properties will be extracted from the event semantic pattern and the titles and descriptions from the event articles. Moreover, as shown in the ontology schema in fig.4, all entities in the semantic pattern, answering or not the 4Ws or topic, will be included in the ontology within the Semantic Pattern class. The event Category, e.g. sports, politics, entertainment, etc.; is also included as a property. Additional context on the event is given by the clustered articles, from which were stored the title, description, URL, news source and image, if present.

[0152] Event Properties Extraction

[0153] The Who, Where and Topic are extracted from the Event Semantic Pattern using a set of filters based on entity types. Entities in VLX-KG have a schema type associated, which denotes if the entity is a person, place, organization, etc. These types will be used to classify the entities in the pattern together with the type of metadata field from which they were extracted. For this task only the entities from the title and description are used. Moreover, the same entity relevance scores computed for multi-regional event matching will be used to pick those most relevant entities. The Who property needs to be an entity of type Person, Organization or Creative Work; the Where is a property of type Place and the topic of type Event. Vilynx defines the What of an event with a sentence (literal) summarizing the main event occurring. As news article's titles should give this information, Vilynx answers the What with the most descriptive title between the clustered articles. This is selected using a voting algorithm where each word in the title sums it's given score in the semantic entities ranking. This approach favors longer titles, which also tend to be semantically richer. To answer When, Vilynx takes the date of the first article published related to an event. Alternatively, by analyzing time expressions in the text. Finally, Vilynx completes the ontology by adding the event Category. Categories come from our pre-defined set of categories, e.g. Sport, Entertainment, Finances, Politics, etc. The categories assigned to the RSS feeds are used to extract this information. One event may belong to more than one category.

[0154] System Analytics

[0155] VLX-Stories was first deployed on July 2018 for its use in the United States. Since then, the system has been growing, adding new world regions and languages. [0156]

[0157] Table 1 contains the activation date of VLX-Stories for each country, the language it process, the number of feeds activated on each country, the average number of events detected each day and the daily number of articles processed and associated to events. Results are provided country by country and also on the worldwide event aggregation. According to this statistics, VLX-Stories grows in a speed average of almost 200 news events/day, classifying an average of over 17 thousand multilingual articles from seven different countries. The system includes the option of analyzing how an event is reported in different world regions and in different languages. Semantic disambiguation is essential for this multilingual aggregation task.

[0158] News Event Detection

[0159] Regarding news aggregator or event detection evaluation, Vilynx used a subset of the UCI-ML News Aggregator Dataset. This dataset consists of 7,231 news stories, constructed from the aggregation of news web pages collected from March 10 to August 10 of 2014. The resources are grouped into clusters that represent pages discussing the same news story or event. Events are categorized into four different categories: entertainment, health, business and technology. For each news article, its title, URL, news story id and category are given.

However, Vilynx had to discard a 13% of the events on the dataset because of the following reasons: 36% of the URL links were broken, our system could not extract enough keywords from 17% of the articles, and finally, Vilynx had to discard some of the remaining news stories because they were not represented by enough articles to be detected by our system ( each of our events needs to be represented by at least 5 news articles). The final dataset subset consists of 6,289 events constructed by 91,728 news articles. For the experiments, the DBSCAN parameters were set to eps = 0.65 and minsomples = 5.

[0160]

[0161] Table 2 presents the news event detection results on the dataset. Most of the events are correctly detected (81.58%), however a lot of articles are not associated to any event. This is reflected by the high average precision (94.57%) but a poor recall (58.548%). This is mostly because of the restrictive parameters set in the system in order to make sure that aggregated news served are correct. Quality of aggregated news is similar across news categories. The lowest quality is found in the business category, because most of the business news share financial terms which are repeated in many articles, even not related ones. Best results are for entertainment, the type of news with more named entities, which improves the representation.

[0162] Dynamic Entity Linking

[0163] The mapping of event keyword patterns to event semantic patterns includes the EL task and the EE detection. However, this tasks also depends on the quality of the NER and POS modules from Spacy. According to Spacy's documentation the quality of these modules is state of the art performance, with accuracies close to the 90%. To evaluate the quality on the EL and

EE detection two experiments were performed.

[0164] Entity Linking:

[0165] The first experiment evaluated EL of the keywords on the event pattern to semantic labels. The experiments were conducted over a corpus of 100 semantic event patterns randomly selected from the United States events, detected by our news aggregator module during the week from the 1st of January to the 7th of January 2019. The keywords from the patterns were mapped to entities from VLX-KG using the dynamic EL module. The correctness of the mapping was evaluated with TP when the semantic entity related to the keyword was correct, FP when the semantic entity related was wrong, TN when no entity was related and it is not an existing entity or it is an error from NER, and FN if no entity was mapped but there is an entity in VLX-KG for it. Results are shown in Table 3, with a total accuracy of the 86%.

However, some mentions do not disambiguate to its correct entities. This is common when finding homonym words or unknown contexts.

[0166]

[0167] Emerging Entities The capacity of detecting EE was evaluated by deleting existing entities from our VLX-KG and testing the capacity of the system to create them again. Vilynx initially built a dataset of 648 news events detected during the week from the 1st to the 7th of January 2019. The multilingual capabilities of the system were tested by choosing events from three regions with different languages: The United States, Spain and Portugal. The dataset was generated by running the Event Semantic Pattern module, removing the corresponding person's entities from VLX-KG, and extracting again the Event Semantic Pattern, expecting for the EE detector to re-generate the deleted entities. As shown in Table 4, an average of 78.86% of the deleted entities were recovered. Some of the missing entities were composed by just one word, like Rhianna or Shakira. Our system did not detect them because it assumes that person entities are always described with two words (name and surname). Other errors were caused by the similarity between entities, which are wrongly disambiguate to existing entities, e.g. when deleting the Donald Trump entity, the EL disambiguated to Donald Trump Jr. because of a perfect match between the alias and the similar usage context.

[0168]

[0169] Finally, a statistical study of the created entities and their quality is done by analyzing data between 12th December 2018 and 15th March 2019. Table 5 presents the average number of EE detected every day in each language. However, not all the detected EE become new entities in VLX-KG. After the human supervision Vilynx extracted the next metrics: 75.45% of the detected EE become new entities, 22.15% are alias of already existing entities and 9.7% are wrong candidates because of NER errors.

[0170]

[0171] Conclusions

[0172] In this exemplary embodiment an online system which aggregates news articles from RSS feeds, discovers new events, and represents this information using semantics from an external KB. The events can be represented from two perspective: with an event semantic pattern, useful for multilingual event matching; or ontologically, answering the who, what, when, where, topic and category of each event. This system has a twofold functionality: discovering EE which populate the external KB (VLX-KG); and generating an event KB which stores temporal information. This system matches unstructured text with Semantic Web resources, by exploiting natural language processing and external knowledge resources. The engine processes an average of 17,000 articles/day, and detects an average of almost 200 events/ day from seven different countries and three languages The detected events are exploited by other services such as news analysis, trends detection, monitoring, search, news context, etc.

[0173] The experimental results show an F-l score of 72.32% for event detection, and a high capacity of detecting EE of people, with an average of 78.86% of the deleted entities being detected again. EE detection statistics show that the system detects an average of almost 68 EE/day, the 75.45% of which become new entities, and 22.15% are used to populate VLX-KG entities with new alias.

Claims

CLAIMS What is claimed is:

1. A self-learning video processing system comprising:

knowledge storage configured to store a plurality of entity data records, each of the entity data records including characteristics of a respective entity, the characteristics including an identify of the entity and relationships between the entity and other entities represented by other members of the entity data records, the entity data records including short-term data records and long-term data records;

a resource crawler configured to crawl a computer network in search of content and to retrieve found content, the content including both text and image content; content analysis logic configured to identify entities within the retrieved content;

discovery logic configured to compare the identified entities to the plurality of entity data records, to assign the identified entities to corresponding data records, and to identify new entities within the identified entities, the entities and

corresponding entity data records being represented by nodes of a knowledge graph;

relationship logic configured to identify relationships between entities identified by the discovery logic, and to add the relationships to the corresponding entity data records, the relationships between entities being represented by connections between the nodes of the knowledge graph, each node corresponding to at least one of the entity data records, the entity data records comprising the short-term and long-term data records;

short-term memory logic configured manage a first subset of knowledge graph in the short-term data records, the first subset including new time-variant content; long term memory logic configured to manage a second subset of the knowledge graph in the long-term data records, the second subset including entity identities and relationships between entities, the identities and relationships being transferred from the short-term data records;

transfer logic configured to transfer the found content from the short-term data records to the long-term data records, the transfer being dependent on a measure of confidence in correct identification of the entities and/or correct identification of the relationships;_and

a microprocessor configured to execute at least the content analysis logic, the discovery logic, or the relationship logic.

2. The system of claim 1, wherein the relationship logic is further configured to determine the relationships between entities based on an ontology of relationships including predicates and constraints, the predicates including rules from which relationships can be deducted and the constraints including limitations on allowed relationships.

3. The system of claim 2 or the method of claim 24-39 or 40, wherein the ontology of

relationships in the long-term data records is different than the ontology of

relationships in the short-term data records, the ontology of the long-term data records requiring additional constraints relative to the ontology of the short-term data records.

4. The system or method of claim 3, wherein the transfer logic is configured to transfer entity identifications and relationships from the short-term records to the long-term records when the additional constraints are satisfied.

5. The system or methods of claim 3 or 4, wherein the additional constraints include verification of a relationship based on an independent source.

6. The system or method of claim 3, 4 or 5, wherein the additional constraints include a stability of the relationship determination over a predetermined time horizon.

7. The system of claim 1-5 or 6, wherein the short-term memory logic is configured to trigger resource crawler to search for information to verify a relationship or an identity of an entity, stored in short-term data records.

8. The system of claim 1-6 or 7, further comprising training logic configured to train the short term memory logic to apply the ontology to the short-term data.

9. The system of claim 1-7 or 8, wherein the training logic is further configured to change

existing predicates and constraints, and/or to identify new predicates and constraints.

10. The system or method of claim 1-8 or 9 or method of claim 24-39 or 40, wherein a time horizon of the short-term data records is shorter than a time horizon of the long-term data records.

11. The system of claim 1-9 or 10, wherein the relationship logic is configured update

relationships in short-term data records at a time period shorter than a time period at which the relationship logic is configured to update relationships in the long-term data records.

12. The system of claim 1-10 or 11 or method of claim 24-39 or 40, wherein an age of a relationship in long-term data records or short-term data records is representative of a confidence in the relationship.

13. The system of claim 1-11 or 12, further comprising correction logic configured to correct relationships and entity identifications in the long-term data records, based on 1) information stored in the short-term data records, 2) violation of constraints, 3) discovery of new relationships, and/or 4) discovery of new entities.

14. The system of claim 1-12 or 13, wherein the transfer logic is configured to transfer

relationships or entity identifications from the short-term data records to the long-term data records based on longevity of the relationships or entity identifications in the short-term data records, reliability of a source of associated content data, and/or independent verification of the relationships or entity identifications.

15. The system of claim 1-13 or 14, further comprising:

video analysis logic configured to generate a summary of a video using a machine learning system and the entity data records; and

video storage configured to store the summary of the video.

16. The system of claim 1-14 or 15, wherein the entities represented by the plurality of entity data records include a person, an organization, an event, an action and/or an activity.

17. The system of claim 1-15 or 16, wherein the relationships represented by the plurality of entity data records include a family relationship, membership in an organization, ownership, or a ranking.

18. The system of claim 1-16 or 17, wherein the characteristics of the entities represented by the plurality of entity data records include a position, an age, a gender, a location, a date/time, a size, a value or a name.

19. The system of claim 1-17 or 18, wherein the resource crawler is configured to retrieve

metadata, video or textual content from news sources.

20. The system of claim 1-18 or 19, wherein the content analysis logic is configured to identify people and objects within image content.

21. The system of claim 1-19 or 20, wherein the content analysis logic is configured to identify an entity using a plurality of visual clues in combination with a textual clue or an audio clue.

22. The system of claim 13-20 or 21, wherein the correction logic is configured to detect the errors by detecting contradictions in the entity data records.

23. The system of claim 1-21 or 22, wherein the relationship logic includes a machine learning system trained using the entity data records.

24. A method of generating a knowledge graph, the method comprising:

crawling a source of video content to identify and retrieve content;

identifying a first entity within the retrieved content, the first entity being one of a plurality of entities and being associated with entity characteristics;

identifying a relationship between the identified first entity and a second entity;

updating the knowledge graph based on the identified first entity and the relationship, the updating including storing the relationship or characteristic of the first entity in one of a plurality of short-term data records, wherein the knowledge graph comprises nodes each representing members of the plurality of entities and connections representing relationships between the plurality of entities; and validating the knowledge graph by increasing a confidence level of the relationship or of the characteristic stored in the short-term data record; and

transferring the relationship or characteristic of the first entity from the short-term data record to a long-term data record in response to the increased confidence level.

25. The method of claim 24, further comprising crawling a source of non-video content to

confirm the identification of the first entity or the relationship of the first entity.

26. The method of claim 24 or 25, further comprising identifying an incorrect relationship in the long-term data records based on conflicts between the incorrect relationship and relationships stored in multiple short-term records.

27. The method of claim 24, 25 or 26, further comprising identifying an incorrect relationship in the short-term data records based on conflicts between the incorrect relationship and relationships stored in the long-term records.

28. The method of claim 24-26 or 27, wherein identifying the relationship between the

identified first entity and the second entity is based on an ontology of relationships including predicates and constraints, the predicates including relationships from which relationships can be deducted and the constraints including limitations on allowed relationships.

29. The method of claim 28, further comprising changing existing predicates and constraints of the ontology based on conflicts between entity identifications or relationships as stored in short-term and long-term data records.

30. The method of claim 24-28 or 29, wherein an ontology of the relationships in long-term data records includes additional constraint relative to an ontology of the relationships in short-term data records, the additional constraints including verification of a relationship based on a non-video source.

31. The method of claim 24-29 or 30, wherein transferring the relationship or characteristic of the first entity from the short-term data record to a long-term data record is responsive to satisfaction of additional constraints.

32. The method of claim 24-30 or 31, wherein the step of identifying the relationship between the first entity and the second entity is based on navigating between long-term data records within the knowledge graph.

33. The method of claim 24-31 or 32, wherein updating the knowledge graph includes creating new short-term data records of the knowledge graph for entities not previously identified in long-term data records.

34. The method of claim 24-32 or 33, wherein increasing the confidence level of the

relationship or of the characteristic includes: 1) identifying evidence collaborating the relationship or characteristic and/or 2) temporal stability of the relationship or characteristic.

35. The method of claim 24-33 or 34, further comprising training a machine learning system to identify entities or relationships between entities, using the knowledge graph.

36. The method of claim 24-34 or 35, further comprising automatically identifying new types of relationships between entities based on analysis of short-term data records included in the knowledge graph.

37. A method of processing video, the method comprising:

receiving a knowledge graph comprising entity data records, each of the entity data records including characteristics of a respective entity, the characteristics including relationships between the respective entity and other entities represented by other members of the entity records, the knowledge graph including short-term data records and long-term data records the short-term data records having a time horizon shorter than a time horizon of the long-term data records;

receiving the video;

identifying entities within the video;

extracting knowledge regarding the identified entities from the knowledge graph, the knowledge including relationships of the identified entities;

identifying representative content within the video based on the identified entities and their relationships; and

generating a summary of the video including the representative content.

38. The method of claim 37, wherein the knowledge graph is generated using a method of claim

24-35 or 36.

39. The method of claim 37 or 38, wherein relationships stored in the short-term data records are subject to a different ontology than relationships stored in the long-term data records.

40. The method of claim 37, 38 or 39, wherein relationships stored in the long-term data records are subject to greater confidence requirements relative to relationships stored in the short-term data records.

41. The method of claim 24-39 or 40, further comprising detecting an emerging entity and assigning the emerging entity to a note in the knowledge graph.