DE102013205737A1 - Method for automatically extracting and organizing information from data sources in e.g. web pages, involves producing combined representation of the equivalence classes in which the order for displayable representation is displayed - Google Patents

Abstract

The method involves applying an information extraction pipeline to the data sources for processing natural language. The information over recognized entities is identified by analysis of products of the pipeline for processing natural language. The identified information is grouped in equivalence classes containing equivalent information. The displayable representation of the equivalence classes is constructed (170), and combined representation of the equivalence classes in which the order for displayable representation is displayed is produced. Independent claims are included for the following: (1) method for processing information by processing unit; and (2) non-volatile computer program memory unit.

Description

DECLARATION ON THE FEDERAL STATE PROMOTION OF RESEARCH OR DEVELOPMENT

This invention was made by government sponsorship under Contract No .: HR0011-08-C-0110 (awarded by Defense Advanced Research Project Agency) (DARPA). The government holds certain rights to this invention.

BACKGROUND

Technical area

The present disclosure relates to information technology, and more particularly, to natural language processing (NLP) systems.

Discussion of the Related Art

News agencies, bloggers, Twitter, scientific journals and meetings all produce extremely large amounts of unstructured data in textual, audio and video form. Large amounts of such unstructured data and information can be collected from multiple modalities in multiple languages, e.g. B. text, audio and video sources on the Internet. There is a need for analyzing the information and generating a compact representation of: 1) information such as actions of particular entities (eg, individuals, organizations, countries); 2) activities (such as the presidential campaign); and 3) events (eg the death of a famous person). Currently, such representations can be generated manually, but this solution is not economical and requires trained staff, especially if the information is collected from multiple languages. Such manually generated representations are generally also not scalable.

SUMMARY

Exemplary embodiments of the present disclosure provide methods for automatically extracting and organizing data so that a user can interactively explore information about entities, activities, and events.

According to exemplary embodiments, information in real time may be automatically extracted from multiple modalities and multiple languages and displayed in a navigable and compact representation of the retrieved information.

Exemplary embodiments may use natural language processing techniques to automatically analyze information from multiple sources, in multiple modalities, and in multiple languages, including web pages, blogs, forums, radio feeds, video, and television.

Exemplary embodiments may use the output of automatic machine translation systems that translate foreign language sources into the user's language, and use the output of automatic speech transcription systems that convert video and audio feeds to text.

Exemplary embodiments may use natural language processing techniques, including information extraction tools, question answering tools, and distilling tools, to automatically analyze the text generated as described above and to extract searchable and summarized information. The system can perform name-entity recognition, cross-document coreference resolution, relationship recognition, and event detection and tracking.

Exemplary embodiments may use automatic relevancy detection techniques and redundancy reduction techniques to provide the user with relevant and non-redundant information.

Exemplary embodiments may display the desired information in a compact and navigable presentation by: providing means for the user to entities, activities or specify events of interest (for example, by typing in natural language queries, by selecting entities from an automatically generated list of entities that meet custom requirements, such as entities that are prominently displayed in data sources during a user-defined time Selecting sections of text when browsing an article or selecting events or topics from views of automatically detected events / topics over a period of time.

Exemplary embodiments may automatically generate a page in response to the user request by adaptively creating a template that best suits the presumed intention of the user (for example, if the user selects a person who is a politician, the system would do so) For information on election campaigns, public appearances, explanations and history of service to the public, if the user selects a company, the system would look for breaking news about the company, for information about the company 's top representatives, for press releases, etc . search).

When the user selects an event in accordance with exemplary embodiments, the system may search for news reports about the event, for responses to the event, for results of the event, or for related events. The system can also automatically detect the entities involved in the event, such as people, countries, local governments, businesses, and organizations, and retrieve relevant information about those entities.

Exemplary embodiments may allow the user to track entities occurring on the generated page, including automatically generating a biography of a person from the available data and listing recent actions by an organization that has been automatically extracted from the available data.

Exemplary embodiments may allow the user to examine events or activities that occur on the page, including:
automatically create a timeline of the most prominent moments in a continuing event.

Exemplary embodiments may allow the user to examine the connections between entities and events (for example: providing information about the role of a company in an event, listing quotes from a person to a topic, describing the relationship between two entities, summarizing Meetings or contacts between two people and optionally retrieve images of the desired entities.

According to an exemplary embodiment, a method for automatically extracting and organizing information from a plurality of data sources by a processing unit is provided. An information extraction pipeline for processing natural language that has automatic recognition of entities is applied to the data sources. Information about detected entities is identified by analyzing results from the natural language processing pipeline. Identified information is grouped into equivalence classes that contain equivalent information. At least one displayable representation of the equivalence classes is created. An order in which the at least one displayable representation is displayed is calculated. A combined representation of the equivalence classes that follows the order in which the displayable representation is displayed is generated.

All equivalence classes can contain a collection of elements. Each element may contain a text area extracted from a document along with a specification of information about a desired entity derived from the text area.

Calculating an order in which the displayable representations are displayed may include randomly calculating the order.

Grouping the identified information into equivalence classes may include assigning each identified information to a separate equivalence class.

Grouping identified information into equivalence classes may include computing a representative instance of each equivalence class, thereby ensuring that representative instances of different classes are non-redundant in relation to each other, and ensuring that that instances of each equivalence class are redundant in relation to the representative instance of the equivalence class.

According to an exemplary embodiment, a method of processing information by a processing unit is provided. A user request is received. A user request intent is suspected from the user request to develop a suspected user intent. In response to the user request, a page is automatically generated by adaptively creating a template that conforms to the presumed user intent, using the natural processing of multiple modalities, including at least one of text, audio, and video.

If the user request selects a person who has political status, it may search for political status, search for information about at least one of campaigning, public appearances, explanations and history of the service to the public, and in response to the User request can automatically generate a page.

When the user request selects company information in at least one of recent news stories about the company, information about the company's top representatives and press releases for the company can be searched, and a page can be automatically generated in response to the user request.

When the user request selects event information about at least one of news stories, points about the event can be selected and responses to the event can be searched, and a page can be automatically generated in response to the user request.

Entities in the event and retrieved relevant information about the entities can be identified and searched.

According to an exemplary embodiment, a method for automatically extracting and organizing information is provided by a processing unit from a collection of documents having multiple modalities of multilingual information for display to a user. The collection of documents is searched to identify and incrementally retrieve documents containing audio / video files. Text from the audio / video files is transcribed to provide a textual representation. Text of a foreign language in the textual representation is translated. Desired information about at least one of entities, activities and events is extracted step by step. Extracted information is organized. Organized extracted information is converted into a navigable display that is displayable to the user.

The step-by-step extraction of desired information may include applying a natural language processing pipeline to each document to repeat all the entities recognized in the collection, and identifying relationship mentions and event mentions involving a selected entity, where an entity is at least one of a physical animated object, a physical inanimate object, something that has a proper name, something that has a measurable physical property, a legal entity and abstract concepts, a mention is a text area that refers to a Entity refers to a relationship being a connection between two entities, a relationship mention being a text area describing a relationship, and an event being a set of relationships between two or more entities involving one or more actions.

Organizing extracted information may include repeating on all of the entities identified in the collection, dividing the information extracted via the entity into selected equivalence classes containing equivalent information, repeating selecting an element in each equivalence class to all of the equivalence Class to represent all elements in the equivalence class, and to record information about the equivalence class and a typical example selected for use in generating the navigable display, wherein each equivalence class may include a collection of elements, each Element has a text area extracted from a document along with a specification of the information about the desired entity derived from the text area.

Converting organized extracted information into a navigable display that is displayable to the user may include evaluating the equivalence classes of information by: the equivalence class is assigned at least one of a highest score for the pieces of information in the class, the average score of its members, the mean score of its members, and the sum of the scores of its members, with the equivalence classes sorted in descending order of the score by an order assign a ranking in which the equivalence classes are displayed to the user, wherein for each equivalence class, creating a displayable representation of a selected entity and combining the displayable representations is repeated to produce a displayable representation of the equivalence classes.

The displayable representation may include a paragraph containing extracted information labeled with visual highlights.

In accordance with an exemplary embodiment, a non-transitory computer program storage unit is provided that includes instructions executable by a processor for interactively displaying information about entities, activities, and events from multimodal natural language sources. An information extraction module contains instruction code for downloading document content from text and audio / video data, parsing document content, recognizing mentions, corfeference, cross-document coreferencing, and extracting relationships. An information gathering module contains instruction code for extracting acquaintances, biography, and participation in events from the information extraction module. An information display module includes instruction code for displaying information from the information gathering module.

The information extraction module may further include instruction code for transcribing audio data from video sources and for translating non-English transcribed audio data into English text.

The information extraction module may include instruction code for clustering mentions under the same entity and cross-document linking the entity clusters.

The information gathering module may include instruction code for inputting a sentence and an entity and for extracting certain information about the entity from the sentence.

The information display module may include instruction code for grouping results into non-redundant sets, for sorting the sentences, for generating a brief description of each sentence, for selecting a representative snippet for each sentence, for highlighting the sections of the snippet containing information relating to a sentence to create navigation links to other pages and to generate data used to graphically display tab content.

In accordance with an exemplary embodiment, a non-transitory computer program storage unit is provided that includes instructions that are executable by a processor to automatically extract and organize information from a variety of data sources. Instruction code is provided to apply to the data sources an information extraction pipeline for processing natural language containing automatic recognition of entities. Instruction code is provided for identifying information about detected entities by analyzing natural-language processing pipeline products. Instruction code is provided for grouping identified information into equivalence classes containing equivalent information. Instruction code is provided for creating at least one displayable representation of the equivalence classes. Instruction code is provided for calculating an order in which the at least one displayable representation is displayed. Instruction code is provided for generating a combined representation of the equivalence classes that complies with the order in which the displayable representation is displayed.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Exemplary embodiments will be better understood from the following detailed description when taken in conjunction with the accompanying drawings, in which:

1 illustrates a sequence of operation steps according to an example embodiment;

2 a sequence of operation steps according to a portion of the operation steps of 1 illustrated;

3 a sequence of operation steps according to a portion of the operation steps of 2 illustrated;

4 a sequence of operation steps according to a portion of the operation steps of 1 illustrated;

5 a sequence of operation steps according to a portion of the operation steps of 1 illustrated;

6 illustrate an example entity page according to an example embodiment;

7 (a) and 7 (b) illustrate example entity pages for a message broadcast application; and

8th a program storage unit and a processor for performing a sequence of operation steps according to an exemplary embodiment.

DETAILED DESCRIPTION

Reference will now be made in more detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same elements throughout.

In the exemplary embodiments, the term "document" may refer to a textual document regardless of its format, media files including streamed audio and video data, and hybrid forms of the foregoing, such as web pages with embedded video and audio streams.

In the exemplary embodiments, the term "collection" refers to a formal or informal collection of multimedia documents, such as all documents published in a scientific journal or all of the English websites published by news agencies in Arabic-speaking countries.

In the exemplary embodiments, the term "entity" may refer to a physical animated object (eg, a person), a physical inanimate object (eg, a building), something that has a proper name (eg, Mount Everest), something that has a measurable physical property (such as a time or a period of time, a company, a place, a country), a legal entity (such as a nation) and abstract concept, such as For example, the unit of measurement and measure system of a physical property.

In the exemplary embodiments, the term "mention" refers to a text area that refers to an entity. In the case of a large structured set of documents, an entity of the collection may be associated with all of its mentions occurring in the structured set of documents, and therefore the term entity may also be used to designate such a collection.

In the exemplary embodiments, the term "relationship" refers to a connection between two entities (eg, Barack Obama is the President of the United States, Michelle Obama and Barack Obama are married). A relationship mention is a text area that explicitly describes a relationship. Therefore, two entity mentions are involved in a relational mention.

In the exemplary embodiments, the term "event" refers to a set of relationships between two or more entities in which one or more actions are involved.

1 FIG. 3 shows an overview of an exemplary embodiment that may be applicable to a collection of news documents consisting of web pages created by news agencies and containing multiple modalities of multilingual information. The multimodal collection 100 is in the step 110 in a methodical automated way (ie, crawling), identifying and incrementally retrieving the multimodal documents in the collection. A Such crawling can work incrementally, in which case it would only retrieve documents that were unavailable during previous crawl operations. Documents containing audio data, such as audio files or video files with audio data, will then be in step 120 analyzed by transcription. After step 120 is a textual representation of all multimodal documents available. Text in foreign languages is in the translation step 130 translated. The result is the textual representation 140 the multimodal collection containing documents in a desired language as well as their original version in their source language.

The textual representation 140 the collection becomes in step 150 analyzed step by step, which extracts the desired information about entities, activities and events (Information Extraction (IE)). The extracted information will be in step 160 and the organized information is converted to a navigable display form that is presented to the user.

2 FIG. 10 illustrates an IE process according to an example embodiment of step. FIG 150 , where information about entities, activities and events is extracted step by step. step 210 consists of applying a natural language processing pipeline to each document in the collection. The pipeline can be applied step-by-step as new documents are added to the collection. step 220 is repeated over all entities that have been recognized in the collection. step 220 can be applied step-by-step by repeating it only on the entities that were recognized in new documents when they are added to the collection. step 230 identifies those by step 210 Extracted relationship mentions involving the entity identified by step 220 was selected. step 240 identifies event mentions involving mentions of the entity by step 220 was selected. step 250 extracts information that belongs to the entity in the step 220 was selected.

3 shows an example of step 210 the pipeline for processing natural language as in 2 described. The text cleanup step 310 removes irrelevant characters from the text, such as formatting marks, HyperText Markup Language (HTML) tags, and the like. step 320 to break into tokens parses the cleaned up text and identifies word and sentence boundaries. step 330 For tagging the part of speech assigns each word a label that describes its grammatical function. The mention recognition step 340 identifies, in the tokenized text, the mentions of entities and the words indicating the presence of events (called event anchors). The parsing step 350 extracts the hierarchical grammatical structure of each sentence and typically presents it as a tree. Step 360 for identifying the semantic function, in which semantic relationship each of the nodes in the tree represented by step 350 were parsed to each of the verbs in the sentence. The coreference resolution step 370 identifies the entities to which the mentions belong by the mention recognition step 340 were generated. The relationship extraction step 380 Detects relationships between entity mention pairs and between entity mention and event anchors. Those skilled in the art would appreciate that these steps can be implemented using well-known statistical methods, rules, or combinations thereof.

4 FIG. 12 shows an exemplary embodiment of organizing the information about entities according to step. FIG 160 from 1 ,

step 410 is repeated on all of the entities identified in the collection. A step-by-step embodiment of step 410 consists of repeating on all of the entities identified in new documents as they are added to the collection.

step 420 shares the information extracted about the selected entity by repeating step 410 in equivalence classes that contain equivalent or redundant information. In an exemplary embodiment, each equivalence class would consist of a collection of elements, each element consisting of a text area extracted from a document along with a specification of the information about the desired entity derived from the text area. Those skilled in the art would appreciate that such equivalence classes might be mutually exclusive or overlapping, where the same element could belong to one or more equivalence classes.

step 430 is repeated on the equivalence classes given by step 420 were generated.

step 440 would select an element in the class that best represents all the elements in the class. Selection criteria selected by the selection step 440 may include, but are not limited to: selecting the most general text area that occurs in the equivalence class (for example, the area "US President Barack Obama" is more commonly called "Barack Obama, the President of the United States", and would, according to this selection criterion, be selected as the representative area for describing the relationship of "Barack Obama" to the "United States"), selecting the text area that conveys the greatest amount of information (for example, "Barack Obama is the 44th. and current President of the United States "more information about the relationship between" Barack Obama "and the" United States "as" US President Barack Obama "and would be selected as representative in accordance with this criterion), and selecting the text area with the highest score by the extraction step 150 generated when the step assigns a score to its results.

step 450 Records the information about the equivalence class and about the step 440 selected typical example, so that the information from the subsequent step 170 from 1 can be used. This in 4 The method shown may be adapted to the case in which equivalence classes may overlap, and it is still desirable to select different typical examples for different classes, for example by means of an optimization procedure having one or more of the selection criteria listed above or equivalent selection criteria would combine a measure of dissimilarity that would favor the selection of different typical examples of overlapping equivalence classes.

In an exemplary embodiment of step 420 For example, a single instance of extracted information may consist of a range (equivalent to a paragraph) from a document along with a specification of the information extracted from the scope via a desired entity. Such a specification may consist of a collection of attribute value pairs, a collection of Research Description Framework (RDF) triples, a set of relationships in a relational database, and the like. The specification may be represented using a description language such as Extensible Markup Language (XML), using the RDF representation language, using a database, and the like.

step 420 may consist of identifying groups of instances of extracted information that satisfy two conditions: the first is that each group contains at least one instance (main instance), after which all other instances in the group are redundant; the second is that main instances of separate groups in relation to each other are not redundant. This result can be achieved by using a conventional clustering algorithm or an incremental clustering algorithm.

5 FIG. 12 shows an exemplary embodiment of a method of step. FIG 170 from 1 for creating a displayable representation of the information that belongs to an entity and according to the in 4 were collected.

In step 510 the equivalence classes of information evaluated by step 420 for example, by assigning the equivalence class the highest score of pieces of information in the class. Alternatively, amounts other than the score of the equivalence class may be used, for example: the average score of their members, the mean score of their members, the sum of the score of their members, and the like. According to the in 5 described method, the score is used to rank the order in which the equivalence classes are displayed to the user.

step 520 sorts the equivalence classes in descending score order.

step 530 selects each equivalence class. For all the instances of the selected equivalence class (step 540 ) creates step 550 a displayable representation of the instance selected from the equivalence class. In an exemplary embodiment, such displayable representation consists of the paragraph containing the extracted information suitably marked with visual highlights. Such highlighting may include color to distinguish the extracted information from each other. In addition, the displayable representation could include visual cues to easily identify other entities for which an information page exists.

step 560 combines the step by step 550 generated representations to produce a displayable representation of the equivalence class. In an exemplary embodiment, this step consists of displaying the representative instance of the equivalence class and providing means for displaying the other members, for example by providing links to the representation of these members.

With reference to below 6 FIG. 2 illustrates an exemplary page describing an entity (ie, an entity page (EP)) for the individual Leon Panetta. The page is divided into a left and a right part. The two frames in the left part contain a picture and biographical information, each automatically extracted from the Internet encyclopedia Wikipedia or from another source of reliable information. The right part contains a set of tabs that organize relevant small parts (snippets) of text according to the type of information they convey. The content in each tab is the output of a series of information extraction modules, which are described in more detail below. Each tab also displays a graphical content summary of its content.

Table 1 below summarizes the information conveyed by the text snippets in each tab. entity Tab title description person affiliations Description of person's affiliations with companies, organizations, governments, authorities, etc. Explanations Report of statements made by the person on any subject Actions Description of the actions of the person Related people Discontinued acquaintances of the person locations List of places and locations that have been visited by the person elections Description of election campaigns of the person Participation in events Description of events in which the person is involved ORG and GPE Actions Description of actions of the organization or official substitutes Associated organizations Description of related organizations such as B. Branches Related people List of people belonging to the ORG / GPE Explanations Reports of statements released by the organization or submitted by proxies Table 1: Description of information contained in the tabs of the graphical

User Interface, Organized by Entity Type These snippets are selected from a collection of Information Gathering Modules (IGMs) specified in a configuration file. A typical IGM relies on a machine learning model, which will be further described below. Each IGM also assigns a relevance score to each snippet.

To compose the tab content, the snippets selected and evaluated by the IGMs are analyzed by appropriate Information Display Modules (IDMs) specified in a configuration file. IDMs group snippets with identical information for a tab in the same equivalence class. IDMs assign a score to each equivalence class and sort the classes according to the score.

For the rendering of each equivalence class, IDMs generate a title that is a brief representation of the information they convey and select a representative snippet. They highlight the sections of the representative snippet that contain the information of interest to the tab and create links to pages of other entities mentioned in the snippets. Additional phrases in the equivalence class are shown by clicking on a link labeled Additional Supporting Results .... Since news agencies frequently use the same sentences over time, such sentences are available by clicking on "Other Identical Results".

IDMs create the data used to create a visual summary of the content in the selected tab, which is shown in the frame on the right side of the upper half of the GUI. For the Related People tab, which is in 6 is illustrated, this pictorial representation is a network of relationships. For other tabs, it is a cloud of content words in the tab.

The interface is useful not only for an analyst tracking an entity in the news, but also for financial analysts tracking news about a business or for web users receiving daily news updates. The redundancy detection and systematic organization of information makes the content easily digestible.

In a message browsing application, entities in articles can be highlighted, as in 7 (a) and those entities for which an EP exists (ie, there are relevant snippets for at least one tab) are hyperlinked to the EP. Users can also access the EP by looking at a searchable list of entities in alphabetical order or by the frequency in the messages, as in 7 (b) illustrated.

8th shows an overview of an exemplary embodiment of a program storage unit 600 wherein instruction code included therein is illustrated for an IE, an IGM and IDM. A processor 700 executes the statement code stored in the program memory unit 600 is stored.

A crawler, as previously described above, may periodically retrieve new content from a set of sites with English text and Arabic text and videos in documents 610 Download. Audio data from video sources can be segmented into data blocks of 2-minute intervals and then transcribed. Arabic may be translated into English using a state of the art machine translation system. Table 2 lists the average number of documents from each modality-language pair on a daily basis. source Number of documents English text 1317 Arab. text 813 Arab. Video 843 Table 2: Number of articles downloaded daily by the crawler in different modalities

Subsequent components in the pipeline process English text documents, and the framework can be easily extended to any language for which translation and transcription systems exist.

Every new textual document 610 can through the IE pipeline 620 to be analyzed. The first step after tokenizing is parsing, followed by mention recognition. Within each document, mentions are clustered by a document-internal coreference resolution algorithm. Thus, in the appropriate context, "Washington" and "White House" are grouped under the same entity (the US), and "Leon Edward Panetta" and "Leon Panetta" are grouped under the same person (Secretary of Defense). Name and pronoun mentions are also added to the clusters. A cross-document coreference system then links the entity clusters across documents. This is done by linking each cluster to the knowledge base (KB) used in the Text Analysis Conference (TAC) task to link the entity, which is a subset of the Internet Encyclopedia Wikipedia was derived. If a match is found in the KB, the cluster is assigned to the KB handle of the match, allowing cross-document cross-referencing to entities. In addition to an exact match of titles in the KB, the cross-document coreference system uses flexible match characteristics and contextual information to match spelling variations and alternate names. The system also uniquely identifies entities with identical names. The next IE component extracts relationships between the entities in the document, such as Employee, Son, and so on. The mention recognition, coreference, and relationship extraction modules are trained on an internally annotated set of 1301 documents, which are in accordance with the Knowledge of Language Understanding ontology and Extraction (KLUE) 2 (knowledge of speech comprehension and extraction). With a development kit of 33 documents, these components each achieve an RCD of 71.6%, 83.7% and 65%. The entity linking component is unmonitored and achieves 73% accuracy on TAC 2009 person queries.

Annotated documents will then pass through the IGMs described above 630 and IDMs 640 analyzed. In its basic version, an IGM takes a sentence and an entity as input and extracts specific information about that entity from the sentence. For example, a particular IGM may recognize whether a family relationship of a particular person is mentioned in the input sentence. A partial list of IGMs and the description of the extracted content is shown in Table 1. The output of the IGMs is then analyzed by IDMs that compose the contents of the user interface tabs. These tabs either correspond to a question template from a pilot program or are derived from the relationships mentioned above. For each entity, IDMs selectively select comments generated by IGMs, group them into equivalence classes, classify the equivalence classes to rank the information displayed to the user, and aggregate the contents of the tab. The IGMs and IDMs are described in more detail below.

IGMs extract certain information belonging to a given entity from a particular sentence in two stages: First, they detect whether the snippet contains relevant information. Then they identify information nuggets.

The snippet relevance detection is based on statistical classifiers trained on three collections created as part of the pilot program: i) data provided to the pilot program teams by the Linguistic Data Consortium (LDC) during the first years of the program; ii) data provided by BAE Systems; and iii) internally annotated data. The data consists of queries and snippets with binary relevance comments. The LDC and internally annotated data was developed specifically for training and testing purposes, while the BAE data also includes annual evaluation queries, responses provided by the teams that participated in the evaluations, and official responses to the responses contain. The statistical models are maximum entropy classifiers or averaged perceptrons that are selected based on empirical performance. They use a wide range of features, including lexical, structural, syntactic, dependency, and semantic features. Table 3 summarizes the performance of the models used in Year 4 of unordered queries based on an internally generated development set. The column "TN" indicates a template number. Templates for person entities template TN P R F information T3 75.60 90.07 82.20 Actions T13 50,00 18.33 26.83 abode T17 86.11 43.66 57.94 electioneering T21 78.72 26.81 40,00 Templates for ORG / GPE entities template TN P R F information T4 71.50 90.79 80,00 Actions T14 45.83 29.73 36.07 Arrests of members T15 75.51 74,00 74.75 Place of the deputy T18 36.36 44.94 40,20 Table 3: Performance of the IGM models

IGMs analyze snippets selected from the template models and extract the information used by the IDMs to compose and visualize the results. This step is referred to as "information nugget extraction" where an information nugget is an elementary answer to a particular question. Extracted nuggets include the focus of the answer (for example, the location visited by a person), the supporting text (a subset of the snippet), a summary of the answer (taken from the snippet, or automatically generated).

Different modules extract certain types of nuggets. These modules can be simple systems based on rules or full-scale models. Each tab uses a different set of nugget extraction programs that can be easily assembled and configured to create customized versions of the system.

IDMs use the information generated by IGMs to visualize the results. This includes grouping results into non-redundant sets, sorting the sentences, generating a brief description of each sentence, selecting a representative snippet for each sentence, highlighting the sections of the snippet that contain information related to the particular tab include creating navigation hyperlinks to other pages and generating data used to graphically display the tab contents.

IGMs generate results in a common format that supports a well-structured application programming interface (API). IDMs query this API to retrieve selected IGM products. For each tab, a configuration file specifies which IGM products to use for redundancy detection. For example, the content of the "Affiliations" tab for individuals (see Table 1) is created from automatic content extraction (ACE) relationships. The configuration file instructs the IDM to use the relationship type and KB identifier of the associated entity for redundancy reduction. Therefore, if one snippet determines that Sam Palmisano was a manager at IBM and another that Sam Palmisano was a manager at International Business Machines and IBM and International Business Machines have the same KB identifier, then the Snippets identified as redundant for the purpose of the "Affiliation" tab.

The redundancy detection groups results into equivalence classes. Each class contains unique values for the IGM products specified in the configuration file. IDMs can also group classes into superclasses or split the equivalence classes according to the values of IGM products. For example, they can partition the equivalence classes according to the date of the document containing the information. The resulting document groups form the display unit. IDMs assign a score to each of these groups, for example, using a function of the score of each snippet and the number of results in the group or in the equivalence class. The groups are sorted by score and the highest rated snippet is selected as a typical example of the group. Each group is then pictorially represented in the tab as a section with a title created using selected IGM products. The score of the group is also optionally displayed. The text of the representative snippet containing proof of relevant information is highlighted in yellow. The named mentions will be linked to the appropriate page, if available, and links will be provided to various views of the document.

Each tab is associated with a graphical representation that summarizes its content and that in the top right-most section of the top half of the UI 6 will be shown. This visualization is dynamically generated by calling an application on a server when the tab is displayed.

Exemplary implementations of the system may support three different pictorial representations: a word cloud and two formats of charts showing connections between entities. A configuration file informs the IDMs which IGM products contain the information contained in to be shown in the graphic representation. This information is then formatted to conform to the API of the program that dynamically creates the visualization.

The exemplary embodiments described above may use methods known in the art for processing natural speech. A fundamental reference is Manning and Schütze's Foundations of Statistical Natural Language Processing, which covers the main techniques that make up such procedures. Creating language models based on Kookbewerb (N-Gram models) is taught in Chapter 6. Identifying the meaning of words using their context, called word-meaning clarification, is taught in Chapter 7. Recognizing the grammatical type of words in a sentence, called word-style tagging, is taught in Chapter 9. Recognizing the grammatical structure of a sentence called parsing is taught in Chapter 11. Automatic translation from a source language to a target language is taught in Chapter 13. The main information retrieval topics are taught in Chapter 15. Automatic text categorization techniques are taught in Chapter 16.

Given the significant amount of new material on the Internet, which is news that focuses on people, organizations, and geopolitical entities (GPEs), named entities make up a key aspect of news documents, and one is often interested in following Articles about a person (eg Leon Panetta), an organization (eg Apple Inc.) or a GPE (eg the United States). Exemplary embodiments described above provide a system that automatically creates summary pages for named entities from message data. The EP page describing an entity is organized into sections that answer specific questions about that entity, such as bibliography information, statements made, acquaintances, actions, and the like. Each section contains text snippets that support the facts that are automatically extracted from the collection. The redundancy reduction results in a succinct summary with only very new and useful snippets, which are displayed in the standard display. The system can be implemented using a variety of sources and displays information extracted not only from English news agency text but also from machine translated text and automatically transcribed audio data.

Although publicly available news aggregators such as Google News show the top entities in the news, clicking them typically results in a keyword search (with possibly some redundancy detection). On the other hand, the exemplary embodiments described above provide a system that organizes and summarizes the content in a systematic manner that is useful to the user. The system is not limited to a bag-of-words search, but uses a deeper NLP technology to recognize mentions of named entities, to resolve coreference (both within a document and across documents), and to mine relationships such as employees of , Spouse of, branch of etc from the text. The framework is highly scalable and can generate a summary in real time for every entity that appears in the messages. The flexible architecture of the system allows for rapid adaptation to domains other than news, such as collections of scientific documents in which the entities of interest are authors, institutions, and countries.

The approaches of the exemplary embodiments of the present disclosure may be particularly well suited for use in an electronic device or an alternative system. Accordingly, exemplary embodiments may take the form of an embodiment that combines software and hardware aspects, which may be generically referred to as "processor," "circuit," "module," or "system." Further, exemplary embodiments may take the form of a computer program product embodied in one or more computer readable media having computer readable program code stored therein.

Any combination of one or more computer-usable or computer-readable media may be used. The computer-usable or computer-readable medium may be a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include: a portable computer diskette, a hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), an optical fiber, a portable CD-ROM, an optical storage unit, a magnetic storage unit or any other suitable Combination of the above. In the context of this document, a computer readable storage medium may be any tangible medium that may contain or store a program that may be used by or in connection with an instruction execution system, apparatus, or device.

Computer program code for performing operations of the exemplary embodiments may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C ++ or the like and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (e.g. Internet using an Internet service provider).

Exemplary embodiments are described herein with reference to flowchart illustrations and / or block diagrams. It is understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be executed by computer program instructions.

The computer program instructions may be stored in a computer readable medium that may control a computer, another device that processes programmable data, or other units to function in a particular manner so that the instructions stored in the computer readable medium produce an article of manufacture including instructions that implement the function / action specified in the flowchart and / or the block or blocks of the block diagram.

It will be appreciated that the term "processor" as used herein is intended to include any processing unit, such as one that includes a central processing unit (CPU) and / or other processing circuitry (eg, digital signal processor (DSP)). , Microprocessor, etc.). It should also be understood that the term "processor" may refer to more than one processing unit, and that various elements associated with a processing unit may be shared by other processing units. The term "memory" as used herein is intended to include memory and other computer-readable media associated with a processor or CPU, such as random access memory (RAM), read-only memory (ROM), fixed storage media (e.g. A hard disk), removable storage media (e.g., a floppy disk), flash memory, etc. Further, as used herein, the term "I / O circuit" is intended to include, for example, one or more input devices (e. Keyboard, mouse, etc.) for inputting data to the processor and / or one or more output devices (eg, printer, monitor, etc.) for presenting the results associated with the processor.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment or code section having one or more executable instructions for performing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions indicated in the block may occur in a different order than indicated in the figures. For example, two blocks shown in succession may in fact be executed substantially in parallel, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams and / or in the flowchart illustration and combinations of blocks in the block diagrams and / or flowchart illustration by specific systems based on hardware performing the specified functions or acts, or combinations of special hardware and computer instructions.

Although illustrative embodiments of the present disclosure have been described herein with reference to the accompanying drawings, it is to be understood that the present disclosure is not limited to these precise embodiments, and that various other changes and modifications may be made thereto by those skilled in the art without departing from to deviate from the scope of the claims in the appendix.

Claims (25)

A method of automatically extracting and organizing information from a plurality of data sources by a processing unit, comprising: applying an information extraction pipeline to the data sources for processing natural language having automatic recognition of entities; identifying information about detected entities by analyzing natural pipeline processing products of the pipeline; grouping identified information into equivalence classes containing equivalent information; the creation of at least one displayable representation of the equivalence classes; calculating an order in which the at least one displayable representation is displayed; and generating a combined representation of the equivalence classes that complies with the order in which the displayable representation is displayed. The method of claim 1, wherein each equivalence class comprises a collection of elements, each element having a text area extracted from a document, along with a specification of information about a desired entity derived from the text area. The method of claim 1, wherein calculating an order in which the displayable representations are displayed further comprises randomly calculating the order. The method of claim 1, wherein grouping identified information into equivalence classes further comprises assigning each identified information to a separate equivalence class. The method of claim 1, wherein grouping of identified information into equivalence classes further comprises: calculating a representative instance for each equivalence class; ensuring that representative instances of different classes are not redundant in relation to one another; ensuring that instances of each equivalence class are redundant in relation to the representative instance of the equivalence class. A method of processing information by a processing unit, the method comprising: receiving a user query; suspecting a user query intent from the user query to develop a suspected user intent; and automatically generating a page in response to the user query by adaptively creating a template that conforms to the presumed user intent using the natural processing of multiple modalities comprising at least one of text, audio, and video data. The method of claim 6, further comprising: if the user query selects a person having a political status, the recognition of political status, looking for information on at least one of a campaign, public appearances, explanations and history of the service to the public, and Automatically generate a page in response to the user query. The method of claim 6, further comprising when the user query selects a company: searching for information about at least one of recent news about the company, information about top company representatives and press releases for the company; and automatically generating a page in response to the user query. The method of claim 6, further comprising when the user query selects an event: searching for information about at least one of news reports about the event and responses to the event; and Automatically generate a page in response to the user query. The method of claim 9, wherein entities in the event are identified and retrieved retrieved relevant information about the entities. A method of automatically extracting and organizing information by a processing unit from a collection of documents having multiple modalities of multi-language information for display to a user, the method comprising: searching the collection of documents to identify and step-by-step retrieve documents containing audio / video files; transcribing text from the audio / video files to provide a textual representation; translating the textual representation that is in a foreign language; incrementally extracting desired information about at least one of entities, activities and events; organizing extracted information; and converting organized extracted information into a navigable display that is displayable to the user. The method of claim 11, wherein stepwise extracting desired information comprises: applying a natural language processing pipeline to each document to repeat all entities detected in the collection; identifying relationship mentions and event mentions involving a selected entity, wherein an entity is at least one of a physical living object, a physical inanimate object, something that has a proper name, something that has a measurable physical property, a legal entity, and abstract concepts, wherein a mention is a text area related to an entity; where a relationship is a connection between two entities, wherein a relational mention is a text area describing a relationship, and where an event is a set of relationships between two or more entities involving one or more actions. The method of claim 11, wherein the organizing of extracted information comprises: repeating on all the entities identified in the collection; dividing the extracted information about the entity into selected equivalence classes containing equivalent information; repeating on all the equivalence classes; selecting an element in each equivalence class to represent all elements in the equivalence class; and recording information about the equivalence class and a typical example selected for use in generating the navigable display, wherein each equivalence class comprises a collection of elements, each element having a text area extracted from a document along with a specification of information about the desired entity derived from the text area. The method of claim 11, wherein transforming organized extracted information to a navigable display representative of the user comprises: evaluating the equivalence classes of information by assigning to the equivalence class at least one of a highest score for the pieces of information in the class, the average score of their members, the mean score of their members, and the sum of the scores of their members; sorting the equivalence classes in descending order of the score to rank orderly in which the equivalence classes are displayed to the user; repeating for each equivalent class of creating a displayable representation of a selected entity; and combining the displayable representations to produce a displayable representation of the equivalence classes. The method of claim 14, wherein the displayable representation comprises a paragraph containing extracted information characterized by visual highlights. A non-transitory computer program storage unit containing instructions executable by a processor to interactively display information about entities, activities and events from multimodal natural language sources, the non-transitory computer program storage unit having memory configured to store: an information extraction module with instruction code for downloading document content from text and audio / video data, parsing the document content, recognizing mentions, corfeference, cross-document coreferencing, and extracting relationships; an information collecting module with instruction code for extracting acquaintances, biographies and participations in events from the information extraction module; and an information display module with instruction code for displaying information from the information collecting module. The non-transitory computer program storage unit of claim 16, wherein the information extraction module further comprises instruction code for transcribing audio data from video sources and translating non-English transcribed audio data into English text. The non-transitory computer program storage device of claim 16, wherein the information extraction module further comprises instruction code for clustering mentions under a same entity and cross-document linking entity clusters. The non-transitory computer program storage device of claim 16, wherein the information collection module further comprises instruction code for entering a sentence and an entity and for extracting certain information about the entity from the sentence. The non-transitory computer program storage device of claim 16, wherein the information display module further comprises instruction code for grouping results into non-redundant sets, sorting the non-redundant sets, generating a brief description of each sentence, selecting a representative snippet for each sentence, Highlight sections of the snippet that contain information related to a specific tab, create navigation links to other pages, and generate data used to graph tab content. A non-transitory computer program storage device containing instructions executable by a processor for automatically extracting and organizing information from a plurality of data sources, the non-transitory computer program storage device having memory configured to store: Instruction code for applying an information extraction pipeline for processing natural language having automatic recognition of entities to the data sources; Instruction code for identifying information about detected entities by analyzing natural pipeline processing products of the pipeline; Instruction code for grouping identified information into equivalence classes containing equivalent information; Instruction code for creating at least one displayable representation of the equivalence classes; An instruction code for calculating an order in which the at least one displayable representation is displayed; and Instruction code for generating a combined representation of the equivalence classes that holds an order in which the displayable representation is displayed. The non-transitory computer program storage device of claim 21, wherein each equivalence class comprises a collection of elements, each element having a text area extracted from a document along with a specification of information about a desired entity derived from the text area. The non-volatile computer program storage unit according to claim 21, wherein calculating an order in which the displayable representations are displayed further comprises randomly calculating the order. The non-transitory computer program storage device of claim 21, wherein grouping of identified information into equivalence classes further comprises assigning each identified information to a separate equivalence class. The non-transitory computer program storage device of claim 21, wherein grouping of identified information into equivalence classes further comprises: calculating a representative instance for each equivalence class; ensuring that representative instances of different classes are not redundant in relation to one another; and ensuring that instances of each equivalence class are redundant in relation to the representative instance of the equivalence class.

Images