WO2019165456A1 - Système automatisé de collecte et d'extraction de données contextuelles évolutives - Google Patents

Système automatisé de collecte et d'extraction de données contextuelles évolutives Download PDF

Info

Publication number
WO2019165456A1
WO2019165456A1 PCT/US2019/019641 US2019019641W WO2019165456A1 WO 2019165456 A1 WO2019165456 A1 WO 2019165456A1 US 2019019641 W US2019019641 W US 2019019641W WO 2019165456 A1 WO2019165456 A1 WO 2019165456A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
processor
knowledge base
memory
dataset
Prior art date
Application number
PCT/US2019/019641
Other languages
English (en)
Inventor
Jason Crabtree
Andrew Sellers
Original Assignee
Fractal Industries, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/905,041 external-priority patent/US10706063B2/en
Application filed by Fractal Industries, Inc. filed Critical Fractal Industries, Inc.
Publication of WO2019165456A1 publication Critical patent/WO2019165456A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques

Definitions

  • the disclosure relates to the field of automated data extraction, particularly to extraction of data from richly formatted data across different modalities while preserving the contest of the data.
  • Richly formatted data continues to pose a problem to die field of machine learning. While unstructured text may be analyzed using natural language processing models without heeding to the overall structure of the body of text, the structure and layout of richly' formatted text may be as important to the context as the content itself. Another problem encountered when processing richly formatted data with machine learning is the multimodal nature of the data, for example, images, audio, and video may exist as supplementary data
  • a. contextual data collection and extraction accesses a plurality of richly formatted data sources, and extracts information based on user- provided context.
  • the system may be configured to work across different modalities, and may even cross-reference information from the different modalities to supplement and enrich data extracted from other modalities.
  • the system may also he configured to store extracted data as data, which may be utilized to analyze phase transitions.
  • the system may also collectively analyze the data from multiple sources in contest, and act as a Data Loss Prevention monitor to ensure that no data leakage has occurred, that may not have been obvious from singly evaluating the data
  • a system for contextual data collection and extraction comprising an extraction engine comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof said operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to: receive a context target from a user for desired information to extract, connect to a data source providing a richly formatted dataset retrieve the richly formatted dataset, process the richly formatted dataset and extract information from a plurality of linguistic modalities within the richly formatted dataset relating at least in part to the context provided by the user, and transform the extracted data into a graph and time series-based dataset; and a knowledge base construction service comprising a memory ' , a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to retrieve the graph and time series-based dataset, create a knowledge base for storing the graph and time series
  • a previously created knowledge base is retrieved to store newly extracted information.
  • data extracted from a first modality is be used to augment data in a second modality.
  • the system further comprises a proxy ' connection service comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to determine that an intermediate proxy connection is required based at leasi; in pari on connection stains in connecting to a data source.
  • the proxy connection service determines an optimal proxy network to use as the intermediate proxy connection based at least on successfully connecting to a target data source.
  • a data marker in labeled data is used by the system to identify and label previously unlaheled similar data.
  • the system further comprises a phase transition analyzer comprising a proxy connection service comprising a memory, a processor, and a plurality of programming instructions stored in the memory ' ⁇ thereof and operable on the processor thereof, wherein the programmable instructions, when operating on tire processor, cause the processor to: retrieve a knowledge base from the data store, and perform a plurality of graph analysis and transformations on the knowledge base to identify data transitions over time.
  • a previously created knowledge base is monitored for unwanted data exfiltration.
  • a method for contextual data collection and extraction comprising the steps of: (a) receiving a context target from a user for desired information to extract, usin a extraction engine; (b) connecting to a data source providing a richly formatted dataset, using the extraction engine; (c) retrieving the richly formatted dataset, using the extraction engine; (d) processing the richly formatted dataset and extract information from a plurality of linguistic: modalities within the richl formatted data.sei; relating at least in part to the context provided by the user using the extraction engine; (e) transforming the extracted data into a graph and lime series-based dataset, using the extraction engine; (f) retrieving the graph and time series-based dataset, rising a knowledge base construction service; (g) creating a knowledge fia.se for storing tire graph and time series- based dataset, usin the knowledge base construction service; and (h) storin the knowledge base in a data store for later reference, using the knowledge base construction service.
  • FIG. 1 is a diagram of an exemplary architecture of a business operating system according to an embodiment of tire invention.
  • FIG. 2 is a diagram of an exemplary architecture of a distributed system for rapid, large volume, search and retrieval of unstructured or loosely structured information found on sources such as the World Wide Web, according to a preferred embodiment of the invention.
  • FIG. 3 is a block diagram of an exemplary system for contextual data collection and extraction according to various embodiments of the invention.
  • FIG. 4 is a block diagram of an exemplary extraction engine according to various embodiments of the invention.
  • FIG. 5 is a block diagram of an exemplary system employing a system for contextual data extraction according to various embodiments of the invention.
  • Fig. 6 is a How diagram illustrating a method for knowledge base construction according to various embodiments of the invention.
  • FIG. 7 is a How diagram illustrating a method for parser testing according to various embodiments of the invention.
  • Fig. 8 is a flow diagram illustrating a method for utilizing data fingerprinting to match unlabeled data according to various embodiments of tire invention.
  • FIG. 9 is a block diagram illustrating an exemplar ) hardware architecture of a computing device used in various embodiments of the invention.
  • FIG. 10 is a block diagram illustrating an exemplary logical architecture for a client device, according to various embodiments of the invention.
  • Fig. 11 is a block diagram illustrating an exemplary itrchitectural arrangement of clients, servers, and external services, according to various embodiments of the invention.
  • Fig. 12 is another block diagram illustrating an exemplary hardware architecture of a computing device used in various embodiments of the invention.
  • Devices that; are in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise.
  • devices that are in communication with each other may communicate directly or indirectly through one or more communication means or intermediaries, logieal or physical.
  • steps may be performed simultaneously despite being described or implied as occurring non-simultaneously (e.g., because one step is described after the other step).
  • tire Illustration of a process by its depiction in a drawing does not imply that tire illustrated process is exclusive of other variations and modifications thereto, does not imply that the illustrated process or any of its steps are necessary to one or more of the aspects, and does not imply that the illustrated process is preferred.
  • steps are generally- described once per aspect but this does not mean they must occur once, or that they may- only occur once each time a process, method, or algorithm is carried out or executed. Some steps may be omitted in some aspects or some occurrences, or some steps may be executed more than once in a given aspect or occurrence.
  • Fig, l is a diagram of an exemplary architecture of a business operating system 100 according to an embodiment of the invention
  • Client access to system 105 for specific data entry system control and for interaction with system output occurs through the system’s distributed, extensible high bandwidth cloud interface 110 which uses a versatile, robust web application driven interface for both input and display of client-facing information and a data store 112 such as, but not limited to MUNGODB ;M , COUCHDB ;M , CASSANDRATM or
  • RED ISTM depending on the embodiment.
  • Directed computational graph module 155 retrieves one or more streams of data from a plurality of sources, which includes, but is not limited to, a plurality of physical sensors, network service providers, web based questionnaires turd surveys, monitoring of electronic infrastructure, crowd sourcing campaigns, and human input device information.
  • data may be split into two identical streams in a specialized pre-programmed data pipeline 155a, wherein one sub stream may be sent for batch processing and storage while the other sub-stream may be reformatted for transformation pipeline analysis, The data may be then transferred to a general transformer service module 160 for linear dates transformation as part of analysis or the decomposable transformer service module 150 for branching or iterative transformations that are pail of analysis.
  • Directed computational graph module 155 represents all data as directed graphs where the transformations are nodes and the result messages between transformations edges of die graph.
  • High-volume web crawling module 115 may use multiple server hosted
  • preprogrammed weh spiders which, while autonomously configured, may be deployed within a web scraping framework 115a of which 5CRAPYTM is an example, to identify and retrieve data of interest from web based sources that are not well tagged by conventional web crawling technology.
  • Multiple dimension time series data store module 120 may receive streaming data from a large plurality of sensors that may be of several different types.
  • Multiple dimension time series data store module 120 may also store any time series data encountered by system 100 such as, but not limited to, environmental factors at insured client infrastructure sites, component sensor readings and system logs of some or all insured client equipment, weather and catastrophic event reports for regions an insured client occupies, political communiques and/or news from regions hosting insured client infrastructure and network service information captures (such as, but not limited teg news, capital funding opportunities and financial feeds, and sales, market condition), and service related customer data.
  • Multiple dimension time series data store module 120 may accommodate irregular- and high-volume surges by dynamically allotting network bandwidth and server processing channels to process the incoming data.
  • programming wrappers 120a for languages— examples of which may include, but are not limited to, C++, PERL, PYTHON, and ERLANG M --allow r s sophisticated programming logic to be added to default functions of multidimensional time series database 120 without intimate knowledge of the core programming, greatly extending breadth of function.
  • Data retrieved by multidimensional dine series database 120 and high-volume web crawling module 115 may- be further analyzed and transformed into task-optimized results by directed computational graph 155 and associated general transformer service 160 and decomposable transformer service 150 modules.
  • graph stack service module 145 represents data in graphical form influenced by any p re-determined scripted modifications 145a and stores it in a graph- based data store 145b such as GiRAPHTM or a key-value pair type data store REIMSTM, or RIAKTM, among others, any of which are suitable for storing graph-based information.
  • Results of the transformative analysis process may then be combined with further client directives, additional business rules and practices relevant to the analysis and situational information external to the data already available in automated planning service module 130, which also runs powerful information theory-based predictive statistics functions and machine learning algorithms 130a to allow future trends and outcomes to be rapidly forecast based upon the current system derived results and choosing each a plurality of possible business decisions. Then, using all or most available data, automated planning service module 130 may propose business decisions most likely to result in favorable business outcomes with a usably high level of certainty. Closely related to the automated planning service module 130 in die use of system-derived results in conjunction with possible externally supplied addition;!!
  • action outcome simulation module 125 with a discrete event simulator rogramming module I25a coupled with an end user-facing observation and state estimation service 140, which is highly scriptable 140b as circumstances require and has a game engine 140a to more realistically stage possible outcomes of business decisions under consideration, allows business decision makers to investigate tire probable outcomes of choosing one pending course of action over another based upon analysis of the current available data,
  • Fig. 2 is a diagram of an exemplary architecture of a distributed system 200 for rapid, large volume, search and retrieval of unstructured or loosely structured information found on sources such as the World Wide Web, according to a preferred embodiment of the invention.
  • scrape campaign requests which are comprised of a plurality of scrape agent (spider) configuration parameters as well as scrape campaign control directives, may be entered from a connected computer terminal 220 or by terminal-like commands issued by external software applications 210 using a built in command line interface 230.
  • similar scrape campaign requests may enter the system through an HTTP REST-based API usingJSON-compliant instructions 240.
  • Scrape campaign parameters enter a distributed scrape campaign controller module 250, where diey are formalized and stored in a scrape request data store 270 as one or more scrape campaign- related spider configurations 272, 273, 274 and associated scrape campaign control directives 271.
  • Scrape campaigns remain persistently stored until a command to run one or more of them is received through command line interface 230 or HTTP-based API 240, at which time request parameters 271, 272 etc. for a campaign are retrieved by distributed scrape campaign controller module 250 from scrape request data store 270.
  • Persistent storage of scrape campaign request parameters also allows ihe same scrape campaign to be run multiple times and used as a stinting point for design of similar scrape campaigns.
  • distributed scrape campaign controller module 250 Upon receipt of a command to run a specific scrape campaign and retrieval of that scrape campaign’s configuration and control parameters, coordinates tire scrape campaign in regards to tire number of spiders 282, 283, 284 to be used, and ihe number of distributed scrape servers 280, 290, 2100 to be used based upon the control directives for that campaign. Distributed scrape campaign controller module 250 then sends appropriate instructions to scrape servers 280, 290, 2100 to initiate and run ihe requested scrape campaign. If there are multiple spider configurations present in a scrape campaign request so as to cause the scraping of multiple web pages or sites, how many spiders to assign to each page and the priority each spider type has for scrape server
  • scrape controller module 250 which directs the scrape servers 280, 290, 2100 accordingly to initiate and run the requested mulii age or multistie scrape campaign.
  • scrape controller module 281, 291, 2101 of each scrape server 280, 290, 2110 executes the required scrapes.
  • Scrape controller module 280 290, 2110 hosts the programming for the spiders into which if loads scrape campaign spider configuration parameters sent to scrape server 280, 290, 2110 from distributed scrape campaign controller module 250 using the co sent scrape campaign control directives to determine the number of spider instances 282, 283, 284 to create and the resource usage priority' ⁇ each spider is given on the server.
  • a single scrape server 290; 2110 may execute spiders scraping different web targets 292, 293, 294; 23.02, 2103, 2104 and the spiders scraping a single web target 282, 283, 293; 292, 2102; 294, 2103 may be distributed across multiple servers 280; 290; 2100 Scrape controller module 281, 291, 2101 of each scrape server 280, 290, 2100 monitors the progress and operational status of the spiders it has executed and returns that information back to distributed scrape controller module 250.
  • Both the progress and operational data. which may be acquired from a request data store monitor 260, are stored as log data 275 in scrape request store 270 and is made available to the authors of the scrape campaign during its operation, which may result in directives being issued that change one or more aspects of the scrape campaign.
  • the invention is designed to allow such mid-campaign parameter changes without downtime or loss of collected, intermediate, data.
  • Results of the scrapes returned to scrape controller module 281, 291, 2100 by individual spiders 282, 283, 284, 292, 293, 294 ⁇ , 2102, 2103, 2104 are sent to persistence service server 2120, which aggregates tire data from individual scrape server spiders 282, 283, 284, 292, 293, 294, 2102, 2103, 2104, and performs any transformations pre-designed by the authors of the scrape campaign prior to outputting the data in a format determined by the authors of the campaign. This may involve sending the output to external software applications 2130 for farther processing. The data may also be processed for storage by persistence service server 2120 and sent to a persistence data store for more permanent archival.
  • the core distributed scrape campaign system distributes load across a pool of scrape servers, coordinates the number of spiders employed within a scrape campaign, and prioritizes allotment of scrape server resources among spiders, it does nest internally manage or control spider web page and link follow restrictions, crawling frequencies, and so forth. Individual spiders must implement suitable controls and crawling orchestration (which is external to the distributed scrape campaign system). All of these considerations are part of the scrape campaign spider configuration parameters that are received from the authors of scrape campaigns 210, 220 by distributed scrape campaign controller module 250, This is done to give the authors of the scrape maximal flexibility in the behavior of the spiders during a scrape campaign while allowing the use of a robust yet easily deployed spider programming interface.
  • FIG. 3 is a block diagram of an exemplary system 300 for contextual data collection and extraction according to various embodiments of the invention.
  • System 300 may comprise graph stack service 145, directed computational graph (DCG) module 155, an extraction engine 310, knowledge base construction (KBC) service 320, a proxy connection service 321, multidimensional time-series data store (MDTSDS) 120 for collecting and storing graph and time-series, and a data store 330.
  • DCG directed computational graph
  • KBC knowledge base construction
  • MDTSDS multidimensional time-series data store
  • Extraction engine 310 may be configured to use processes of business operatin system 100, such as connector module 135, web crawler 1.15, and multidimensional time series data store 120 to conned to data sources to extract data, which may be richly formatted data, structured data, unstructured data, and the like. Extraction engine 310 may be configured to not only work across different modalities of data and preserve context across the different modalities, but data extracted from the various modalities may be used to augment data from one modality to another. Extracted data from the same modality from different sources may also be able to augment one another. During the extraction process, extraction engine 310 may take into consideration user-provided context. The context may then be used by extraction engine 310 to refine the types of the data that is extracted. Once data has been extracted, tire data may be subjected to external feedback as a means lor quality assurance for the extracted data. Some sources are listed below in Fig. 5,
  • extraction engine 310 Another capability of extraction engine 310 is tagging extracted data with relevant timestamp data and store the data as time-series data. This may be useful for classifying dates in phases so that transitions over time may be captured using graph edge analysis. This may be useful, for example, for tracking development in expert judgement in particular fields overtime, as well as let interested parlies explore data from specific time periods,
  • extraction engine 310 may comprise an image analysis engine 310a, an audio analysis engine 310b, a video analysis engine 310c, a text analysis engine 310d, and data formatting service 310e.
  • Tmage analysis engine 310a may be configured to analyze images using image recognition models. Images may include, but is not limited to, pictures, charts and graphs, tables, and the like. Data from images may be extracted and may then processed by data formatting service 31Ge, so that the data may conform to any preset standards for usage in a knowledge base.
  • Audio analysis engine 310b may be configured to use audio analysis models to process audio data, for example, performing general speee.h-io-te.xi operations or to analyze tonal cues in voice recordings. This may provide additional insight by cross referencing the tones and inflections with presented facts, for example, it may reveal whether or not certain statements can he considered truthful or not Data extracted from audio may then be processed by data formatting service 310e, so that the data may conform to any preset standards for usage in a knowledge base.
  • Video analysis engine 310c may be configured to use video analysis models to process videos, and capture information from videos. For example, analyzing body language to glean concealed information or perform lip-reading analysis as a means to increase accuracy of speech didai.ion.
  • Text analysis engine 310d may be configured to use natural language processing (NLP) models to analyze text-based data, which may include, system logs, news articles, blog posts, tabular data, and the like. Text analysis engine 310d may contain an extensible collection of parsers that may be utilized to parse text data in a known format.
  • NLP natural language processing
  • Data formatting sendee 310e may be configured to user graph stack service 145 to clean and formalize data gathered by other processes of extraction engine 310 and convert the data into a graph representation to ensure that the data conforms to any preset standards for compatibility with knowledge bases that are in use by system 300
  • Knowledge base construction service 320 may be configured to assemble and maintain extracted and processed data.
  • Knowledge bases may be divided in context collections provided by user, for example, a knowledge base may be based on a particular company, a technical held of interest, financial data, and the like.
  • KBC service 320 may update existing knowledge bases with the newly extracted data, or create a new knowledge base if a suitable knowledge base doesn’t exit.
  • Knowledge bases may be stored in system 300 in data store 330. in some embodiments, knowledge bases may also be actively monitored and evaluated, for example, by using DCG module 155 with the associated transformer modules 150, 160 and observation and state estimation service 140, to locate information originating from multiple sources that, when evaluated collectively, are valuable.
  • filtration may be unearthed.
  • Personally Identifiable Information (PIT) encodings such as name, phone number, and address collectively may constitute a Data Loss Prevention breach under some jurisdictions, and a Universal Unique Identifier (IIIJID) associated with each of these three pieces of info are sent separat ely to the same recipient, this approach will identify that all three were sent based upon enrichment and ongoing analysis of the knowledge base,
  • Proxy connection service 321 may be configured to automatically connect to a proxy network to facilitate anonymous connections to data sources. This may be useful in cases, for instance, in which a particular data source aggressively blocks web crawlers from accessing pages, when bypassing a firewall is required, to concetti one’s true identity, and the like.
  • Proxy connection service 321 may automatically determine when a proxy ⁇ connection is required and may automatically determine optimal proxy networks to use.
  • Plrase transition analyzer 322 may be configured to use DCG module 155 along with the associated transformer modules 155 to analyze graph and time-series data for shifts and changes in data over time, for example, changes in lingo in a particular field or development that changes understaoding of a subject overtime. This may provide useful, for instance, when considering data sources from particular time periods, especially if the field of interest has undergone significant change over time.
  • Fig. 5 is a block diagram of an exemplary system 500 employing a system for contextual data extraction 300 according to various embodiments of the invention.
  • System 300 may connect to a pluralit of data sources 505a ⁇ /i, which may comprise richly formatted data sources, structured data somc.es, unstructured data sources, system logs, repositories, and the like.
  • data sources 5G5a-,o may require connection via one or more proxy networks 515a-i? which may comprise botnets 515a, TOR networks ,515b, residential networks 515c, data centers 515d, and mobile devices 515e.
  • System 300 may also take into consideration feedback from a piundity of feedback sources 510a ⁇ 3 ⁇ 4 which may include, crowd-sourced data 510a, expert judgement 510b, generative adversarial networks (GAN’s) 510c, transfer learning 51 Od, and the like as a means for quality ’ assurance for extracted data.
  • feedback sources 510a ⁇ 3 ⁇ 4 may include, crowd-sourced data 510a, expert judgement 510b, generative adversarial networks (GAN’s) 510c, transfer learning 51 Od, and the like as a means for quality ’ assurance for extracted data.
  • GAN generative adversarial networks
  • Fig. 6 is a How diagram illustrating a method 600 for knowledge base construction according to various embodiments of the invention.
  • system 300 retrieves richly formatted data from a plurality of sources, which may include, local storage, cloud storage, web pages, and the like.
  • sources which may include, local storage, cloud storage, web pages, and the like.
  • a user may provide the system with context to refine types of data that are extracted, for instance, financial data for a particular company ’ .
  • the system analyzes the richly formatted data using the various functions of extraction engine 310, extracts the relevant information, and formalizes the data.
  • the system may receive feedback regarding the data from a variety of sources, a few of which are disclosed above in Fig. 5,
  • the data is labeled, and stored in an appropriate knowledge base. If no knowledge base exists, the system may create a new knowledge base to store the data.
  • Fig, 7 is a flow diagram illustrating a method 700 for parser testing according to various embodiments of the invention.
  • system receives a log from services commonly used in die art, for example, syste logs fro APACHE, NGINX M, CISCO ASA, SQUID, BLUECOAT, PALOALTONETW ORKS, and the like.
  • the log is tested with a first log parser. If the log is of substantial size a portion of the log may be used to test the parser.
  • the system determines that the parsing has failed with the first parser, for example, if the output is incorrect, the system checks to see if there are anymore parsers to test at decision block 704. If there are more parsers, a second parser may be selected at step 705. The testing process stalls again with the second parser. If there are no more parsers at decision block 704, a list of parsers that passed Is presented to the user at step 708, and the user may pick one or more parsers for the system to use.
  • the system may dynamically arrange the parsers in a particular order, for example, based on a score automatically generated by the system for the quality ' and usefulness the output is determined to be by the system in some embodiments, a threshold may be preset so that If a qualify score surpasses the threshold, the system automatically parsers to parse die logs.
  • the system further evaluates the output at step 706 to determine the quality and usefulness of the output, The output may also be scored at this step.
  • the present parser choice is stored on a list to be presented to the user at step 708, along with a store.
  • Fig. 8 is a flow diagram illustrating a method 800 for utilizing data fingerprinting to match unlabeied data according to various embodiments of the invention.
  • system 300 extracts data from a source of richly formatted data.
  • the system determines which knowledge base to the store the extracted data.
  • the system discovers a knowledge base with « «labeled and incomplete data that contains identical data markers.
  • the previously « «labeled data is labeled, and may be moved to an appropriate knowledge base.
  • previously unlabeled financial information containing incomplete stock pricing may be matched nidi historical stocks pricing of a known company during known time-periods.
  • the system may automatically associate tire two bodies of data during graph analysis of the datasets, The system may then populate the incomplete dataset with any additional information, label the data, fixed any mislabeling, and the like.
  • the techniques disclosed herein may be implemented on hardware or a combination of software and hardware. For example, they may be implemented in an operating system kernel, in a separate user process, in a library package bound into network applications, on a specially constructed machine, on an application-specific integrated circuit (ASIC), or on a network interface card.
  • ASIC application-specific integrated circuit
  • Sofivvare/hardware hybrid implementations of at least some of the aspects disclosed herein may be implemented on a programmable network-resident machine (which should be understood to include intermittently connected network-aware machines) selectively activated or reconfigured by a computer program stored in memory.
  • Such network devices may have multiple network interfaces that may be configured or designed to utilize different types of network communication protocols.
  • a general architecture for some of these machines may he described herein in order to illustrate one or more exemplary means by which a given unit of functionality may be implemented.
  • At least some of the features or functionalities of tire various aspects disclosed herein may be implemented on one or more general-purpose computers associated with one or more networks, such as for example an end-user computer system, a client computer, a network server or other server system, a mobile computing device (e.g., tablet computing device, mobile phone, smartphone, laptop, or oilier appropriate computing device), a consumer electronic device, a music player, or any other suitable electronic device, router, switch, or other suitable device, or any combination thereof.
  • at least some of the features or functionalities of the various aspects disclosed herein may be implemented in one or more virtualized computing environments (e.g., network computing clouds, virtual mach ines hosted on one or more physical computing machines, or other appropriate virtual environments),
  • FIG. 9 there is shown a block diagram depicting an exemplary computing device 10 suitable for implementing at least a portion of the features or functionalities disclosed herein.
  • Computing device 10 may be, for example, any one of tire computing machines listed in the previous paragraph, or indeed any other electronic device capable of executing software- or hardware- based instructions according to one or more programs stored in memory.
  • Computing device 10 may be configured to communicate with a plurality of oilier computing devices, such as clients or servers, over communications networks such as a wide area network a metropolitan area network, a local area network, a wireless network, the Internet, or any other network, using known protocols for such communication, whether wireless or wired.
  • communications networks such as a wide area network a metropolitan area network, a local area network, a wireless network, the Internet, or any other network, using known protocols for such communication, whether wireless or wired.
  • computing device 10 indudes one or more central processing units (CPU) 12, one or more interfaces 15, and one or more busses 14 (such as a peripheral component interconnect (PCI) bus).
  • CPU 12 may be responsible for implementing specific functions associated with the functions of a specifically configured computing device or machine.
  • a computing device 10 may be configured or designed to function as a server system utilizing CPU 12, local memory 11 and/or remote memory' ⁇ 16, and interface(s) 15.
  • CPU 12 may be caused to perform one or more of the different types of functions and/or operations under the control of software modules or components, which for example, may include an operating system and any appropriate applications software, drivers, and the Mice.
  • CPU 12 may include one or more processors 13 such as, for example, a processor from one of the Intel, ARM, Qualcomm, and AMD families of microprocessors.
  • processors 13 may include specially designed hardware such as application-specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEFROMs), field-programmable gate arrays (FPGAs), and so forth, for controlling operations of computing device 10.
  • ASICs application-specific integrated circuits
  • EEFROMs electrically erasable programmable read-only memories
  • FPGAs field-programmable gate arrays
  • a local memory 11 such as non volatile random access memory (RAM) and/or read-only memory’ (ROM), including for example one or more levels of cached memory
  • RAM non volatile random access memory
  • ROM read-only memory
  • Memory ⁇ ’ 11 may be used for a variety of purposes such as, for example, caching and/or storing data, programming instructions, and the like. It should be farther appreciated that CPU 12 may be one of a variety of system-on-a-chip (SOC) type hardware that may include additional hardware such as memory or graphics processing chips, such as a QUALCOMM
  • SNAPDRAGON 11 ' 1 or SAMSUNG EXYNOSTM CPU as are becoming increasingly common in die art, such as for use in mobile devices or integrated devices.
  • interfaces 15 are provided as network interface cards (NICs). Generally, NICs control the sending and receiving of data packets over a computer network; other types of interfaces 15 may for example support other peripherals used with computing device 10.
  • NICs network interface cards
  • the interfaces th t may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, graphics interfaces, and the like.
  • interfaces may be provided such as, for example, universal serial bus (USB), Serial, Ethernet, FIREWIRETM, THUNDERBOLTTM, PCI, parallel, radio frequency (RF), BLUETOOTHTM, near-field communications (e.g., using near-field magnetics), 802.11 (WiFi), frame relay, TCP/IP, ISDN, fast Ethernet interlaces, Gigabit Ethernet interfaces, Serial ATA (SAT A) or external SATA (ESATA) interfaces, high- definition multimedia interface (HDMI) digital visual interface (DVI), analog or digital audio interfaces, asynchronous transfer mode (ATM) interfaces, high-speed serial interface (PIS SI) interfaces, Point of Sale (POS) interfaces, fiber data distributed interfaces (FDDIs), and the like.
  • USB universal serial bus
  • RF radio frequency
  • BLUETOOTHTM near-field communications
  • near-field communications e.g., using near-field magnetics
  • WiFi WiFi
  • frame relay e.g., WiFi
  • such interfaces 15 may include physical ports appropriate for communication with appropriate media. In some cases, they may' also include an independent processor (such as a dedicated audio or video processor, as is common in the art for high-fidelity A/V hardware interfaces) and, in some instances, volatile and/or non-volatile memory (e.g., RAM).
  • an independent processor such as a dedicated audio or video processor, as is common in the art for high-fidelity A/V hardware interfaces
  • volatile and/or non-volatile memory e.g., RAM
  • FIGs 9 illustrates one specific architecture for a computing device 10 for implementing one or more of the aspects described herein, it is by no means the only device architecture on which at ieast a portion of the features and techniques described herein may be impiemented.
  • architectures having one or any number of processors 13 may be used, and su h processors 13 may be present in a single device or distributed among any number of devices.
  • a single processor 13 handles communications as well as routing computations, while in other aspects a separate dedicated communications processor may be provided.
  • different types of features or functionalities may be implemented in a system according to the aspect that includes a client device (such as a tablet device or smartphone running client software) and server systems (such as a server system described in more detail below).
  • the sy tem of an aspect may employ' one or more memories or memory modules (such as, for example, remote memory' ⁇ block 16 and local memory 11) configured to store data, program instructions for the general-purpose network operations, or other infonnation relating to the functionality of the aspects described herein (or any combinations of the above).
  • Program instructions may control execution of or comprise an operating system and/or one or more applications, for example.
  • Memory ' ⁇ 16 or memories 1 1 , 16 may also be configured to store data structures configuration data, encryption data, historical system operations information, or any other specific or generic non-program information described herein.
  • At least some network device aspects may include nontransitory machine-readable storage media, which, for example, may be configured or designed to store program instructions, state information, and the like for petforming various operations described herein.
  • nontransitory machine- readable storage media include, but are not limited to, magnetic medi such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM), flash memory (as is common in mobile devices and integrated systems), solid state drives (SSD) and hybrid SSD” storage drives that may combine physical components of solid state and hard disk drives in a single hardware device (as are becoming increasingly common in the art with regard to personal computers), merorisior memory, random access memor (RAM), and the like.
  • ROM read-only memory
  • flash memory as is common in mobile devices and integrated systems
  • SSD solid state drives
  • hybrid SSD hybrid SSD
  • such storage means may be integral and non-removable (such as RAM hardware modules that may be soldered onto a motherboard or otherwise integrated into an electronic device), or they may be removable such as swappable flash memory modules (such as“thumb drives/' or other removable media designed for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removable optical storage discs, or other such removable media, and that such integral and removable storage media may be utilized interchangeably.
  • Examples of program instructions include both object code, such as may be produced by a compiler, machine code, such as may be produced by an assembler or a linker, byte code, such as may be generated by for example a JAVATM compiler and may be executed using a Java virtual machine or
  • FIG. 10 there is shown a block diagram depicting a typical exemplary architecture of one or more aspects or components thereof on a standalone computing system.
  • Computing device 20 includes processors 21 that may run software that: carry out one or more functions or applications of aspects, such as for example a client application 24.
  • Processors 21 may carry out computing instructions under control of an operating system 22 such as, for example, a version of MICROSOFT WINDOWSTM operating system, APPLE macOSTM or iOSTM operating systems, some variety of the Linux operating system,
  • one or more shared services 23 may be operable in system 20, and may be useful for providing common services to client applications 24.
  • Services 23 may for example be WINDOWSTM services, user-space common services in a Linux environment, or any other type of common service architecture used with operating system 21.
  • Input devices 28 may be of any type suitable for receiving user input, including for example a keyboard, touchscreen, microphone (for example, for voice input), mouse, touchpad, trackball, or any combination thereof.
  • Output devices 27 may be of any type suitable for providing output to one or more users, whether remote or local to system 20, and may/ include for example one or more screens for visual output, speakers, printers, or any combination thereof.
  • Memory 25 may be random-access memory ⁇ having any structure and architecture known in the art, for use by processors 21, for example to run software.
  • Storage devices 26 may he any magnetic, optical, mechanical, memristor, or electrical storage device for storage of dat in digital form (such as those described above, referring to Fig. 9), Examples of storage devices 26 include flash memory, magnetic hard drive, CD-ROM, and/or the like.
  • systems may be implemented on a distributed computing network, such as one having any number of clients and/or servers.
  • Fig. I I there is shown a block diagram depicting an exemplary architecture 30 for Implementing at least a portion of a system according to one aspect on a distributed computin network.
  • any ⁇ number of clients 33 may be provided.
  • Each client 33 may run software for implementing client-side portions of a system; clients may comprise a system 20 such a that illustrated in Fig. 10.
  • arty number of servers 32 may be provided for handling requests received from one or more clients 33.
  • Clients 33 and servers 32 may communicate with one another via one or more electronic networks 31, which may be in various aspects any of the Internet, a wide area network, a mobile telephony network (such as CDMA or GSM cellular networks), a wireless network (such as WiFi, WiMAX, LTE, and so forth), or a local area network (or indeed any network topology known in the ait; the aspect does not prefer any one network topology' over any other).
  • Networks 31 nray be implemented using any known network protocols, including for example wired and/or wireless protocols.
  • servers 32 may call external services 37 when needed to obtain additional information, or to refer to additional data concerning a particular call. Communications with external services 37 may take place, for example, via one or more networks 31.
  • external services 37 may comprise web-enabled services or functionality ⁇ related to or installed on the hardware device itself. For example, in one aspect where client applications 24 are implemented on a smartphone or other electronic device, client applications 24 may obtain information stored in a server system 32 in the cloud or on an external service 37 deployed on one or more of a particular enterprise’s or riser’ premises,
  • clients 33 or servers 32 may make use of one or more specialized services or appliances that may be deployed locally or remotely across one or more networks 31.
  • one or more databases 34 may be used or referred to by one or more aspects. It should be understood by one having ordinary skill in the art that databases 34 may be arranged in a wade variety of architectures and using a wide variety of data access and manipulation means.
  • one or more databases 34 may comprise a relational database system using a structured query language (SQL), while others may comprise an alternative data storage technology such as those referred to in the art as“NoSQL” (for example, HADOOP CASSANDRATM, GOOGLE BIGTABLETM, and so forth).
  • SQL structured query language
  • variant database architectures such as column-oriented databases, in-memory databases, clustered databases, distributed databases, or even flat file data repositories may be used accordin to the aspect, It will be appreciated by one havin ordinary skill in the art that any combination of known or future database technologies may ⁇ be used as appropriate, unless a specific database technology or a specific arrangement of components is specified for a. particular aspect described herein. Moreover, it shook! he appreciated that the term“database * as used herein may refer to a physical database machine, a cluster of machines acting as a single database system, or a logical database within an overall database management system.
  • security systems 36 and configuration systems 35 may make use of one or more security systems 36 and configuration systems 35, Security surd configuration management are common information technology (IT) and web functions, and some amount of each are generally associated with any IT or web systems. It should be understood by one having ordinary skill in the art that any configuration or security subsystems known in the art now or in the future may be used in conjunction with aspects without limitation, unless a specific security 36 or configuration system 3,5 or approach is specifically required by the description of any specific aspect.
  • IT information technology
  • Fig. 12 shores an exemplary overview' of a computer system 40 as may be used in any of the various locations throughout the system. It is exemplary of any computer that may execute code to process data, Various modifications and changes may be made to computer system 40 without departing from the broader scope of the system and method disclosed herein.
  • Central processor unit (CPU) 41 is connected to bus 42, to which bus is also connected memory 43, nonvolatile memory 44, display 47, input/ouiput (I/O) unit 48, and network interface card (NIC) 53.
  • I/O unit 48 may, typically, be connected to keyboard 49, pointing device 50, hard disk 52, and real-time clock 51.
  • NIC 53 connects to network 54, which may be tire Internet or a local network, which local network may or may not have connections to the Internet. Also shown as pari: of system 40 is power supply unit 45 connected, in this example, to a main alternating current (AC) supply 46. Not shown are batteries that could be present, and many other devices and modifications that are well known but are not applicable to the specific novel functions of tire current system and method disclosed herein.
  • AC alternating current
  • functionality for implementing systems or methods of various aspects may be distributed among any number of client and/or server components.
  • various software modules may be implemented for performing various functions in connection with the system of any particular aspect, and such modules may he variouslyimplemented to run on server and/or client components.

Abstract

L'invention concerne un système de collecte et d'extraction de données contextuelles, comprenant un moteur d'extraction configuré pour recevoir un contexte d'un utilisateur pour une information souhaitée à extraire, se connecter à une source de données fournissant un ensemble de données richement formaté, récupérer l'ensemble de données richement formaté, traiter l'ensemble de données richement formaté et extraire une information d'une pluralité de modalités linguistiques à l'intérieur de l'ensemble de données richement formaté, et transformer les données extraites en un ensemble de données extrait; et un service de construction de base de connaissances configuré pour récupérer l'ensemble de données extrait; et un service de construction base de connaissances pour récupérer l'ensemble de données extrait, et stocker la base de connaissances dans un magasin de données.
PCT/US2019/019641 2018-02-26 2019-02-26 Système automatisé de collecte et d'extraction de données contextuelles évolutives WO2019165456A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/905,041 US10706063B2 (en) 2015-10-28 2018-02-26 Automated scalable contextual data collection and extraction system
US15/905,041 2018-02-26

Publications (1)

Publication Number Publication Date
WO2019165456A1 true WO2019165456A1 (fr) 2019-08-29

Family

ID=67688530

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/019641 WO2019165456A1 (fr) 2018-02-26 2019-02-26 Système automatisé de collecte et d'extraction de données contextuelles évolutives

Country Status (1)

Country Link
WO (1) WO2019165456A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112231376A (zh) * 2020-09-25 2021-01-15 中国建设银行股份有限公司 一种用于离线数据采集的方法和装置
CN112463984A (zh) * 2020-12-04 2021-03-09 北京明略软件系统有限公司 数据库模式扩展方法、装置、设备及计算机可读介质
CN112612866A (zh) * 2020-12-29 2021-04-06 北京奇艺世纪科技有限公司 知识库文本同步方法、装置、电子设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060218192A1 (en) * 2004-08-31 2006-09-28 Gopalakrishnan Kumar C Method and System for Providing Information Services Related to Multimodal Inputs
US20110119613A1 (en) * 2007-06-04 2011-05-19 Jin Zhu Method, apparatus and computer program for managing the processing of extracted data
US20130111038A1 (en) * 2010-07-09 2013-05-02 Attila Technologies, Llc Transparent Proxy Architecture for Multi-Path Data Connections
US20180039890A1 (en) * 2016-08-03 2018-02-08 Electronics And Telecommunications Research Institute Adaptive knowledge base construction method and system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060218192A1 (en) * 2004-08-31 2006-09-28 Gopalakrishnan Kumar C Method and System for Providing Information Services Related to Multimodal Inputs
US20110119613A1 (en) * 2007-06-04 2011-05-19 Jin Zhu Method, apparatus and computer program for managing the processing of extracted data
US20130111038A1 (en) * 2010-07-09 2013-05-02 Attila Technologies, Llc Transparent Proxy Architecture for Multi-Path Data Connections
US20180039890A1 (en) * 2016-08-03 2018-02-08 Electronics And Telecommunications Research Institute Adaptive knowledge base construction method and system

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112231376A (zh) * 2020-09-25 2021-01-15 中国建设银行股份有限公司 一种用于离线数据采集的方法和装置
CN112463984A (zh) * 2020-12-04 2021-03-09 北京明略软件系统有限公司 数据库模式扩展方法、装置、设备及计算机可读介质
CN112463984B (zh) * 2020-12-04 2024-02-27 北京明略软件系统有限公司 数据库模式扩展方法、装置、设备及计算机可读介质
CN112612866A (zh) * 2020-12-29 2021-04-06 北京奇艺世纪科技有限公司 知识库文本同步方法、装置、电子设备及存储介质
CN112612866B (zh) * 2020-12-29 2023-07-21 北京奇艺世纪科技有限公司 知识库文本同步方法、装置、电子设备及存储介质

Similar Documents

Publication Publication Date Title
US11314764B2 (en) Automated scalable contextual data collection and extraction system
US20210211471A1 (en) Highly scalable four-dimensional web-rendering geospatial data system for simulated worlds
US20210182254A1 (en) Distributable model with biases contained within distributed data
US10938683B2 (en) Highly scalable distributed connection interface for data capture from multiple network service sources
US20210385251A1 (en) System and methods for integrating datasets and automating transformation workflows using a distributed computational graph
US20180247321A1 (en) Platform for management of marketing campaigns across multiple distribution mediums
US11546380B2 (en) System and method for creation and implementation of data processing workflows using a distributed computational graph
US20170124490A1 (en) Inclusion of time series geospatial markers in analyses employing an advanced cyber-decision platform
US10885085B2 (en) System to organize search and display unstructured data
US11636549B2 (en) Cybersecurity profile generated using a simulation engine
WO2019165456A1 (fr) Système automatisé de collecte et d'extraction de données contextuelles évolutives
US10860951B2 (en) System and method for removing biases within a distributable model
US20210258349A1 (en) System and method for data extraction, processing, and management across multiple communication platforms
WO2020167539A1 (fr) Système et procédé d'annotation, de traçage, d'analyse et de simulation de processus it complexe
US20180130077A1 (en) Automated selection and processing of financial models
US20220043806A1 (en) Parallel decomposition and restoration of data chunks
US11321637B2 (en) Transfer learning and domain adaptation using distributable data models
US11755957B2 (en) Multitemporal data analysis
WO2019071055A1 (fr) Amélioration d'un modèle distribuable à données distribuées
WO2019050557A1 (fr) Profil de sécurité informatique généré à l'aide d'un moteur de simulation
US11960978B2 (en) System and method for removing biases within a distributable model
WO2019071057A1 (fr) Amélioration d'un modèle distribuable avec des biais contenus dans des données distribuées

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19757941

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19757941

Country of ref document: EP

Kind code of ref document: A1