WO2017176944A1 - Système de capture entièrement intégrée et d'analyse d'informations commerciales aboutissant à une prise de décision et une simulation prédictives - Google Patents

Système de capture entièrement intégrée et d'analyse d'informations commerciales aboutissant à une prise de décision et une simulation prédictives Download PDF

Info

Publication number
WO2017176944A1
WO2017176944A1 PCT/US2017/026239 US2017026239W WO2017176944A1 WO 2017176944 A1 WO2017176944 A1 WO 2017176944A1 US 2017026239 W US2017026239 W US 2017026239W WO 2017176944 A1 WO2017176944 A1 WO 2017176944A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
business
analysis
sensor data
scrape
Prior art date
Application number
PCT/US2017/026239
Other languages
English (en)
Inventor
Jason Crabtree
Andrew Sellers
Original Assignee
Fractal Industries, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/091,563 external-priority patent/US10204147B2/en
Priority claimed from US15/141,752 external-priority patent/US10860962B2/en
Application filed by Fractal Industries, Inc. filed Critical Fractal Industries, Inc.
Priority to CN201780033615.6A priority Critical patent/CN109478296A/zh
Priority to EP17779786.7A priority patent/EP3440569A4/fr
Publication of WO2017176944A1 publication Critical patent/WO2017176944A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201—Market modelling; Market analysis; Collecting market data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Definitions

  • the present invention is in the field of use of computer systems in business information management, operations and predictive planning. Specifically, the development of a system that integrates the functions of business information and operating data, complex data analysis and use of that data, preprogrammed commands and parameters and machine learning to create a business operating system capable of predictive decision making and action path outcome simulation.
  • the present invention is in the field of receipt, storage, and analysis of large volumes of remote sensor data, specifically where the rate of sensor data delivery from the remote sensors is highly heterogeneous over time.
  • PALANTIRTM offers software to isolate patterns in large volumes of data
  • D AT AB RICKSTM offers custom analytics services
  • ANAPLANTM offers financial impact calculation services and there are other software sources that mitigate some aspect of business data relevancy identification, analysis of that data and business decision automation, but none of these solutions handle more than a single aspect of the whole task.
  • Boundary has reported successful capture and short term storage of data streams generated from their time series network monitoring sensors by paring down the number of sensor parameters that they were recording, aggregating the stream entering the system such that, for example ten seconds of sensor data would be committed to their data store as a single block and, also limiting each committed block to include readings from an arbitrary and finite number of their network sensors.
  • a block measuring a defined time period by an arbitrary subpopulation of the total sensors are written as blocks to a straight key:value pair data store for a predetermined amount of time, but then must be purged to maintain low enough data storage levels to keep the system functioning. There is no provision for long-term storage of the captured data.
  • the developers were also able to retrieve some of the raw data based on time recorded sensor ID and a keyword from the data store into which the sensor data was stored.
  • This system dubbed Kobayashi by the developers (Hungry Hungry Kobayashi-Dietrich Featherston from Boundary; https://vimeo . com/42902962 : 2012), was a very important foundation for sensor data stream capture and simple retrieval of sensor data from a data store over a short period of time. While Kobayashi advanced the art by its ability to capture and store simplified data streams from a set of sensors, Boundary's development could not perform several tasks required for functional sensor data stream capture and analysis.
  • What is needed is a system and method to capture time series data to a data store from a plurality of sensors which may send data at irregular intervals, where a large amount of data may come from the array of sensors concurrently.
  • What is further needed is a system and method for the long term archive storage of older data, possibly no longer needed for current analysis and which would lead to unacceptably high maintenance overhead in the live database, but which could be crucial in longer term trend or antecedent event analysis.
  • a system and methods to allow straightforward, standardized search and analysis of specific sensor data based upon complex search criteria followed by employment of transformation tools to maximize data informational potential as well as a robust set of presentation formats to maximize the informational value of the transformed data.
  • a system for comprising: a business data retrieval engine stored in a memory of and operating on a processor of a computing device, a business data analysis engine stored in a memory of and operating on a processor of a computing device and a business decision and business action path simulation engine stored in a memory of and operating on a processor of one of more computing devices.
  • the business information retrieval engine retrieves a plurality of business related data from a plurality of sources, accepts a plurality of analysis parameters and control commands directly from human interface devices or from one or more command and control storage devices, and stores accumulated retrieved information for processing by data analysis engine or predetermined data timeout.
  • the business information analysis engine retrieves a plurality of data types from the business information retrieval engine, performs a plurality of analytical functions and transformations on retrieved data based upon the specific goals and needs set forth in a current campaign by business process analysis authors.
  • the business decision and business action path simulation engine employs results of data analyses and transformations performed by the business information analysis engine, together with available supplemental data from a plurality of sources as well as any current campaign specific machine learning, commands and parameters from business process analysis authors to formulate current business operations and risk status reports and employs results of data analyses and transformations performed by the business information analysis engine, together with available supplemental data from a plurality of sources, any current campaign specific commands and parameters from business process analysis authors, as well as input gleaned from machine learning algorithms to deliver business action pathway simulations and business decision support to a first end user.
  • the system's business information retrieval engine is stored in the memory of and operating on a processor of a computing device, employs a portal for human interface device input at least a portion of which are business related data and at least another portion of which are commands and parameters related to the conduct of a current business analysis campaign.
  • the business information retrieval engine employs a high throughput deep web scraper stored in the memory of an operating on a processor of a computing device, which receives at least some spider configuration parameters from the highly customizable cloud based interface, coordinates one or more world wide web searches (scrapes) using both general search control parameters and individual web search agent (spider) specific configuration data, receives scrape progress feedback information which may lead to issuance of further web search control parameters, controls and monitors the spiders on distributed scrape servers, receives the raw scrape campaign data from scrape servers, aggregates at least portions of scrape campaign data from each web site or web page traversed as per the parameters of the scrape campaign.
  • the archetype spiders are provided by a program library and individual spiders are created using configuration files. Scrape campaign requests are persistently stored and can be reused or used as the basis for similar scrape campaigns.
  • the business information retrieval engine employs a high throughput deep web scraper stored in the memory of an operating on a processor of a computing device, which receives at least some spider
  • multidimensional time series data store stored in a memory of and operating on a processor of a computing device to receive a plurality of data from a plurality of sensors of heterogeneous types, some of which may have heterogeneous reporting and data payload transmission profiles, aggregates the sensor data over a predetermined amount of time, a predetermined quantity of data or a predetermined number of events, retrieves a specific quantity of aggregated sensor data per each access connection predetermined to allow reliable receipt and inclusion of the data, transparently retrieves quantities of aggregated sensor data too large to be reliably transferred by one access connection using a further plurality access connections to allow capture of all aggregated sensor data under conditions of heavy sensor data influx and stores aggregated sensor data in a simple key- value pair with very little or no data transformation from how the aggregated sensor data is received.
  • the business data analysis engine employs a directed computational graph stored in the memory of an operating system on a processor of a computing device which, retrieves streams of input from one or more of a plurality of data sources, filters data to remove data records from the stream for a plurality of reasons drawn from, but not limited to a set comprising absence of all information, damage to data in the record, and presence of in-congruent information or missing information which invalidates the data record, splits filtered data stream into two or more identical parts, formats data within one data stream based upon a set of predetermined parameters so as to prepare for meaningful storage in a data store, sends identical data stream further analysis and either linear transformation or branching transformation using resources of the system.
  • a method for fully integrated capture, and transformative analysis of business impactful information resulting in predictive decision making and simulation comprising the steps of: (a) retrieving business related data and analysis campaign command and control information using a business information retrieval engine stored in the memory of an operating on a processor of a computing device; (b) analyzing and transforming retrieved business related data using a business information analysis engine stored in the memory of an operating on a processor of a computing device in conjunction with previously designed analysis campaign command and control information; and (c) presenting business decision critical information as well as business action pathway simulation information using a business decision and business action path simulation engine based upon the results of analysis of previously retrieved business related data and previously entered analysis campaign command and control information.
  • the inventor has developed a distributed system for the capture and storage of time series data from sensors with heterogeneous reporting profiles which can scale to receive periods of high data throughput.
  • the system further includes the capacity to archive data that has surpassed a predetermined age within the live data store but which is still required.
  • the system includes a robust SQL-like query language that not only permits users to use complex logic to specifically select desired data, but also to employ data transformation processes on selected data before data is displayed.
  • a system for capture, analysis and storage of data time series from sensors with heterogeneous report interval profiles comprising a data stream management engine stored in a memory of and operating on a processor of a computing device, a multidimensional time series data store stored in a memory of and operating on a processor of a computing device, a data query and output engine stored in a memory of and operating on a processor of one of more computing devices is disclosed.
  • the data stream management engine receives a plurality of sensor data, aggregates the sensor data over a predetermined amount of time, a predetermined quantity of data, or a predetermined number of events for transmission into the multidimensional time series data store, transmits a specific quantity of aggregated sensor data per each access connection predetermined to allow reliable transmission to and inclusion of the data into the multidimensional time series data store, and transparently transmits quantities of aggregated sensor data too large to be reliably transferred by one access connection using a further plurality access connections to allow capture of all aggregated sensor data by the multidimensional time series data store under conditions of heavy sensor data influx.
  • the multidimensional time series data store stores aggregated sensor data in a simple key- value pair format with very little or no data transformation from how the aggregated sensor data is received, and stores data for a predetermined number of samples and then
  • the data query and output engine is a point of interaction to set up analysis prior to sensor data collection by specifying such parameters as number of events or time units to be placed within each quantum of aggregated sensor data, the number of connection lanes between the data stream management engine and the multidimensional time series data store, number of lanes that can be combined at one time to transfer aggregated sensor data to the multidimensional data store, number of interrelated dimensions to be stored per sensor, has an SQL like query language to retrieve sensor data of interest from the multidimensional time series database in a useful format and is the point of interaction for selecting transformations performed on the retrieved multidimensional time series data store as well as specifying the format of data output.
  • a method for capture, analysis and storage of data time series from sensors with heterogeneous report interval profiles comprising the steps of: (a) receiving raw data from remote sensors with irregular reporting interval profiles; (b) aggregating the sensor data based upon the number of sensors included in the analysis and a predetermined time interval or a predetermined number of events; (c) transferring aggregated sensor data to a multidimensional time series data store using one to a predetermined plurality of communication lines dependent upon the number of raw sensors reporting data; (d) storing un-transformed aggregated sensor data in a key- value pair data store for a predetermined period of time, removing the stored aggregated sensor data as it times out; (e) retrieving aggregated sensor data of interest to an ongoing analysis from the multidimensional data store using an SQL-like query language; and (f) transforming then formatting that data as best suited to its role in the ongoing analysis.
  • FIG. 1 is a diagram of an exemplary architecture of a business operating system according to an embodiment of the invention.
  • Fig. 2 is a process flow diagram showing an exemplary set of steps used in the function of the very high bandwidth cloud interface.
  • FIG. 3 is a diagram of an exemplary architecture for a linear transformation pipeline system which introduces the concept of the transformation pipeline as a directed graph of transformation nodes and messages according to an embodiment of the invention.
  • Fig. 4 is a process flow diagram of a method for an embodiment of modeling the
  • transformation pipeline module of the invention as a directed graph using graph theory.
  • Fig. 5 is a process flow diagram of a method for one embodiment of a linear transformation pipeline.
  • Fig. 6 is a process flow diagram of a method for one embodiment of a transformation pipeline where one transformation node in a transformation pipeline receives data streams from two source transformation nodes.
  • Fig. 7 is a process flow diagram of a method for one embodiment of a transformation pipeline where one transformation node in a transformation pipeline sends output data stream to two destination transformation nodes in potentially two separate transformation pipelines.
  • Fig. 8 is a diagram showing exemplary world wide web target sites containing the type of loosely structured, large volume, data that make them candidates for search and retrieval by the invention according to an embodiment of the invention.
  • Fig. 9 is a process flow diagram of a method for a high volume web crawling module.
  • Fig. 10 is a listing of a very simple example Scrapy web spider configuration file.
  • Fig. 11 is a method flow diagram showing an exemplary set of step used in the capture and storage of time series data from sensors with heterogeneous reporting profiles according to an embodiment of the invention.
  • Fig. 12 is a process flow diagram of a method for the use of metaswimlanes to transparently accommodate levels of data streaming which would overload a single swimlane according to an embodiment of the invention.
  • Fig. 13 is a simplified example of the use a Kalman filter to extract and smooth estimated system state from noisy sensor data according to an embodiment of the invention.
  • FIG. 14 is a block diagram illustrating an exemplary hardware architecture of a computing device used in various embodiments of the invention.
  • Fig. 15 is a block diagram illustrating an exemplary logical architecture for a client device, according to various embodiments of the invention.
  • Fig. 16 is a block diagram illustrating an exemplary architectural arrangement of clients, servers, and external services, according to various embodiments of the invention.
  • Fig. 17 is another block diagram illustrating an exemplary hardware architecture of a computing device used in various embodiments of the invention
  • Fig. 18 is a method process flow diagram showing the operation of an automated planning service module according to an embodiment of the invention.
  • Fig. 19 is a diagram of an exemplary architecture of a system for the capture and storage of time series data from sensors with heterogeneous reporting profiles according to an embodiment of the invention.
  • the inventor has conceived, and reduced to practice, a system and method for capture, analysis, and storage of time series data from sensors with heterogeneous report interval profiles.
  • steps may be performed simultaneously despite being described or implied as occurring sequentially (e.g., because one step is described after the other step).
  • the illustration of a process by its depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications thereto, does not imply that the illustrated process or any of its steps are necessary to one or more of the invention(s), and does not imply that the illustrated process is preferred.
  • steps are generally described once per embodiment, but this does not mean they must occur once, or that they may only occur once each time a process, method, or algorithm is carried out or executed. Some steps may be omitted in some embodiments or some occurrences, or some steps may be executed more than once in a given embodiment or occurrence.
  • a “swimlane” is a communication channel between a time series sensor data reception and apportioning device and a data store meant to hold the apportioned data time series sensor data.
  • a swimlane is able to move a specific, finite amount of data between the two devices. For example a single swimlane might reliably carry and have incorporated into the data store, the data equivalent of 5 seconds worth of data from 10 sensors in 5 seconds, this being its capacity. Attempts to place 5 seconds worth of data received from 6 sensors using one swimlane would result in data loss.
  • a "metaswimlane” is an as-needed logical combination of transfer capacity of two or more real swimlanes that is transparent to the requesting process. Sensor studies where the amount of data received per unit time is expected to be highly heterogeneous over time may be initiated to use metaswimlanes.
  • FIG. 1 is a diagram of an exemplary architecture of a business operating system 100
  • BEANSTALKTM both used for standards compliance and ease of development.
  • the directed computational graph retrieves one or more streams of data from a plurality of sources, which includes, but is in no way not limited to, a number of physical sensors, web based questionnaires and surveys, monitoring of electronic infrastructure, crowd sourcing campaigns, and human input device information.
  • data may be split into two identical streams, wherein one sub-stream may be sent for batch processing and storage while the other sub-stream may be reformatted for transformation pipeline analysis.
  • the data is then transferred to general transformer service 160 for linear data transformation as part of analysis or decomposable transformer service 150 for branching or iterative transformations that are part of analysis.
  • the directed computational graph 155 represents all data as directed graphs where the transformations are nodes and the result messages between transformations edges of the graph. These graphs which contain considerable intermediate transformation data are stored and further analyzed within graph stack module 145.
  • High volume web crawling module 115 uses multiple server hosted preprogrammed web spiders to find and retrieve data of interest from web based sources that are not well tagged by conventional web crawling technology.
  • Multiple dimension time series database module 120 receives data from a large plurality of sensors that may be of several different types.
  • the module is designed to accommodate irregular and high volume surges by dynamically allotting network bandwidth and server processing channels to process the incoming data.
  • Data retrieved by the multidimensional time series database 120 and the high volume web crawling module 115 may be further analyzed and transformed into task optimized results by the directed computational graph 155 and associated general transformer service 150 and decomposable transformer service 160 modules.
  • Results of the transformative analysis process may then be combined with further client directives, additional business rules and practices relevant to the analysis and situational information external to the already available data in the automated planning service module 130 which also runs powerful predictive statistics functions and machine learning algorithms to allow future trends and outcomes to be rapidly forecast based upon the current system derived results and choosing each a plurality of possible business decisions.
  • the automated planning service module 130 may propose business decisions most likely to result is the most favorable business outcome with a usably high level of certainty.
  • the business outcome simulation module 125 coupled with the end user facing observation and state estimation service 140 allows business decision makers to investigate the probable outcomes of choosing one pending course of action over another based upon analysis of the current available data.
  • the pipelines operations department has reported a very small reduction in crude oil pressure in a section of pipeline in a highly remote section of territory. Many believe the issue is entirely due to a fouled, possibly failing flow sensor, others believe that it is a proximal upstream pump that may have foreign material stuck in it. Correction of both of these possibilities is to increase the output of the effected pump to hopefully clean out it or the fouled sensor.
  • Fig. 2 is a process flow diagram showing an exemplary set of steps used in the function of the very high bandwidth cloud interface 200, also depicted in Fig. 1, 110.
  • Data flowing into and out of the very high bandwidth cloud interface 200 may come from human interactions through desktop or mobile computing devices 202, reading data sent from remote sensor arrays 203 and data retrieved from web pages 204 both of which 203, 204 may reach a very high instantaneous volume for moderate time intervals which must be accommodated by the interface to assure reliable data capture.
  • the cloud 201 may usually mean the internet, often the World Wide Web in the current context, it also extends here to data transmitted from the confines of the client business to the business operating system which may use a separate network topology.
  • web apps constructed and supported using mostly open source resources, present graphical interfaces for end users to both submit new information 207 and to visualize the results of analyses and predictive decisions as well as simulations created by the business operating system 208.
  • Programming is also used to accept and properly route command line directives and parameters from analysts and programmers to the system as analyses are carried out 208.
  • Sensor data and raw webpage data being retrieved by the multiple dimension time series database module, depicted in 120, and high volume web crawling module depicted in 115, also may pass through the high volume interface 205. While this embodiment represents the cloud interface as a monolithic portion of the business operating system architecture, the invention has no such requirement and thus in other embodiments, data, programming command and campaign parameters may enter the system from multiple portal to the cloud.
  • FIG. 3 is a block diagram of a preferred architecture for a transformation pipeline within a system for predictive analysis of very large data sets using distributed computational graph 300.
  • streaming input 315 serves as input to the first transformation node 320 of the transformation pipeline. Transformation node's function is performed on input data stream and transformed output message 325 is sent to transformation node 2 330.
  • the progression of transformation nodes 320, 330, 340, 350, 360 and associated output messages from each node 325, 335, 345, 355 is linear in configuration this is the simplest arrangement and, as previously noted, represents the current state of the art.
  • transformation nodes are described according to various embodiments as uniform shape, such uniformity is used for presentation simplicity and clarity and does not reflect necessary operational similarity between transformations within the pipeline. It should be appreciated that one knowledgeable in the field will realize that certain transformations in a pipeline may be entirely self-contained; certain transformations may involve human interaction through a program running on a desktop or mobile device 330, such as selection via dial or dials, positioning of switch or switches, or parameters set on control display, all of which may change during analysis; other transformations may require external aggregation or correlation services or may rely on remote procedure calls to synchronous or asynchronous analysis engines as might occur in simulations among a plurality of other possibilities. Further according to the embodiment, individual transformation nodes in one pipeline may represent function of another transformation pipeline.
  • Fig. 4 is a process flow diagram of a method 400 for an embodiment of modeling the transformation pipeline module 160 of the invention as a directed graph using graph theory 155.
  • the individual transformations 402, 404, 406 of the transformation pipeline ti.. t n such that each t, T are represented as graph nodes. Transformations belonging to T are discrete transformations over individual datasets d, , consistent with classical functions. As such, each individual transformation tj, receives a set of inputs and produces a single output.
  • the output of an individual transformation is defined as the function out: t, [Idi] to describe transformations that produce a single output (usable by other transformations).
  • a dependency function can now be defined such that dep(t a ,tb) out(t a )in(t b )The messages carrying the data stream through the transformation pipeline 401, 403, 405 make up the graph edges.
  • Fig. 5 is a process flow diagram of a method 500 for one embodiment of a linear
  • Fig. 6 is a process flow diagram of a method 600 for one embodiment of a transformation pipeline where one transformation node 607 in a transformation pipeline receives data streams from two source transformation nodes 601. The invention handles this transformation pipeline
  • Fig. 7 is a process flow diagram of a method 700 for one embodiment of a transformation pipeline where one transformation node 703 in a transformation pipeline sends output data stream to two destination transformation nodes 701, 706, 708 in potentially two separate transformation pipelines.
  • the invention handles this transformation pipeline configuration by decomposing or serializing the output events 704,705-706, 707-708.
  • the results of the source transformation node 703 just antecedent to the destination transformation nodes 706 and placed into a single specialized data storage transformation node 704, 705, 707 (shown three times as storage occurs and retrieval occurs twice).
  • the results of the antecedent transformation node may then be retrieved from a data store 704 and serves as the input stream for the transformation nodes two downstream
  • transformation pipeline 706, 708 The example depicted in this diagram was chosen to convey the configuration of transformation pipelines with individual transformation nodes that send output streams to two destination nodes 706, 708 and is the simplest form of the configuration felt to show the point. It in no way implies limitation of the invention. One knowledgeable in the art will realize the great number of permutations and topologies possible, especially as the invention places no design restrictions on the number of transformation nodes sending output to greater than one destination or the number destinations receiving input from a source node. This example
  • FIG. 8 is a block diagram 800 of websites on the world wide web that are example target types of a distributed system for large volume extraction of deep web data, www.sei.smi .org 810 is a website of geoseismic data which by nature is non-textual and therefore has very few tags that might be useful to conventional web crawlers.
  • tjmmifce ⁇ 811 is a web site that publishes raw spending data reports with are largely textual, but has extremely few, if any, web related tags and is thus poorly indexed or retrieved by conventional scraping.
  • This type of web site also is expected to have a very large volume of data which again serves to thwart conventional web crawling tools.
  • toolkit.cjimate.gov 812 like www.seisrni.org 810, is a site that would be expected to have large amounts of non-textual climate data that needs to be processed with few if any web related tags meaning that climate intrinsic keywords would need to be employed for meaningful retrieval of the scraped data and, again both data transformation steps and pre-storage processing may be needed prior to meaningful storage.
  • http://hall-of- justice.herokuapp.c >m/categorv y /corrections/ 813, htip://hal3 ⁇ 4-of- justice.herokuappxora/categon'/financial/ 814, and http://www. electionpassport.com 815 are all similar in that they are sites with extremely large volumes of free form textual data with few if any web tags and high probability that data retrieved will need to be processed prior to output or storage.
  • Fig. 9 is a process flow diagram of a method 900 for a high volume web crawling module 115.
  • Parameters for one or more scrape campaigns configuration data which may be comprised of, but is not limited to: web sites or web pages to be traversed, keywords or tags for web document data to be parsed, and search expansion rules for following links or other references found on the sites scraped, as well as any other spider configuration information included by the authors of the scrape campaign; and scrape campaign control directives which may include but would not be limited to: the number of spiders to be used in the campaign, relative resource usage priorities for specific web sites or pages within the intended scrape campaigns, directives for adjustments to be made to the scrape campaign upon the encounter of specific results or types of results, directives for application of specific scrape campaign result data pre- processing and post- processing steps and output format directives including persistent storage formalization rules; are received through either a command line interface 910 which may receive commands either from an interactive terminal 105 or another software application on a computing system 115 or from software applications 110
  • REST and JSON within the API should not be construed to mean that the invention is dependent on use of only those protocols for this task as one knowledgeable in the art will realize that any similar protocols such as, but not limited to, MQTT-based messaging, SOAP or AJAX could be employed.
  • the use of REST and JSON is only in accordance with current practice and inventor decision. Scrape campaign control and spider configuration parameters received are formalized, as necessary and stored in data store for future use when the scrape campaign is initiated. In initiation may be immediate or delayed and the same scrape campaign may be repeatedly run as parameters persist until purged.
  • the invention uses the control directives passed to it by the scrape campaign authors to coordinate the scrape campaign 940.
  • Directives from a list comprising the number and complexity of the web sites to be scraped, the priorities assigned to specific web sites or pages, the number different spider configurations to be employed, the speed the author desires the scrape to progress among other factors are used to determine the number of spiders that will be deployed and the number of scrape servers to be included in the scrape as per predetermined programming within the invention.
  • scrape campaign controller module While the scrape is active, progress and operational information such as stuck spiders and intermediate scrape results is continuously monitored 950 by the scrape campaign controller module through the scrape controllers 115 such that the authors of the scrape campaign can determine the progress made in the scrape, have some indication of what results have been produced, know what tasks the spiders still have pending as well as any links that may have been followed and the impact on the scrape as a whole of those additions as per pre-programmed reporting parameters 980. Monitoring 950 and reporting 980 aware of operational issues that have arisen, if any.
  • Scrape results obtained by the individual spiders are passed through the scrape controller modules 115 of the scrape servers 115 and are aggregated and then possibly transformed in specific ways depending on the predetermined goals of the scrape campaign 970.
  • the invention offers pre-programmed algorithm toolsets for this purpose and also offers API hooks that allow the data to be passed to external processing algorithms prior to final output in a format pre-decided to be most appropriate for the needs of the scrape campaign authors.
  • Result data may also be appropriately processed and formalized for persistent storage in a document based data store 990 such as MongoDB, although, depending on the needs of the authors and the type of data retrieved during the scrape, any NOSQL type data storage or even a relational database may be used.
  • a document based data store 990 such as MongoDB
  • NOSQL type data storage or even a relational database may be used.
  • the invention has no dependency for any particular data store type for persistent storage of scrape results.
  • Scrapy was chosen for several reasons some of which are: The programming, in Python, for the function of a basic web spider is already present and scrape authors therefore do not need extensive programming expertise in designing spiders to use the framework; The format and keywords for the remaining configuration parameters 900 needed to create scrape campaign specific spiders is well defined, feature robust and well documented ( http://doc.8crapy.org/en/la.test/index.html ), and the Scrapy framework has been shown reliable and stable during use by such high data throughput web sites as CareerBuilder.com, BiteFinder.com and Data.gov.uk. While the invention currently makes integral use of the Scrapy framework for the definition of spiders used, it is not programmatically dependent on the Scrapy framework to the point that another web crawling agent framework (e.g. OXPath - http://oxpath.org) could not be substituted if a better alternative were to be found and the use of Scrapy should not be seen to strictly define the invention in that capability.
  • OXPath http://oxpath.org
  • Fig. 10 is a listing of a very simple example Scrapy web spider configuration file 1000. This listing requires that the Scrapy framework as well as libraries on which Scrapy depends
  • the listing 1000 shows all of the major sections needed to create a scrape specific spider 1010, 1020, 1030,1040 At the top of the listing 1010 is found a section that declares the portions of the Scrapy framework that is to be included in the creation of the current spider. Going down the listing, the next section 1020 declares a name to be used to identify this spider type as well as the world wide web domains the spider is allowed to traverse during the scrape and last, the url of the starting point of the scrape.
  • the spider is scraping specific types of HTML links from the example.com domain.
  • the last section 1040 has the instructions on how to process the target data, including instructions for data associated to specific web tags. While the spider created by this sample configuration would have limited capability, it is functional and would, as written complete its scrape. One will immediately appreciate that all of the directives in the listed spider definition have to do with retrieving the data and not the minutia of how the spider gets to the web site or implements the instructions given in the listed file, etc. This provides the rationale for this framework being used in the invention.
  • Fig. 11 is a method flow diagram showing an exemplary method 1100 used in the capture and storage of time series data from sensors with heterogeneous reporting profiles according to an embodiment of the invention.
  • data is received from a set of sensors connected to a capture and analysis device as in the embodiment.
  • the sensor data received might be captured and stored under two main paradigms. One is that the sensor data arrives at a defined, reliable periodicity, which may be continuously, but the amount of data per unit time is reliably homogeneous and thus the capture and storage of the sensor data is easy to perform using simple time based models. This paradigm and its resolution is prior art and is not depicted.
  • the second paradigm occurs when the sensors being monitored send data at irregular intervals and the amount of data received by the capture and analysis device can vary greatly overtime.
  • This heterogeneous sensor data behavior demands different processing strategies than does the homogeneous counterpart.
  • Sensor data capture devices that store sensor data at strictly regular time intervals fair badly as the amounts of data per storage cycle can vary greatly.
  • Two strategies that have been found to work reliably in conditions of heterogeneous data influx are event driven and stream capture.
  • the event driven strategy holds data in the memory of a data stream management engine 120 until a preset number of data events have occurred 1110-1120. Data is processed by selecting the parameters, or dimensions within it that are of importance to the administrator and then stored to the data store when a predetermined threshold of events is reached 1120, 1130.
  • the streaming strategy uses the quantity of data accumulated in a data stream management engine 120 as the trigger 1110-1115 to commit the processed sensor data to storage 1115-1130.
  • an administrator may preselect either event driven or stream driven commitment, as well as many other parameters pertaining to analysis of sensor data using the administration device 120
  • All sensor data, regardless of delivery circumstances are stored in a multidimensional time series data store 1130 which is designed for very low overhead, rapid data storage and minimal maintenance needs to sap resources.
  • the embodiment uses a key-value pair data store examples of which are RIAKTM, REDISTM, and BERKELEY DBTM for their low overhead and speed although the invention is not specifically tied to a single data store type that is known in the art should another with better response to feature characteristics emerge.
  • data store commitment reliability is dependent on data store data size under the conditions intrinsic to time series sensor data analysis.
  • the number of data records must be kept relatively low for the herein disclosed purpose.
  • one group of developers restrict the size of their multidimensional time series key- value pair data store to approximately 8.64 x 10 4 records, equivalent to 24 hours of one-second interval sensor readings or 60 days of one-minute interval readings.
  • the oldest data is deleted from the data store and lost. This loss of data is acceptable under development conditions but in a production environment, the loss of the older data is almost always significant and unacceptable.
  • the invention accounts for this need to retain older data by stipulating that aged data be placed in long term storage.
  • the archival storage is included 1170.
  • This archival storage as shown provided by data archive 120 might be locally provided by the user, might be cloud based such as that offered by Amazon Web Services or Google or could be any other available very large capacity storage method known to those skilled in the art.
  • Sensor data can be specifically retrieved, using complex query logic 1135 and transformed using such tools as mean reading of all query included sensors, variance of all readings of all sensors queried, standard deviation of queried sensors and more complex types such as standard linear interpolation, Kalman filtering and smoothing, may be applied.
  • Data can then be represented in various formats such as, but not limited to text, JSON, KML, GEOJSON and TOPOJSON by the system depending on the ultimate use of the resultant information 1180.
  • Fig. 12 is a process flow diagram of a method for the use of metaswimlanes to transparently accommodate levels of data streaming which would overload a single swimlane according to an embodiment of the invention.
  • a data stream management engine 1210 also shown in context to an entire embodiment of the invention as 120 to a multidimensional time series data store 1220 shown in context as 120 in a system embodiment of the invention 100, will exceed the instantaneous data capacity of a single data channel, or swimlane 1211a between the data stream management engine 1210 and the
  • the remedy taken and shown in this embodiment is the ability of the system to, when configured, combine the transfer and commitment bandwidth of two or more real swimlanes 1215a, 1215b in a way that is transparent to the committing process.
  • This means that the invention handles the physical transfer pathway as well as the logical details such as tracking the multiple key-value pairs, process identifications and any application specific bookkeeping involved as overhead to the process and then creating a data structure to have the data records act as a single entity in subsequent data manipulations.
  • Kalman filter 13 is a simplified example of the use a Kalman filter to extract and smooth estimated system state from noisy sensor data according to an embodiment of the invention. Because of its ability to extract reliably accurate, interpretable data in cases of noisy input data, heavy use is made of Kalman filters in data transformation functions of various embodiments of the invention. It is useful to provide a simple demonstration of how such filters might work in one or more
  • a system 120 may be connected to an array of CO2 sensors to monitor the progress of the cleanup.
  • 50% CO2 registers as 1000 on the CO2 sensors and as a whole the manufacturer states the array will have a noise level of 400. It is believed that the efforts can remove 15.0%) of the present CO2 per hour.
  • a is equal to the percent of C02 that will be left, compared to the percent in the previous
  • a x*-i represents the previous estimated result r is the publish noise level of the sensor or sensor array zk represents the current observed result p* is the prediction error between the last previous expected result and the last previous observed result.
  • gk is the factor by which the difference between the last expected result and the current observed result that when added to the last expected result will produce the current expected result.
  • Fig. 18 is a method process flow diagram showing the operation of an automated planning service module according to an embodiment of the invention.
  • the analytics data results from the system are supplied to the automated planning service module 1802 as depicted in 130.
  • the analytic data results are mapped to all possible business actions or decisions which are suggested by the broad findings and known within the system. Many of these actions may have been entered specifically for the current campaign 1802. Any external source information such as existing business practices that impact the decision, legal and regulatory considerations that impact the proposed action among an additional plurality of possible factors known to the art, are then incorporated into the action selection process 1803. Once the broadest set of possible prospective actions accounting for external parameters is known, information theory statistics algorithms and machine learning principles are employed on the analytic data developed by the system 100 to reliably predict the probable outcomes of pursuing each choice and provide statistical data associated with each action 1804. The data pertaining to actions with a favorable outcome value above a predetermined threshold are sent to the simulation module 125 and the observation and state estimation 140 modules for appropriate presentation to end users as dictated by the authors of the related analytical campaign 1805.
  • Fig. 19 is a diagram of an exemplary architecture of a system for the capture and storage of time series data from sensors with heterogeneous reporting profiles according to an embodiment of the invention 1900.
  • a plurality of sensor devices 1910a-n stream data to a collection device, in this case a web server acting as a network gateway 1915.
  • These sensors 1910a- n can be of several forms, some non-exhaustive examples being: physical sensors measuring humidity, pressure, temperature, orientation, and presence of a gas; or virtual such as programming measuring a level of network traffic, memory usage in a controller, and number of times the word "refill" is used in a stream of email messages on a particular network segment, to name a small few of the many diverse forms known to the art.
  • the sensor data is passed without transformation to the data management engine 1920, where it is aggregated and organized for storage in a specific type of data store 1925 designed to handle the multidimensional time series data resultant from sensor data.
  • Raw sensor data can exhibit highly different delivery characteristics. Some sensor sets may deliver low to moderate volumes of data continuously. It would be infeasible to attempt to store the data in this continuous fashion to a data store as attempting to assign identifying keys and the to store real time data from multiple sensors would invariably lead to significant data loss.
  • the data stream management engine 1920 would hold incoming data in memory, keeping only the parameters, or "dimensions" from within the larger sensor stream that are pre-decided by the administrator of the study as important and instructions to store them transmitted from the administration device 1912.
  • the data stream management engine 120 would then aggregate the data from multiple individual sensors and apportion that data at a predetermined interval, for example, every 10 seconds, using the the timestamp as the key when storing the data to a multidimensional time series data store over a single swimlane of sufficient size.
  • This highly ordered delivery of a foreseeable amount of data per unit time is particularly amenable to data capture and storage but patterns where delivery of data from sensors occurs irregularly and the amount of data is extremely heterogeneous are quite prevalent. In these situations, the data stream management engine cannot successfully use strictly single time interval over a single swimlane mode of data storage.
  • the invention also can make use of event based storage triggers where a predetermined number of data receipt events, as set at the administration device 1912, triggers transfer of a data block consisting of the apportioned number of events as one dimension and a number of sensor ids as the other.
  • the system time at commitment or a time stamp that is part of the sensor data received is used as the key for the data block value of the value-key pair.
  • the invention can also accept a raw data stream with commitment occurring when the accumulated stream data reaches a predesigned size set at the administration device 1912.
  • the embodiment of the invention can, if capture parameters pre-set at the administration device 1912, combine the data movement capacity of two or more swimlanes, the combined bandwidth dubbed a metaswimlane, transparently to the committing process, to accommodate the influx of data in need of commitment.
  • a diagrammatic representation of the formation of metaswimlanes 1215a, 1215b from individual swimlanes 1211a-z is shown in Figure 12.
  • All sensor data, regardless of delivery circumstances are stored in a multidimensional time series data store 1925 which is designed for very low overhead and rapid data storage and minimal maintenance needs to sap resources.
  • the embodiment uses a key-value pair data store examples of which are Riak, Redis and Berkeley DB for their low overhead and speed, although the invention is not specifically tied to a single data store type to the exclusion of others known in the art should another data store with better response and feature characteristics emerge. Due to factors easily surmised by those knowledgeable in the art, data store commitment reliability is dependent on data store data size under the conditions intrinsic to time series sensor data analysis. The number of data records must be kept relatively low for the herein disclosed purpose.
  • the archival storage is included 1930. This archival storage might be locally provided by the user, might be cloud based such as that offered by Amazon Web Services or Google or could be any other available very large capacity storage method known to those skilled in the art.
  • One or more transformational filters which include but a not limited to: mean, median, variance, standard deviation, standard linear interpolation, or Kalman filtering and smoothing, may be applied and then data formatted in one or more formats examples of with are text, JSON, KML, GEOJSON and TOPOJSON among others known to the art, depending on the intended use of the data.
  • At least some of the features or functionalities of the various embodiments disclosed herein may be implemented on one or more general-purpose computers associated with one or more networks, such as for example an end-user computer system, a client computer, a network server or other server system, a mobile computing device (e.g., tablet computing device, mobile phone, smartphone, laptop, or other appropriate computing device), a consumer electronic device, a music player, or any other suitable electronic device, router, switch, or other suitable device, or any combination thereof.
  • at least some of the features or functionalities of the various embodiments disclosed herein may be implemented in one or more virtualized computing environments (e.g., network computing clouds, virtual machines hosted on one or more physical computing machines, or other appropriate virtual environments).
  • FIG. 14 there is shown a block diagram depicting an exemplary computing device 10 suitable for implementing at least a portion of the features or functionalities disclosed herein.
  • Computing device 10 may be, for example, any one of the computing machines listed in the previous paragraph, or indeed any other electronic device capable of executing software- or hardware-based instructions according to one or more programs stored in memory.
  • Computing device 10 may be configured to communicate with a plurality of other computing devices, such as clients or servers, over communications networks such as a wide area network a metropolitan area network, a local area network, a wireless network, the Internet, or any other network, using known protocols for such communication, whether wireless or wired.
  • communications networks such as a wide area network a metropolitan area network, a local area network, a wireless network, the Internet, or any other network, using known protocols for such communication, whether wireless or wired.
  • computing device 10 includes one or more central processing units (CPU) 12, one or more interfaces 15, and one or more busses 14 (such as a peripheral component interconnect (PCI) bus).
  • CPU 12 may be responsible for implementing specific functions associated with the functions of a specifically configured computing device or machine.
  • a computing device 10 may be configured or designed to function as a server system utilizing CPU 12, local memory 11 and/or remote memory 16, and interface(s) 15.
  • CPU 12 may be caused to perform one or more of the different types of functions and/or operations under the control of software modules or components, which for example, may include an operating system and any appropriate applications software, drivers, and the like.
  • CPU 12 may include one or more processors 13 such as, for example, a processor from one of the Intel, ARM, Qualcomm, and AMD families of microprocessors.
  • processors 13 may include specially designed hardware such as application-specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEPROMs), field- programmable gate arrays (FPGAs), and so forth, for controlling operations of computing device 10.
  • ASICs application-specific integrated circuits
  • EEPROMs electrically erasable programmable read-only memories
  • FPGAs field- programmable gate arrays
  • a local memory 11 such as non- volatile random access memory (RAM) and/or read-only memory (ROM), including for example one or more levels of cached memory
  • RAM non- volatile random access memory
  • ROM read-only memory
  • interfaces 15 are provided as network interface cards (NICs).
  • NICs control the sending and receiving of data packets over a computer network; other types of interfaces 15 may for example support other peripherals used with computing device 10.
  • NICs network interface cards
  • the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, graphics interfaces, and the like.
  • various types of interfaces may be provided such as, for example, universal serial bus (USB), Serial, Ethernet,
  • FIREWIRETM FIREWIRETM, THUNDERBOLTTM, PCI, parallel, radio frequency (RF), BLUETOOTHTM, near- field communications (e.g., using near-field magnetics), 802.11 (WiFi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SAT A) or external SATA
  • ESATA high-definition multimedia interface
  • HDMI high-definition multimedia interface
  • DVI digital visual interface
  • ATM asynchronous transfer mode
  • HSSI high-speed serial interface
  • POS Point of Sale
  • FDDIs fiber data distributed interfaces
  • interfaces 15 may include physical ports appropriate for
  • ⁇ may also include an independent processor (such as a dedicated audio or video processor, as is common in the art for high-fidelity A/V hardware interfaces) and, in some instances, volatile and/or non-volatile memory (e.g., RAM).
  • an independent processor such as a dedicated audio or video processor, as is common in the art for high-fidelity A/V hardware interfaces
  • volatile and/or non-volatile memory e.g., RAM
  • processors 13 may be used, and such processors 13 may be present in a single device or distributed among any number of devices.
  • a single processor 13 handles communications as well as routing computations, while in other embodiments a separate dedicated communications processor may be provided.
  • different types of features or functionalities may be implemented in a system according to the invention that includes a client device (such as a tablet device or smartphone running client software) and server systems (such as a server system described in more detail below).
  • the system of the present invention may employ one or more memories or memory modules (such as, for example, remote memory block 16 and local memory 11) configured to store data, program instructions for the general-purpose network operations, or other information relating to the functionality of the embodiments described herein (or any combinations of the above).
  • Program instructions may control execution of or comprise an operating system and/or one or more applications, for example.
  • Memory 16 or memories 11, 16 may also be configured to store data structures, configuration data, encryption data, historical system operations information, or any other specific or generic non-program information described herein.
  • At least some network device embodiments may include nontransitory machine-readable storage media, which, for example, may be configured or designed to store program instructions, state information, and the like for performing various operations described herein.
  • nontransitory machine- readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD- ROM disks; magneto-optical media such as optical disks, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM), flash memory (as is common in mobile devices and integrated systems), solid state drives (SSD) and "hybrid SSD” storage drives that may combine physical components of solid state and hard disk drives in a single hardware device (as are becoming increasingly common in the art with regard to personal computers), memristor memory, random access memory (RAM), and the like.
  • ROM read-only memory
  • flash memory as is common in mobile devices and integrated systems
  • SSD solid state drives
  • hybrid SSD hybrid SSD
  • such storage means may be integral and non-removable (such as RAM hardware modules that may be soldered onto a motherboard or otherwise integrated into an electronic device), or they may be removable such as swappable flash memory modules (such as “thumb drives” or other removable media designed for rapidly exchanging physical storage devices), "hot-swappable” hard disk drives or solid state drives, removable optical storage discs, or other such removable media, and that such integral and removable storage media may be utilized
  • Examples of program instructions include both object code, such as may be produced by a compiler, machine code, such as may be produced by an assembler or a linker, byte code, such as may be generated by for example a JAVATM compiler and may be executed using a Java virtual machine or equivalent, or files containing higher level code that may be executed by the computer using an interpreter (for example, scripts written in Python, Perl, Ruby, Groovy, or any other scripting language).
  • object code such as may be produced by a compiler
  • machine code such as may be produced by an assembler or a linker
  • byte code such as may be generated by for example a JAVATM compiler and may be executed using a Java virtual machine or equivalent
  • files containing higher level code that may be executed by the computer using an interpreter (for example, scripts written in Python, Perl, Ruby, Groovy, or any other scripting language).
  • systems according to the present invention may be implemented on a standalone computing system.
  • Fig. 15 there is shown a block diagram depicting a typical exemplary architecture of one or more embodiments or components thereof on a standalone computing system.
  • Computing device 20 includes processors 21 that may run software that carry out one or more functions or applications of embodiments of the invention, such as for example a client application 24.
  • Processors 21 may carry out computing instructions under control of an operating system 22 such as, for example, a version of Microsoft's WINDOWSTM operating system, Apple's Mac OS/X or iOS operating systems, some variety of the Linux operating system, Google's
  • an operating system 22 such as, for example, a version of Microsoft's WINDOWSTM operating system, Apple's Mac OS/X or iOS operating systems, some variety of the Linux operating system, Google's
  • one or more shared services 23 may be operable in system 20, and may be useful for providing common services to client applications 24.
  • Services 23 may for example be WINDOWSTM services, user-space common services in a Linux environment, or any other type of common service architecture used with operating system 21.
  • Input devices 28 may be of any type suitable for receiving user input, including for example a keyboard, touchscreen, microphone (for example, for voice input), mouse, touchpad, trackball, or any combination thereof.
  • Output devices 27 may be of any type suitable for providing output to one or more users, whether remote or local to system 20, and may include for example one or more screens for visual output, speakers, printers, or any combination thereof.
  • Memory 25 may be random-access memory having any structure and architecture known in the art, for use by processors 21, for example to run software.
  • Storage devices 26 may be any magnetic, optical, mechanical, memristor, or electrical storage device for storage of data in digital form (such as those described above).
  • systems of the present invention may be implemented on a distributed computing network, such as one having any number of clients and/or servers.
  • a distributed computing network such as one having any number of clients and/or servers.
  • FIG. 16 there is shown a block diagram depicting an exemplary architecture 30 for implementing at least a portion of a system according to an embodiment of the invention on a distributed computing network.
  • any number of clients 33 may be provided.
  • Each client 33 may run software for implementing client-side portions of the present invention; clients may comprise a system 20 such as that illustrated above.
  • any number of servers 32 may be provided for handling requests received from one or more clients 33.
  • Clients 33 and servers 32 may communicate with one another via one or more electronic networks 31, which may be in various embodiments any of the Internet, a wide area network, a mobile telephony network (such as CDMA or GSM cellular networks), a wireless network (such as WiFi, Wimax, LTE, and so forth), or a local area network (or indeed any network topology known in the art; the invention does not prefer any one network topology over any other).
  • Networks 31 may be implemented using any known network protocols, including for example wired and/or wireless protocols.
  • servers 32 may call external services 37 when needed to obtain additional information, or to refer to additional data concerning a particular call.
  • External services 37 may take place, for example, via one or more networks 31.
  • external services 37 may comprise web-enabled services or
  • client applications 24 may obtain information stored in a server system 32 in the cloud or on an external service 37 deployed on one or more of a particular enterprise's or user's premises.
  • clients 33 or servers 32 may make use of one or more specialized services or appliances that may be deployed locally or remotely across one or more networks 31.
  • one or more databases 34 may be used or referred to by one or more embodiments of the invention. It should be understood by one having ordinary skill in the art that databases 34 may be arranged in a wide variety of architectures and using a wide variety of data access and manipulation means.
  • one or more databases 34 may comprise a relational database system using a structured query language (SQL), while others may comprise an alternative data storage technology such as those referred to in the art as "NoSQL” (for example, Hadoop Cassandra, Google BigTable, and so forth).
  • SQL structured query language
  • variant database architectures such as column-oriented databases, in-memory databases, clustered databases, distributed databases, or even flat file data repositories may be used according to the invention. It will be appreciated by one having ordinary skill in the art that any combination of known or future database technologies may be used as appropriate, unless a specific database technology or a specific arrangement of components is specified for a particular embodiment herein. Moreover, it should be appreciated that the term "database” as used herein may refer to a physical database machine, a cluster of machines acting as a single database system, or a logical database within an overall database management system.
  • security systems 36 and configuration systems 35 may make use of one or more security systems 36 and configuration systems 35.
  • Security and configuration management are common information technology (IT) and web functions, and some amount of each are generally associated with any IT or web systems. It should be understood by one having ordinary skill in the art that any configuration or security subsystems known in the art now or in the future may be used in conjunction with embodiments of the invention without limitation, unless a specific security 36 or configuration system 35 or approach is specifically required by the description of any specific embodiment.
  • FIG. 17 shows an exemplary overview of a computer system 40 as may be used in any of the various locations throughout the system. It is exemplary of any computer that may execute code to process data. Various modifications and changes may be made to computer system 40 without departing from the broader scope of the system and method disclosed herein.
  • Central processor unit (CPU) 41 is connected to bus 42, to which bus is also connected memory 43, nonvolatile memory 44, display 47, input/output (I/O) unit 48, and network interface card (NIC) 53.
  • I/O unit 48 may, typically, be connected to keyboard 49, pointing device 50, hard disk 52, and real-time clock 51.
  • NIC 53 connects to network 54, which may be the Internet or a local network, which local network may or may not have connections to the Internet.
  • power supply unit 45 connected, in this example, to a main alternating current (AC) supply 46.
  • AC alternating current
  • functionality for implementing systems or methods of the present invention may be distributed among any number of client and/or server components.
  • various software modules may be implemented for performing various functions in connection with the present invention, and such modules may be variously implemented to run on server and/or client

Abstract

L'invention concerne un système de collecte entièrement intégrée de données d'impact commercial, d'analyse de ces données et de génération aussi bien de décisions commerciales guidées par l'analyse que de simulations guidées par l'analyse d'autres actions commerciales candidates, comprenant un moteur de récupération de données commerciales stocké dans une mémoire d'un dispositif informatique et fonctionnant sur un processeur de ce dernier, un moteur d'analyse de données commerciales stocké dans une mémoire d'un dispositif informatique et fonctionnant sur un processeur de ce dernier, et un moteur de décision commerciale et de simulation de chemin d'action commerciale stocké dans une mémoire d'un ou plusieurs dispositifs informatiques et fonctionnant sur un processeur de ces derniers. L'invention concerne également un système et un procédé de capture, de stockage et d'analyse de données en série chronologique multidimensionnelles provenant de sources avec des profils de rapports hétérogènes. Des données provenant d'ensembles de capteurs qui envoient des quantités variables de données multidimensionnelles à intervalles irréguliers sont reçues par un dispositif de traitement de données, qui traite les données brutes pour extraire les paramètres présentant un intérêt, et conservées jusqu'à ce qu'un nombre prédéfini d'événements de capteur ou une quantité prédéfinie de données de flux aient été reçus. Les données sont ensuite validées dans un magasin de données jusqu'à un moment d'analyse. Des données plus anciennes sont écrites dans un dispositif d'archivage. Le système permet une sélection et une transformation complexes de données de magasin de données par un langage d'interrogation robuste.
PCT/US2017/026239 2016-04-05 2017-04-05 Système de capture entièrement intégrée et d'analyse d'informations commerciales aboutissant à une prise de décision et une simulation prédictives WO2017176944A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201780033615.6A CN109478296A (zh) 2016-04-05 2017-04-05 用于完全集成捕获和分析商业信息以产生预测决策和模拟的系统
EP17779786.7A EP3440569A4 (fr) 2016-04-05 2017-04-05 Système de capture entièrement intégrée et d'analyse d'informations commerciales aboutissant à une prise de décision et une simulation prédictives

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US15/091,563 2016-04-05
US15/091,563 US10204147B2 (en) 2016-04-05 2016-04-05 System for capture, analysis and storage of time series data from sensors with heterogeneous report interval profiles
US15/141,752 US10860962B2 (en) 2015-10-28 2016-04-28 System for fully integrated capture, and analysis of business information resulting in predictive decision making and simulation
US15/141,752 2016-04-28

Publications (1)

Publication Number Publication Date
WO2017176944A1 true WO2017176944A1 (fr) 2017-10-12

Family

ID=60000709

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2017/026239 WO2017176944A1 (fr) 2016-04-05 2017-04-05 Système de capture entièrement intégrée et d'analyse d'informations commerciales aboutissant à une prise de décision et une simulation prédictives

Country Status (3)

Country Link
EP (1) EP3440569A4 (fr)
CN (1) CN109478296A (fr)
WO (1) WO2017176944A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114000907A (zh) * 2021-12-10 2022-02-01 重庆邮电大学 一种基于数字孪生技术的矿井通风设备智能调控系统
US11571811B2 (en) 2019-10-15 2023-02-07 UiPath, Inc. Process evolution for robotic process automation and workflow micro-optimization

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10425353B1 (en) 2017-01-27 2019-09-24 Triangle Ip, Inc. Machine learning temporal allocator
CN111582488A (zh) * 2020-04-23 2020-08-25 傲林科技有限公司 一种事件推演方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050165822A1 (en) * 2004-01-22 2005-07-28 Logic Sight, Inc. Systems and methods for business process automation, analysis, and optimization
US7171515B2 (en) * 2001-05-22 2007-01-30 Fujitsu Limited Storage unit with improved performance by purging unnecessary data and transferring data with high probability of future hits to a segment buffer
US20140359552A1 (en) * 2011-09-19 2014-12-04 Tata Consultancy Services Limited Computer Platform for Development and Deployment of Sensor Data Based Applications and Services
US20160006629A1 (en) * 2013-07-07 2016-01-07 George Ianakiev Appliance clearinghouse with orchestrated logic fusion and data fabric - architecture, system and method

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9275059B1 (en) * 2011-11-07 2016-03-01 Emc Corporation Genome big data indexing
US9672283B2 (en) * 2012-06-06 2017-06-06 Data Record Science Structured and social data aggregator
US9734220B2 (en) * 2012-12-04 2017-08-15 Planet Os Inc. Spatio-temporal data processing systems and methods
CN103309990A (zh) * 2013-06-18 2013-09-18 上海晶樵网络信息技术有限公司 基于互联网用户公开信息的用户多维度分析与监测方法
CN103514301A (zh) * 2013-10-24 2014-01-15 深圳市同洲电子股份有限公司 分布式网络爬虫任务调度的方法及系统
CN104077402B (zh) * 2014-07-04 2018-01-19 用友网络科技股份有限公司 数据处理方法和数据处理系统
CN104182389B (zh) * 2014-07-21 2018-01-19 安徽华贞信息科技有限公司 一种基于语义的大数据分析商业智能服务系统
CN104966172A (zh) * 2015-07-21 2015-10-07 上海融甸信息科技有限公司 一种用于企业经营数据分析的大数据可视化分析处理系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7171515B2 (en) * 2001-05-22 2007-01-30 Fujitsu Limited Storage unit with improved performance by purging unnecessary data and transferring data with high probability of future hits to a segment buffer
US20050165822A1 (en) * 2004-01-22 2005-07-28 Logic Sight, Inc. Systems and methods for business process automation, analysis, and optimization
US20140359552A1 (en) * 2011-09-19 2014-12-04 Tata Consultancy Services Limited Computer Platform for Development and Deployment of Sensor Data Based Applications and Services
US20160006629A1 (en) * 2013-07-07 2016-01-07 George Ianakiev Appliance clearinghouse with orchestrated logic fusion and data fabric - architecture, system and method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3440569A4 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11571811B2 (en) 2019-10-15 2023-02-07 UiPath, Inc. Process evolution for robotic process automation and workflow micro-optimization
US11919165B2 (en) 2019-10-15 2024-03-05 UiPath, Inc. Process evolution for robotic process automation and workflow micro-optimization
CN114000907A (zh) * 2021-12-10 2022-02-01 重庆邮电大学 一种基于数字孪生技术的矿井通风设备智能调控系统

Also Published As

Publication number Publication date
EP3440569A4 (fr) 2019-12-11
CN109478296A (zh) 2019-03-15
EP3440569A1 (fr) 2019-02-13

Similar Documents

Publication Publication Date Title
US11295262B2 (en) System for fully integrated predictive decision-making and simulation
US11321085B2 (en) Meta-indexing, search, compliance, and test framework for software development
US11588793B2 (en) System and methods for dynamic geospatially-referenced cyber-physical infrastructure inventory and asset management
US11206199B2 (en) Highly scalable distributed connection interface for data capture from multiple network service sources
US20170124497A1 (en) System for automated capture and analysis of business information for reliable business venture outcome prediction
US10204147B2 (en) System for capture, analysis and storage of time series data from sensors with heterogeneous report interval profiles
US10860660B2 (en) Method and apparatus for crowdsourced data gathering, extraction, and compensation
US20210160288A1 (en) Highly scalable distributed connection interface for data capture from multiple network service and cloud-based sources
US20170124501A1 (en) System for automated capture and analysis of business information for security and client-facing infrastructure reliability
US11805106B2 (en) System and method for trigger-based scanning of cyber-physical assets
US11636549B2 (en) Cybersecurity profile generated using a simulation engine
US20210226927A1 (en) System and method for fingerprint-based network mapping of cyber-physical assets
WO2017176944A1 (fr) Système de capture entièrement intégrée et d'analyse d'informations commerciales aboutissant à une prise de décision et une simulation prédictives
US20220019451A1 (en) System and methods for creation and use of meta-models in simulated environments
WO2017205845A1 (fr) Système de capture et d'analyse automatisées d'informations d'entreprise
US11714991B2 (en) System and methods for creation of learning agents in simulated environments
US20180181914A1 (en) Algorithm monetization and exchange platform
US20230208820A1 (en) System and methods for predictive cyber-physical resource management
WO2019104312A1 (fr) Architecture de méta-indexation, de recherche, de conformité et de test permettant le développement de logiciel
US20230388277A1 (en) System and methods for predictive cyber-physical resource management
WO2019165384A1 (fr) Système et procédés pour inventaire d'infrastructure cyber-physique à référence géospatiale dynamique et gestion d'actifs

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2017779786

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2017779786

Country of ref document: EP

Effective date: 20181105

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17779786

Country of ref document: EP

Kind code of ref document: A1