WO2020086851A1 - Système et procédé de fourniture de sciences de données en tant que service - Google Patents

Système et procédé de fourniture de sciences de données en tant que service Download PDF

Info

Publication number
WO2020086851A1
WO2020086851A1 PCT/US2019/057870 US2019057870W WO2020086851A1 WO 2020086851 A1 WO2020086851 A1 WO 2020086851A1 US 2019057870 W US2019057870 W US 2019057870W WO 2020086851 A1 WO2020086851 A1 WO 2020086851A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
contest
participants
prediction
contests
Prior art date
Application number
PCT/US2019/057870
Other languages
English (en)
Inventor
Peter Cotton
Original Assignee
Jpmorgan Chase Bank, N.A.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/172,212 external-priority patent/US11562382B2/en
Application filed by Jpmorgan Chase Bank, N.A. filed Critical Jpmorgan Chase Bank, N.A.
Publication of WO2020086851A1 publication Critical patent/WO2020086851A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201—Market modelling; Market analysis; Collecting market data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201—Market modelling; Market analysis; Collecting market data
    • G06Q30/0202—Market predictions or forecasting for commercial activities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/06—Asset management; Financial planning or analysis
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/53—Network services using third party service providers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/12—Computing arrangements based on biological models using genetic models
    • G06N3/126—Evolutionary algorithms, e.g. genetic algorithms or genetic programming

Definitions

  • the present invention relates generally to data science, and more particularly to a system and method for providing data science as a service (DSaaS) using data prediction contests.
  • DSAaS data science as a service
  • Historical data competition model is often not an effective approach. Historical data competitions also tend to be slow moving and thus they generally do not provide the sponsor of the project with a useful predictive model in an expedited time frame.
  • Exemplary embodiments of the invention provide a new style of prediction contest in which participants build models (e.g., published as web services) that respond in real- time to a sequence of questions of a pre-defmed type.
  • Successful ideas in machine learning can motivate design patterns for combinations of these real-time prediction contests, with an objective of furthering forecasting, industrial control and decision making in general.
  • the invention comprises a method of
  • the method may comprise: (a) transmitting a live stream of questions in a pre-defmed format from a central server on behalf of a contest sponsor seeking real-time answers; (b) receiving real-time responses from a plurality of contributed web services generated by participants; and (c) recording answers in real time for offline or online scoring, aggregation of estimates, and deferred compensation to participants.
  • a preferred embodiment of the invention comprises prediction of future values of a source of updating data.
  • the invention relates to a computer-implemented system and method for providing data science as a service (DSaaS) using a real time data prediction contest.
  • the method may be executed on a specially programmed computer system comprising one or more computer processors, electronic storage devices, and networks.
  • the method may comprise the steps of: presenting a consumer interface via a network, wherein the consumer interface allows a data consumer to (a) identify a subject data source having data fields that can be predicted, (b) specify at least one data field to be predicted, and (c) specify timing constraints and cost constraints on the prediction of the data fields; presenting a participant interface via the network, wherein the participant interface (y) allows a participant to select a competition in which to participate and (z) provides a web services interface that enables the participant to provide web services via the web services interface, wherein the web services can be called by the web services interface, the web services provide a prediction of the at least one data field, and the web services are permitted to utilize external information sources in generating the prediction; and transmitting, by the consumer interface, the prediction to the data consumer via the network.
  • the invention relates to a computer- implemented method for providing data science as a service (DSaaS) using a real time data prediction contest.
  • Participants in the real time data prediction contest are permitted to execute and submit algorithms, utilize third party data sources, and utilize sub-contests to generate data predictions for the data prediction contest.
  • the participants in the data prediction contest may be humans or software robots.
  • a category of sponsor confidential information related to the data prediction is defined and maintained as confidential by the sponsor, while various methods are implemented to obtain relevant algorithms and data for the data prediction.
  • the sponsor receives data predictions from the participants on a real time or near real time basis, calculates a score for the data predictions, and compensates participants according to their score.
  • the invention also relates to computer-implemented system for providing data science as a service, and to a computer readable medium containing program instructions for executing a method for providing data science as a service.
  • Exemplary embodiments of the invention can provide a number of advantages to a business or organization in need of predictive data analytics.
  • the DSaaS system can provide an inexpensive but accurate way to predict any stream of data points by adopting the prediction protocol as described herein.
  • the DSaaS system can improve quality and reduce costs for the system operator and its data consumers’ business processes.
  • the DSaaS system can take advantage of a significant and growing community of data scientists or other analysts or individuals who are willing to compete and build services that benefit the requesting data consumers.
  • the DSaaS system can efficiently identify the most relevant machine learning, statistical algorithms, and relevant data for a given task, avoid time consuming in-house iteration, and reduce time to market.
  • the invention can produce an interwoven mesh of searchable, real-time forecasts that collectively map a wide variety of future events and values. Participants have the ability to add more commercial value as compared with traditional historical data science contests by maintaining live algorithms.
  • exemplary embodiments of the invention can present participants with an opportunity to provide relevant data, not just models.
  • exemplary embodiments of the invention can be understood to comprise an ultra-lightweight market whose price discovery provides an alternative to in-house, quantitative development.
  • inventions of the invention can provide a streaming data feed registry, real- time model deployment, and request-response delivery of results for easy integration into applications (e.g., a prediction API), resulting in decreasing marginal costs for prediction.
  • applications e.g., a prediction API
  • Sub- contests, feature markets and meta-contests facilitate separation of concerns.
  • Embodiments of the invention can provide fast, seamless, self-service contest creation via a web dialog that can convert any updating source of data in to a contest specification.
  • the network can rapidly grow to be a network for prediction constituting an essential sub-stratum for pricing, inventory management, logistics, recommendation and recognition, for example, across diverse industries.
  • Figure l is a diagram of a system for providing competitive data science as a service (DSaaS) according to an exemplary embodiment of the invention.
  • Figure 2 illustrates an example of landing web page for the DSaaS system according to an exemplary embodiment of the invention.
  • Figure 3 shows an example of a data consumer interface of the DSaaS system according to an exemplary embodiment of the invention.
  • Figure 4 depicts an example of a data consumer interface that allows a data consumer to specify the data fields to be predicted according to an exemplary embodiment of the invention.
  • Figure 5 is an example of a data consumer interface that allows the data consumer to specify timing and cost parameters according to an exemplary embodiment of the invention.
  • Figure 6 depicts an example of a data consumer interface of the DSaaS system that allows the data consumer to specify additional timing parameters according to an exemplary embodiment of the invention.
  • Figure 7 is an example of a data consumer interface that provides a number of interface options for web services provided by participants according to an exemplary embodiment of the invention.
  • Figure 8 illustrates an example of a competition browser for a participant interface according to an exemplary embodiment of the invention.
  • Figure 9 is an example of a leaderboard that ranks data science contests or participants by certain variables according to an exemplary embodiment of the invention.
  • Figure 10 is a drawing illustrating an example of the real-time provision of data science as a service according to an exemplary embodiment of the invention.
  • Figure 11 illustrates an algorithm for aggregating probabilistic options and compensating participants based on accuracy, according to an exemplary embodiment of the invention.
  • Figure 12 is a table providing examples of methods to defend against misuse of data supplied to participants in a real-time prediction contest according to an exemplary embodiment of the invention.
  • Figure 13 illustrates an indirect approach for producing an inference model from a generative model according to an exemplary embodiment of the invention.
  • Figure 14 illustrates a method in which public inference contests bait good approximate inference algorithms, according to an exemplary embodiment of the invention.
  • Figure 15 is a table illustrating question parameters and interpretation for a bond volume prediction contest according to an exemplary embodiment of the invention.
  • Figure 16 is a table that includes a listing of contests, relative commercial sensitivity of the data streams involved, and techniques employed according to an exemplary embodiment of the invention.
  • Figure 17 illustrates an example of FINRA TRACE reporting of corporate bond trades according to an exemplary embodiment of the invention.
  • data science and predictive data analytics often can be conducted more efficiently by searching for existing external data analytics resources (e.g., computer models) than by building analytics capabilities in-house.
  • data science can be procured efficiently via searching external resources in the space of computer models, meta-parameters, relevant correlated external data sources, data cleaning mechanisms, compute technologies, caching, state management and storage mechanisms, for example. Searching for and obtaining these components of predictive data analytics can provide significantly enhanced efficiencies as compared with developing comparable resources in-house. Forecasting and recommendations can be delivered by a large pool of talent, worldwide, rather than only by an organization’s internal data analytics resources.
  • the invention outsources model searching by providing a data science as a service (DSaaS) system for real-time data science competitions.
  • the DSaaS system can turn a significant liability of known historical data competitions (use of outside data, which permits cheating) into a key asset, by encouraging participants to discover relevant exogenous variables (e.g., external data sources) and use them.
  • the DSaaS system encourages participants to deliver true value to data consumers by launching web services to answer a multitude (e.g., thousands) of questions the DSaaS system relays in real-time, but the DSaaS system provides these participants with a model deployment platform in which this additional step can be accomplished in an expedited time frame, e.g., in minutes.
  • One embodiment for simultaneously crowd-sourcing both model selection and data gathering comprises a lightweight extension of the programmable web, involving the following components.
  • a real-time contest can be created from any source of updating data. The task before participants is that of predicting the future of this data.
  • a request for a forecast is sent to participants’ web services, who reply within a very short period of time, e.g., a few seconds or less.
  • the DSaaS system can create a new, efficient workflow that can be exploited by leading companies and organizations and their customers.
  • participants e.g., web-based software robots
  • Businesses looking to optimize their operations can have a one stop shop in which forecast models tailored to their specific workflows can be developed in an expedited manner with high reliability and virtually zero or negligible time to market, for example.
  • all that is required from the data consumer is a sequence of questions and/or solutions.
  • the DSaaS system can enable computer model search to go the way of internet search, i.e., enabling it to be simple, inexpensive and immediate for the data consumer.
  • the DSaaS system can provide an inexpensive but accurate way to predict any defined data points by adopting a“prediction protocol.”
  • the DSaaS system can improve quality and reduce costs for the system operator and its data consumers’ business processes.
  • the DSaaS system can take advantage of a significant and growing community of data scientists or other analysts or individuals who are willing to compete and build services that benefit the requesting data consumers (also sometimes referred to herein as the“sponsors” of the real time data prediction contests).
  • the DSaaS system can efficiently identify the most relevant machine learning, statistical algorithms, and relevant data for a given task, avoid time consuming in-house iteration, and reduce time to market.
  • the DSaaS system can provide these advantages because, in part, it includes a front end interface making the system accessible and user-friendly to data consumers and participants having little or no computer programming expertise.
  • the DSaaS system makes it easy for data scientists (participants) to launch a REST API service, for example, which receives, manipulates and responds to questions using algorithms they develop themselves.
  • the participant finds a question/answer stream of interest on the system’s participant interface and devises an approach.
  • the participant writes a function in a language such as Python, R, Scripte or Julia that takes a standardized question and returns an answer.
  • the participant interface allows the participant to easily back-test the function and launch the web services.
  • the DSaaS system pays the participant a stream of income if the participant rises up the leaderboard, thus demonstrating the utility of the offered solution.
  • the DSaaS system can be configured to allow a participant to offer to sell intellectual property rights to his or her algorithms and code to the data consumer (e.g., the sponsor of the contest) or to other participants. This feature allows a participant to monetize his or her cash flows immediately. It may also satisfy certain
  • Figure l is a diagram of a system for providing data science as a service (DSaaS) according to an exemplary embodiment of the invention.
  • the system may include one or more networks and one or more computing devices, such as servers and personal computers.
  • the DSaaS system may be operated by a business, a data science company, or other type of organization that seeks to identify requests for data science contests or predictive data analytics from its“data consumers” (e.g., sponsors of the real-time contest), solicit data science solutions (e.g., predictions of certain identified data fields) from a community of“participants” (e.g., data scientists, analysts, or other individuals), and provide some or all of those solutions to its data consumers.
  • “data consumers” e.g., sponsors of the real-time contest
  • solicit data science solutions e.g., predictions of certain identified data fields
  • a community of“participants” e.g., data scientists, analysts, or other individuals
  • the DSaaS system may be embodied primarily in a server
  • the server 120 owned and/or operated by the company or organization providing the service (the“DSaaS provider”).
  • the server 120 may be operated by a data science contest manager 121 at the DSaaS provider using a personal computing device such as a laptop computer 122
  • the manager 121 may be responsible for executing a particular data science contest or series of contests.
  • the server 120 may also be accessed by one or more data analysts 123 using their personal computing devices such as laptop computer 124
  • the data analyst 123 may access the DSaaS system and associated data and results to evaluate the accuracy and utility of various predictive models or results provided by participants in a data science contest, for example.
  • the DSaaS system 120 communicates with other systems via a network 110 as shown in Figure 1.
  • the network 110 may comprise any one or more of the Internet, an intranet, a Local Area Network (LAN), a Wide Area Network (WAN), an Ethernet connection, a WiFi network, a Global System for Mobile Communication (GSM) link, a cellular phone network, a Global Positioning System (GPS) link, a satellite communications network, or other network, for example.
  • LAN Local Area Network
  • WAN Wide Area Network
  • Ethernet connection a WiFi network
  • GSM Global System for Mobile Communication
  • GSM Global System for Mobile Communication
  • GPS Global Positioning System
  • the data consumers may be businesses or other organizations that need data predictions and predictive data analytics delivered to them in a cost effective and timely manner, such as business intelligence, financial predictions, scientific predictions, and/or automation.
  • businesses or organizations that are data consumers.
  • One particular example might be a financial institution that needs to predict the trading volume of a security such as a particular bond over a future period of time.
  • Other examples may include an electric utility that needs to predict demand for electricity during various defined future time periods, an online retailer that needs to predict demand for various products and services it offers during defined time periods, a bicycle sharing service such as www.citibikenvc.com that has a need to predict the number of bikes available at various stations, and a business that needs to predict sales conditional on a hypothetical choice of price.
  • an objective of the business would be to capture the price optimization category, such as predicting movie sales according to discounts offered, hotel room bookings according to price, or even the number of candy bars sold by a vending machine conditional on a hypothetical choice of price.
  • Figure 1 depicts two such data consumers.
  • a first data consumer such as a retailer may have an analyst 141 who interacts with the DSaaS system 120 via a personal computing device 140 and corporate server 142, for example.
  • a second data consumer such as a bicycle sharing service may have an analyst 145 who interacts with the DSaaS system 120 via a personal computing device 144 and corporate server 146.
  • Many other examples of data consumers and types of predictive data exist for different retailers, service providers, financial institutions, healthcare providers, government agencies, and other organizations and individuals.
  • Figure 1 also depicts participants who provide data science solutions, modeling, and/or predictions to the DSaaS system according to exemplary embodiments of the invention.
  • the participants may be data scientists, analysts in a particular industry, or simply individuals who have knowledge or experience in predictive data analytics and who are interested in providing proposed solutions (e.g., predictions of certain data fields) to the DSaaS system in exchange for potential compensation based on the effectiveness of their proposed solution or data prediction.
  • Figure 1 depicts a first participant 161 who may interact with the DSaaS system via their personal computing device 160 and the network 110, for example.
  • the first participant 161 may also arrange to make his or her predictive analytic solution continuously available by uploading it to a server 162, e.g., a cloud server of a cloud computing service such as Amazon Web Services (AWS).
  • a server 162 e.g., a cloud server of a cloud computing service such as Amazon Web Services (AWS).
  • Figure 1 depicts a second participant 165 who may interact with the DSaaS system via his or her personal computing device 164 and the network 110.
  • the second participant 161 may also arrange to make his or her predictive analytic solution continuously available by uploading it to a server 166.
  • the third party servers 182, 186 represent external data sources that participants can draw upon and incorporate as input to their models.
  • the participant 165 may create a web service that predicts the number of bicycles available at a particular bike sharing station.
  • the participant may create a REST API that automatically queries weather station data maintained by a third party server 182 (because the weather may influence the likelihood that people will rent bikes).
  • the participant may also create a REST API that automatically queries traffic data maintained on another third party server 186 (because heavy traffic may influence the likelihood that people will rent bikes). The participant can therefore improve his or her predictive capability by drawing upon third party data using an appropriate interface and query.
  • FIG. 1 is merely one example of a DSaaS system configuration and its interfaces with other systems and is not intended to be limiting.
  • FIG. 1 is merely one example of a DSaaS system configuration and its interfaces with other systems and is not intended to be limiting.
  • FIG. 1 is merely one example of a DSaaS system configuration and its interfaces with other systems and is not intended to be limiting.
  • FIG. 1 is merely one example of a DSaaS system configuration and its interfaces with other systems and is not intended to be limiting.
  • other types and configurations of networks, servers, databases and personal computing devices e.g., desktop computers, tablet computers, mobile computing devices, smart phones, etc.
  • Figure 2 illustrates an example of a landing web page of the DSaaS system providing an interface between the DSaaS system, the data consumers and the participants.
  • a data consumer may begin under the header entitled“Define the Future” where the web page or software application (“app”) indicates that the consumer can choose a web page or other subject data source that updates and have the crowd (participants) predict updating numbers.
  • the data consumer can click on the button 222 entitled“SHOW ME HOW” to get started.
  • the participant such as a data scientist, analyst, or individual, can get started under the heading entitled“Predict the future” 230.
  • the web page or app indicates that the participant can create a prediction hot and start earning money. It also includes a search bar 232 to search available data competitions, a number of icons 234 that the participant can click on to browse various available data competitions, and an indication 236 of the total amount of prize money available.
  • the web page also includes a button 238 entitled“SHOW ME HOW” to enable the participant to obtain detailed information on the rules and resources available for participating in a data science competition.
  • the data consumer can begin, for example, by identifying a subject data source containing the one or more data fields to be predicted.
  • the subject data source may comprise any identifiable data source such as a website or a table in a database, for example.
  • the subject data source may include data fields that are updated periodically, according to a preferred embodiment of the invention.
  • the data consumer may identify the subject data source with an address, e.g., by inputting a uniform resource locator (URL) for a website that includes the data fields to be predicted.
  • URL uniform resource locator
  • Figure 3 illustrates an example of a consumer interface 300 provided by the DSaaS system as a web page or app in which the consumer has identified http://citibike.mobi/ in the address bar 310 for the URL.
  • the consumer interface 300 also includes a progress bar 312 that illustrates for the data consumer how far along he or she is in the process of defining the data fields to be predicted.
  • One of the advantages that various embodiments of the invention can provide is a process that is simple and expedient for both the data consumer and the participant, thus enhancing the likelihood that data consumers and participants will use and rely on the DSaaS system.
  • FIG 3 illustrates an example of the structured data 314 (JSON in this example) that is provided by the subject data source.
  • the Citibike website broadcasts the number of available bikes and free bike docks at New York City bike stations, together with static attribute data such as the name, latitude and longitude of the bike station.
  • This web page is public and maintained by the bike sharing program in order that developers can access the data in a convenient programmatic manner.
  • the page updates at regular intervals, approximately every minute, and thus provides real-time time series data of interest to a large number of interested parties.
  • the current value of this data would be consumed by application developers, but according to exemplary embodiments of the present invention, a substantially identical page is maintained by the DSaaS system comprising future estimates of these quantities, e.g., 15, 30, 60, and/or 90 minutes in the future.
  • the predicted data is made available to the data consumer that requested it, according to the format, content, interface, and timing requirements specified by the data consumer according to one embodiment of the invention.
  • the data consumer pays compensation to the DSaaS system that it agreed to pay when requesting the predicted data.
  • the DSaaS system compensates the participants based on the quality score of their predictions, as will be discussed further below.
  • Figure 4 depicts an embodiment of a consumer interface 400 provided by the
  • the website containing data fields to be predicted is the Citibike website that is used to manage the bike share program in New York.
  • the consumer interface 400 includes an address bar 410 in which a consumer can specify an address such as a URL or other identifier of the data source containing the one or more data fields to be predicted.
  • the consumer interface 400 includes a progress bar 412 showing the relative progress of the consumer in defining the data fields to be predicted.
  • the consumer interface 400 also includes a table 414 that lists one or more attributes, examples of the attributes, data types for each attribute, and check boxes to allow a consumer to indicate whether the attribute is an index value, a value to be predicted, or an attribute that is fixed.
  • the table 414 provides the attributes, including station name, number of available docks, total number of docks, latitude and longitude of the station, and the number of available bikes.
  • the website allows the data consumer to specify with check boxes which data fields are to be predicted, which data fields are attributes, and which data fields constitute an index.
  • the data consumer has used the check boxes to specify that the station name (Franklin St & W Broadway) is to be used as the index, the latitude and longitude are to be treated as attributes, and the available docks, total docks, and available bikes are the data fields to be predicted.
  • the submit button 416 can click the submit button 416 to submit the designations to the DSaaS system.
  • the process that a data consumer undertakes to choose a subject data source of interest and to define the data fields to be predicted in that subject data source can be completed in a short time frame and is easy to understand. This enhances the likelihood of use of the DSaaS system by potential data consumers who are not necessarily tech savvy or well versed in computer languages.
  • the DSaaS system allows the data consumer to define the desired data predictions with additional parameters and constraints.
  • the consumer interface 500 includes a sliding button 518 that allows the data consumer to specify how far into the future the data consumer would like the data predictions. Multiple buttons 518 can be included to request predictions at multiple points in the future.
  • the interface 500 includes a button bar 520 that allows the data consumer to specify how often the page should be updated.
  • the consumer interface 500 also allows the consumer to indicate with a sliding button 522 the amount of money it is willing to pay per day for access to the data predictions it has requested. Once the data consumer is happy with the selections, he or she can click on the submit button 516.
  • Figure 6 is an illustration of a consumer interface providing additional functionality to further define the predictions that are desired.
  • the consumer interface provided by the DSaaS system allows the consumer to specify custom timing options, such as the time of day to start predictions 614, the time of day to stop making predictions 616, the days of the week to make predictions 618, whether to make data predictions on US public holidays 620, and custom exclusion days 622.
  • the data consumer can click the “Next” button 624 after making his or her selections.
  • the data prediction needs defined by the data consumer using the consumer interface may be used by the DSaaS provider to create a data science contest. Using the functionality shown in Figures 3-6, the data consumer is able to quickly and easily indicate in detail the data fields it would like to see predicted, the exact times during which predictions are desired, and the amount of expenses it is willing to pay per day.
  • a non programmer can specify quantities to be predicted by the crowd (participants) in a manner that does not require programming. This is because web pages or other sources of updating data (often but not always in tabular format) are implicitly time series. However, not all of the numbers, strings, categorical or other data reported on said sources of data need be the target for prediction.
  • Some data, as with the name of a Citibike location, is a static attribute and can be considered a way of indexing (parametrizing) the precise question to be asked.
  • Other data such as the latitude, is considered ancillary information - possibly of interest or use to forecasting but not essential to the definition of the task for the crowd.
  • a non technical user can be guided through a dialog, as indicated in 414, in order to instruct the system which quantities are to be predicted, which are to be used to index distinct predictions, and which are merely ancillary attributes.
  • a means of interrogating the user for forecast horizon and times at which predictions should occur is shown and in this example the data consumer is instructing the DSaaS system that they wish to trigger predictions every time the page updates and that contestants (participants) should be judged based on the values shown in the page fifteen minutes henceforth from those trigger times. It will be apparent to one skilled in the art that other means of soliciting these preferences are possible.
  • FIG 522 an embodiment of a slider is illustrated by which the data consumer determines how much money to spend on a per diem basis.
  • examples of additional customizations a data consumer might achieve are provided, again without programming.
  • the data consumer user is able to instruct the system that they are only interested in predictions between a supplied start time and end time of day.
  • the data consumer is able to instruct the system of weekday, holiday and day of month preferences, thus allowing them to run the contest on only Sundays, for example.
  • other ways of providing scheduling information are possible, including the provision of a dialog box permitting the user to enter terse but more powerful scheduling syntax such as the CRON format.
  • the DSaaS system can provide the data consumer with information that assists him or her with obtaining and using the results of the data science contest when they begin to be available.
  • Figure 7 shows one example of the type of information that the DSaaS system can provide to the data consumer.
  • the DSaaS system includes a consumer interface that has a number of different methods/interfaces for accessing the results of the data science contest.
  • the data consumer may access the results via web page 710, REST API 712, commercially available software 714, charts 716, and apps 718.
  • Web page access may be provided using languages such as JSON, HTML table, XML, RDF, BigQuery, CKAN, OData and SQL, as shown in Figure 7.
  • REST API access may be provided using languages such as Bash, Python, Ruby, PHP, Java and Go.
  • Popular software may include Excel, Tableau, and SAS.
  • Available charts may include D3 and Plotly.
  • iPhone and Android apps may also be provided.
  • the consumer interface may also include a button 720 allowing the data consumer to submit a request to the DSaaS system for additional interfaces.
  • the DSaaS system provides many options to a data consumer for automatically interfacing to solutions provided by the DSaaS system.
  • the data consumer can program its own systems (e.g., 142, 146 in Figure 1) to call or query the DSaaS system 120 at predetermined times using such interfaces.
  • the DSaaS system may publicize the data science contest on a participant interface so that the community of participants has the ability to begin devising models and producing web services that are responsive to the data science contest.
  • Figure 8 illustrates an example of a competition browser that is part of the participant interface.
  • the competition browser lists data competitions that are available to participants wishing to submit proposed solutions (e.g., predicted data derived from software models).
  • the competition browser 800 may be configured to provide a list of categories and subcategories of data competitions.
  • the categories may include Finance, Medicine and Real Estate. Under each category may be listed at least one level of subcategories. For example, under the Finance category is listed Volume prediction, Earnings announcements and Entitlements prediction. Similar subcategorization is shown in Figure 8 for the Real estate category. Additional levels of subcategorization may also be implemented.
  • the competition browser 800 may also show the amount of prize money available for each category and subcategory. This configuration allows potential participants to easily identify competitions in which they may be interested in submitting a response based on subject matter and potential income.
  • a participant selects a data science competition in which to participate, he or she can then begin to create a web service that automatically interfaces to the DSaaS system to provide predictions at predetermined times and/or in response to queries from the DSaaS system.
  • the DSaaS system preferably includes a web services interface that is a component of the participant interface and that makes the DSaaS system accessible and user-friendly to
  • Azure (azure.microsoft.com), and Google (cloud.google.com), for example.
  • the participant uses RESTful web services to provide his or her predictions of the data field(s) as requested by the data consumer.
  • the web service may comprise a REST API service that receives, manipulates, and responds to queries using a software-based model developed by the participant. The participant finds a
  • question/answer stream of interest on the DSaaS system s participant interface and devises an approach, e.g., builds a software model or function.
  • the participant may write a function or model in Python, R, Scripte, Julia, or other programming language, for example, that takes a standardized question and returns an answer.
  • the RESTful web services may utilize an http protocol, for example.
  • the web services can employ a software-based model and can be hosted by the participant and identified by a uniform resource identifier (LIRI) or uniform resource locator (LIRL).
  • the web services can also be programmed to call other web services (e.g., third party data services provided via a third party server) in use of the software-based model.
  • the web services provide the prediction of the at least one data field in real time or near real time to the DSaaS system.
  • the participant can back-test the function or model underlying the web services with one click according to an exemplary embodiment of the invention.
  • the web services interface that is a component of the participant interface may allow the participant to identify the web services with a LIRL and to back-test the function on a known data set by pressing a“back-test” button. This functionality allows the participant to more easily evaluate the accuracy of his or her model prior to submission of data predictions to the DSaaS system.
  • the participant can also launch the web service with one click according to an exemplary embodiment of the invention. For example, after the participant is satisfied with the model’s performance, he or she can click a“submit” button on the web services interface to allow the web services interface to thereafter call the participant’s web services using a predefined protocol.
  • a“submit” button on the web services interface to allow the web services interface to thereafter call the participant’s web services using a predefined protocol.
  • the participants are permitted to use third party data sources in providing their web services.
  • This feature can provide significant advantages over known historical data science competitions in which the training data and test data are carefully defined and limited. According to exemplary
  • the participant can identify and use external data sources in innovative ways to improve the performance of his or her predictive model. That is, the participant can provide web services that provide predictive data that is generated with both (1) a model or function created by the participant and (2) one or more external data sources identified by the participant that are used as input to the model.
  • the DSaaS system obtains the benefits of external models created by participants as well as innovate identification and use of external data sources used in the participants’ models.
  • the external data sources fed into the model can also provide the benefit of real time information for more accurate predictions.
  • the participants who provide the web services also manage other aspects of the model and input data.
  • the participant creates the model, hosts the model, finds and uses external data sources, maintains and improves the model and data sources, manages related aspects of the process such as data storage, and uses the model and data to provide web services that provide predicted data in real time or near real time to the DSaaS system upon request or at predetermined times.
  • the participant after using and observing the model over time, may modify the model to improve its accuracy, or may change the input data sources.
  • the participant may reevaluate the relevance of the model or components of the model, may change the model computations, or may otherwise maintain the model over time.
  • the participant may also make certain decisions regarding the type and amount of data to store for the model and how and where to store the data, including considerations as to data compression and state management.
  • the participant also has the ability to combine and reuse models according to exemplary embodiments of the invention.
  • the participant may utilize a software model that has been previously used in a data science competition, e.g., an historical data science competition in which only a defined, public data set is permitted to be used and no external information sources are permitted to be used.
  • the participant may also combine different models to improve predictive performance, such as by calling other models with web services, as will be discussed further below.
  • the participant’s web services can be designed to automatically search for relevant third party algorithms to be used in providing the data predictions to the DSaaS system.
  • the participants can design the web services so that they provide forecasts for a plurality of contingent scenarios.
  • Real-time competitions can be used to assess action-conditional outcomes.
  • the participants predict the results of every action that might be taken, and choose the best action. The result of that action should be quantifiable.
  • One example is baseball pitch selection.
  • exemplary embodiments of the invention can be used to make real-time decisions based on forecasts for a plurality of contingent scenarios.
  • the data predictions from a number of participants can be aggregated to provide an improved prediction.
  • the DSaaS system records all predictions made by all participants and performs both an allocation of points to participants (governing subsequent payments made to them - this is done ex post when the true quantities are revealed by the passage of time) and an aggregation of forecast opinion into a single number or collection of numbers.
  • both the scores allocated to participants pertaining to a fixed time interval and the weights used to combine participants’ forecasts into a single forecast coincide, and are equal to the inverse of the mean square error of participants’ entries.
  • Other aggregation methods can be used to aggregate data provided by multiple participants according to an exemplary embodiment of the invention, as will be discussed further below. In fact, the aggregation method itself can be the subject of a data science competition.
  • participant supply probabilities for a finite number of discrete, mutually exclusive and collectively exhaustive outcomes.
  • weights assigned to participants’ forecasts are initially equal, but thereafter are adjusted up or down based on a scoring system modeled after the pari-mutuel system employed at racetracks.
  • Figure 9 is an example of a leaderboard that can be generated and provided by the
  • the leaderboard 900 lists the names of the leading data science contests in a particular data science competition.
  • the leaderboard 900 also provides related information for each data science contest, such as reputation (a measure of a participant’s overall performance in all contests, not just the current contest), score (for the current contest over the current time window - for instance going back one month), uptime (the percentage of questions they have answered during said time), share (of total compensation paid out), host (the location where participant’s solution resides),“buy now” (the participant’s decision whether to sell its IP rights to its solution), or exclude (a check-box the sponsor of the competition can use to ignore certain participants).
  • the leaderboard can be configured to allow participants to sort the leaderboard by any of the foregoing variables. In the Figure 9 example, the contests are ranked by share.
  • the DSaaS system can be configured to automatically make payments to participants, e.g., by direct deposit, PayPal deposit, or other payment process, based on a predefined criteria, such as the participant’s score, the total amount of compensation authorized by the requesting data consumer, and the applicable time period.
  • the payment terms, including amount, criteria, and timing for payments, are provided to each participant at the beginning of the data science competition according to a preferred embodiment.
  • FIG. 10 is a drawing that depicts particular examples of use of the DSaaS system according to exemplary embodiments of the invention.
  • the data consumers may include organizations in need of business intelligence, financial predictions, scientific predictions, and/or automation, for example.
  • the data consumer is interested in obtaining a prediction of the trading volume for a particular bond (e.g., CUSIP 033863ab6) over the next two weeks.
  • the data consumer can submit this real-time question by identifying the website (subject data source) and data field where this data is posted using the consumer interface as described above.
  • the DSaaS system 120 makes a real- time request to participants by creating a data science contest and posting it on a competition browser, such as the one shown in Figure 8. Participants can begin generating real-time responses using data science platforms and technologies such as Domino
  • the DSaaS system can provide other related functionalities, such as aggregation, performance, leaderboards, forums and payments.
  • Aggregation refers to the process of combining participants’ forecasts into a single number or other summary information (such as lowest quartile and highest quartile) as will be discussed below.
  • Performance refers to calculation of a performance metric for each proposed solution submitted by a participant according to predetermined, agreed upon criteria. The performance metric may be used in calculating each participant’s compensation for a particular data science contest. Some examples of the calculation of the performance metric (score) are discussed above.
  • the method involves assigning a point to any participant finishing in the top decile. It will be apparent to those skilled in the art that other methods are possible and that this decision might be left to the data consumer.
  • Leaderboards such as the example shown in Figure 9, provide data on the leading contests and may be ranked by different variables. Forums provide a communication platform whereby participants and data consumers can exchange ideas and data relating to data science
  • Real-time data prediction contests can merge the model selection capability of historical data contests with the strengths of live trading markets.
  • Participants in a real-time data prediction contest generally must maintain a real, live web service or by other means answer questions with little delay.
  • participants are able to utilize various commercial and open source solutions that facilitate the implementation of a coded algorithm as a REST endpoint.
  • An advantage of exemplary embodiments of the invention is that it allows participants to pass through more economic value to an end user of prediction than they are able to in a traditional historical data contest.
  • real-time data prediction contests In contrast to historical prediction contests, the answers to real-time contests lie in the future.
  • This aspect of real-time data prediction contests completely eliminates a variety of data leakage present in historical contests.
  • some participants in historical contests may use exogenous data causally connected with (or in some cases identical to) the contest data.
  • Use of the exogenous data in an historical contest can make the models submitted by the participants worthless.
  • Exemplary embodiments of the invention involving real time prediction contests do not have this disadvantage because use of exogenous data is a desirable objective that can enhance the predictive capability of the participant’s solution.
  • Real-time prediction contests free participants to use whatever ingenuity they possess in finding anything that can help them make better predictions, whether that means locating real-time sources of data or creating them from scratch.
  • real time data prediction contests Another advantage that can be provided by real time data prediction contests is the re-usability of contests.
  • One contest can be used as a regressor for another, opening up options for combining and integrating solutions, as will be described further below.
  • real-time contests can be described as a web of interrelated, bi-temporally indexed random variables available for use in the present without any modification: a canonical term structure for any desired prediction, as defined by the desired domain of application, e.g., a set of problems for which there is sufficient temporal or cross-sectional data that differentiation between good models and bad is possible on a reasonable time scale.
  • IoT internet of things
  • Another advantage that can be provided by exemplary embodiments of the invention is the ability to fragment tasks. Prediction tasks can be fragmented into smaller domains to improve accuracy and participation. For example, a contest to predict a stock market index over a long horizon would likely fall into the realm of traditional human oriented prediction markets and have no clear resolution. On the other hand, a contest to predict sales of individual items in hundreds of stores would be much more likely to unearth strong modeling approaches and sources of exogenous data. Hence, exemplary embodiments of the invention can be utilized to encourage the generation of strong models and data sources by fragmenting the tasks to a manageable size. [0081] Other advantageous characteristics that can be provided by real time data prediction contests are timeliness and relevance.
  • Real-time data prediction contests can provide an immediately usable product whose creation time depends on the amount of time it takes to accumulate enough data to differentiate contestants. Once created, the solutions generally do not lose statistical relevance. Contestants understand that in order to remain near the top of the leaderboard, and thus continue to receive compensation, they need to maintain their model and data sets so that they are timely and relevant.
  • the common task framework (CTF) in an historical data contest is primarily a research tool that has a long development time before it can be used commercially. With a historical data contest, there is a great divide between research and production, and due to timeliness of supplied regressors and rapidly changing business environments, a long running historical data contest may never directly impact the sponsor’s business.
  • exemplary embodiments of the invention can provide is stability. Out of the intense and continuous competition comes stability, achieved through redundancy of the contestants and their solutions. If one successful participant ceases to provide answers, due to some unforeseen error in model or input data for example, it will make only a small difference to the ultimate consumer of the data because there are many other participants whose answers already contribute to the consensus. These aspects of a real time data prediction contest are not part of a typical software development life cycle and model review controls.
  • exemplary embodiments of the invention can facilitate the process for defining a real time data prediction contest.
  • a potential sponsor of such a data contest can identify a source of updating data, such as might be available on a table in a web page or a public JSON page. Identification of the source of updating data may be much less involved than the process of setting up a traditional historical data contest, which typically involves both the collection of relevant curated data and contest design, including the avoidance of data leakage.
  • Real time data prediction contests may also provide the advantage of decreased costs. With increased accessibility of data predictions, more prediction and formalized decision making will be used. Increased access to crowd based prediction may result in a large amount of granular, individually tailored applications. As the availability of real-time random variables of high quality rises, the marginal cost of creating further forecasts on which these depend will drop significantly. Some low cost contests may be dominated by fully automated entries.
  • Real time data prediction contests also provide the advantage of enabling conditional prediction contests, including, for example, action-conditional forecasting for industrial control. Crowd-sourced conditional prediction may play a significant role in areas where control theory and reinforcement learning are now applied.
  • the depth of the real-time web is another aspect of exemplary embodiments of the invention.
  • “depth” may refer not only to the use of deep networks in competitors’ entries, but to the creation of computation graphs (e.g., networks) interweaving and stacking the contributions of different participants en route to an accurate real-time answer.
  • sub-contests may be used to enhance the effectiveness of a real time data prediction contest.
  • Arrival times t are generally stochastic but not so the service response time ⁇ 5).
  • Participant a can assign prize-money for sub-contest k equal to some fraction of their own ongoing compensation in the top level contest j. This permits the participant to fragment the task in different ways according to their objectives and constraints. Participant b entering sub-contest k may have every reason to accept this arrangement even in the degenerate case where the sub-contest asks precisely the same question and participant a merely forwards participant b’s answer verbatim. This is because contestant a might attach some added data such as lagged values, exogenous data, cleaned data or data comprising useful features to the question in the sub-contest, thereby saving b from time consuming work.
  • a may offer some other benefit such as free computation, curated library distribution, simple REST deployment or a desirable data science environment with back- testing capabilities. Failing that, participant a might merely provide superior marketing of the sub-contest than the parent or search engine optimization, or perhaps arrange superior access to an expert community.
  • a“feature” generally refers to a submission to a contest that may be useful as a regressor but may not in and of itself constitute an unbiased estimate of the target.
  • Features can be provided to participants in a number of ways. According to one embodiment, participants can buy features a la carte via a market mechanism adjacent to the contest.
  • a meta-contest parametrized by a choice of link function (such as affine, rectified linear or logistic) can be implemented.
  • the task of meta-contest participants is choosing how to use the link function to combine the most promising competition entries, and useful features, into an even more accurate response.
  • a participant in this meta contest supplies a weight vector w, offset b and any other parameters as required by the link function.
  • Their decision is based on the historical performance of contestants as they do not see the present values of entries until after the submission time t of the child contest.
  • the calculation is owned by the parent. This arrangement defends against the piracy concern: the participant entering the parent contest could easily enter the child contest.
  • the weight contest may provide immediate transparency into the relative worth of features and entries submitted to the child contest - thereby boosting the efficiency of the feature market and, to use economic terms, helping to allocate resources to where they are, on the margin, most needed. Additionally, some entries in the child contest may themselves use sub-contests in which, once again, a blind weight meta-contest mechanic is employed. In this way combinations of contests may grow downward and begin to resemble neural networks. The perspective can be reversed. For example, if the participant begins with a well-trained deep network, establishing a weight contest could improve the final layer.
  • a contestant might arrange a subcontest to predict their own residuals. While superficially this is equivalent to supplying their entry as a feature, deep residual learning experience suggests that learning to hit zero can be easier than an arbitrary target.
  • the tree can extend downward.
  • a contestant entering a residual contest can create their own residual subcontest and provide hints as to why they think their model can be improved. For example“I think my model is pretty good but my errors are high on long weekends.” As participants sub-divide prediction or reconstitute it, they are slotting themselves into a“production line” version of prediction.
  • a“derivative” contest may use an underlying contest or exogenous market as target (i.e. solution).
  • an action-conditional prediction contest is combined with an underlying contest or exogenous market as target (i.e. solution).
  • Table 1 Sample response to an action-conditional prediction contest.
  • the target is the post-play consensus game winning probability as defined by a second, underlying contest.
  • conditional contest may require all participants to provide forecasts for every eventuality, though in the example above there is a shortened pitch classification list for brevity.
  • Contestants can only be assessed on one of their predictions because only one actual pitch is thrown. However it may be reasonable to assume that the predictive capability of participants carries over from one action choice to another.
  • Exemplary embodiments of the invention enable separation of concerns, in which participants with orthogonal expertise combine their skills. For example, one participant might have a firm grasp of game theory and possess great insight into the mixed strategy for pitch selection that trades off predictability versus exploitation. The participant may have some insight into the batter’s psychological state or have an excellent model for game state transition conditional on successful contact with certain types of pitches. However, lack of an historical database might leave this same participant at a great disadvantage in predicting game results from post-play game states or end of innings scores. This baseline might best be provided by someone else participating in either the prediction market or the underlying end of play contest. The derivative contest does not require participants to estimate the probability of a particular action taken by the pitcher. This is analogous to the separation achieved by Q-leaming.
  • temporal difference (TD) contests can be used.
  • the consensus crowd prediction in the underlying contest is denoted by V (t k ), where t k indexes pitches.
  • the prediction contest may be set up as follows:
  • V(t n ) is defined as the current state of the underlying competition (aggregate forecast) after time state t n if the game has not finished, or equal to the game result if t n extends to the finish of the game.
  • the system employs a meta contest in which the crowd is used to predict the best predictions of the crowd. That is, the crowd itself can be used to opine on the efficacy and future efficacy of participants, and also the manner in which the consensus is derived.
  • a meta-competition may be analogous to a meta- parameter in a hierarchical model. It may straddle a plurality of contests to avoid circularity and manipulation. The particularities of the contest and participant can be omitted to encourage a crowd-search for generally applicable consensus calculations.
  • Point estimate contests may be utilized.
  • participant i supplies only a single number X;(t), perhaps contributing to a consensus estimate x(t)for the customer.
  • the running mean square error is
  • [00117] is the running mean square error for participant i's responses in contests who truth has been revealed thus far.
  • errors e,(/) are serially correlated.
  • One simple example is to down-weight contribution from participants with the highest serial correlation.
  • a simple, robust aggregation uses the median forecast from a high percentile sub-group of participants.
  • Likelihood Contests According to another embodiment, a likelihood contest is utilized. In some contests participants respond with probability vectors for discrete outcomes. A means of scoring participants' entries is the posterior log-likelihood.
  • Implied Trading According to another embodiment, a market inspired mathematical fiction may be used to determine compensation and aggregation. According to this embodiment, the real time data prediction system will:
  • Real-time data contests can be viewed as an attempt to maintain the immediacy and incentives of a market mechanism without commingling prediction modeling with human decision making.
  • Real-time contest design still generally needs a means of deriving a representative answer from the crowd responses and also a means of compensating participants who play a material role in its construction. These objectives can be met by building robotic investment decisions directly into the scoring mechanism. The setup then resembles a“robo- market” where participants have relinquished explicit investment decisions to an investment robot, though not the task of creating the estimates on which the automated investment decisions depend.
  • exemplary embodiments of the invention can utilize a REST- based protocol for competitive yet collaborative forecasting, and various design patterns can be used to further facilitate a division of labor in data gathering, feature generation, model selection, meta-modeling and ongoing performance analysis.
  • exemplary embodiment can provide an online, redundant, constantly updating and collectively computed computation graph culminating in the production of a consensus forecast for a large number of variables of interest.
  • Exemplary embodiments of the invention can be applied to a wide variety of industries which may need pricing, recognition, logistics, recommendation or bespoke forecasting in some form or another. From insurance and broker dealers to agriculture to manufacturers of handbags, most any industry category, or sub-category, or sub-sub-category may benefit from superior prediction and decision making.
  • Examples of applications of the invention to predict events are numerous and could include various commercial predictions such as: when a product will fail; what items a person will buy, whether a person will like a purchased item, what is the future price of certain components of a product, what is the optimal price for any item in commerce, what are the expected ratings of a TV show based on demographics of the audience, and so on. While initial implementations may be driven by commercial motivation, many educational, health, civic and scientific applications will benefit from improvements to the system, resulting from commercial applications. Engagement and system feedback ari sing from commercial appli cations will generate commonality in feature spaces across many real-time applications.
  • predictions can be applied, including in risk, compliance, trading, operations, security, real estate, credit card services, banking loans, student loans, auto loans, investment management, private wealth management, merchant services, middle marked banking, commercial term lending, human resources and technology, etc.
  • Healthcare applications may include genomics, microfluidics, imagery, inventory, equipment monitoring, patient monitoring, behavioral campaigns and many other applications.
  • Other applications that can benefit from a prediction web may involve wheat fields, oil rigs, combat, combinatorial chemistry, production line optimization, robotics, procurement, voice logs, expedited delivery, spoilage, loyalty, component costs, recovery logistics and countless other commercial activities.
  • any number presented on an app, an enterprise data feed, or a trader’s desktop may be automatically replaced by a forward looking term structure for the same.
  • Enhancing weak, universal data feeds According to exemplary embodiments of the invention, it is possible to create a large number of highly relevant nodes in short order. This does not necessarily require an equivalently large number of data sources, due to the notion of a weak universal source and the ability of the prediction web, as described herein, to strengthen them. The key observation is that good data can be a highly competitive predictor of bad data.
  • Real-time contests can turn weak data into strong, dirty data into clean, and intermittent data into continuous in much the same way that continuously traded stock prices effectively convert a mix of lagged accounting data into a forward looking estimate.
  • any sufficiently broad source of text data can be a universal data source insofar as n-gram prediction (or more sophisticated variants) can be used to promote the location of higher quality data sources.
  • n-gram prediction or more sophisticated variants
  • this weak data source is nonetheless correlated with reality, and may be sufficient incentive for participants to locate, interpret or even create stronger data sources from inexpensive WiFi enabled sensors. The value to the sponsor of such a contest, or the community at large if it is public, is likely to be far greater than the prediction of the weak data source per se.
  • Establishing a contest to predict n-grams in weak data sources canbe established automatically by a hot looking for good data sources with a view to winning money in other contests, as will be described further below.
  • Good mechanisms for encouraging a division of labor can further facilitate the creation of a nutrient rich environment for a species of specialist text mining hots whose purpose is ensuring that the incremental lift to prediction provided by alternative data sources (which today is a mostly labor intensive, bespoke activity) is realized whenever applicable.
  • location data prediction can provide valuable information, even if it pertains to a relatively small percentage of the population. Often this comes attached to free text data generated by users on their phones, and can signal a variety of highly specific intents. Are you running because it is raining? Are you inside a nightclub or waiting outside? Yet it may be very difficult for any one company to fully monetize this data, since in the absence of a statistical contest the incremental economic value for any given prediction is largely unknowable in advance. It is likely, therefore, that this will instead be accomplished by hots, as discussed below, perhaps using cryptographic methods to avoid invasion of privacy.
  • One method for generating a large number of weak data feeds is the joining of location data against databases of interest whose attributes include GIS coordinates. Every column of any such database is a potential contest. For example, a database containing locations of libraries is a real-time contest for library usage, and so it goes.
  • An evaluated price for a bond provided by a vendor may not be a good, or even a mediocre, statistical estimate of its future value. Evaluated prices are not martingales, nor trivially converted to martingales (by coupon adjustments, pull to par, convexity, carry or other easily accommodated deterministic adjustments).
  • real-time nano-markets (which might be the preferred terminology when many tributaries feed eventually into an easily monetizable market prediction) can provide a new, resilient variety of transparency precisely because market forces replace model assumptions in parameters, meta-parameters and so forth which might otherwise be subject only to the much weaker checks and balances that may be utilized traditionally.
  • an accuracy metric can be selected by the customer.
  • the R package mlmetrics and similarly named python package ml metrics provide an enumeration of scoring rules. Examples for point estimates include mean absolute deviation and root mean square deviation. If quantities vary over several orders of magnitude Root Mean Squared Logarithmic Error may be more useful. For probabilistic forecasts, log-likelihood is one example of a method that can be used for scoring.
  • Ensemble learning and inverse probability weighting may be used in connection with data streams.
  • the crowd will, or can, perform multi-sensor data fusion.
  • the relative time scales on which predictions and meta-predictions are made and evaluated will typically be an important part of overall design.
  • a key challenge is how best to compensate participants or provide a consensus forecast if some are recently arrived or fail to answer some questions.
  • One solution would penalize missing answers going all the way back to the time at which the real-time contest was established; however, this approach may be perceived to be penal.
  • the contest could be periodically reset and scoring could consider only those participants present for the entirety of an epoch.
  • Another example would use imputation of missing answers, which would be less punitive and in some cases straightforward to implement.
  • participants are allowed to choose imputation formulas based on contemporaneous forecasts by their peers, such as defaulting to an average of other contributors forecasts.
  • a different class of solutions may be modeled after market clearing mechanisms.
  • forecasts provided by participants are characterized as nano- investments in contingent claims that permit a market clearing mechanism to be defined and payouts made when the future out-come is resolved. This can simultaneously perform compensation and aggregation.
  • Parimutual aggregation is another compensation method, as was described above.
  • the prediction web described herein comprises a mesh of random variables spanning the future, each one the outcome of never-ending competition. Rather than standing in isolation, as with historical contests, the nodes can be stacked and combined in many ways. Evolving relationships are established between them.
  • AutoML Automated machine learning
  • the R package Caret was designed to streamline the process for creating predictive models by standardizing data splitting, pre-processing, feature selection, model tuning using resampling, and variable importance estimation. Caret brings together over a hundred and fifty disparate machine learning and statistical approaches under a common interface. The full list is provided by the authors at GitHub.
  • TPot a Python library that automatically creates and optimizes skleam machine learning pipelines. Some approaches are suitable to time series, others to images.
  • sentinels may refer to a new form of artificial life that combines many or all of the capabilities of spiders, robos, trolls and brokers mentioned below. Sentinels are bots that automatically enter real-time contests, perform meta-model search, and spider for relevant regressor data to improve their own performance. New contests will be rapidly covered by a multitude of these entities as soon as they are posted. Once scale has been achieved, sentinels may become endemic to large webs of real-time forecasting contests.
  • the sentinel may constantly survey existing data sources and related contests. They will looking for mutual information, test for Granger causality and search features spaces and variable transformations for whatever might be useful.
  • An example of a sentinel is an algorithm that maintains a list of contest variables that have proven useful regressors in the past, then applies some version of regularized regression to a new contest, such as Lasso. Using strong rules, Sentinels may quickly pre-process new contest and data sources as they arrive without undue computational burden.
  • Emus Emus search contests for participant entries that fail to perform well but accidentally have use elsewhere.
  • Trolls may compri se any program capable of entering contests programmatically, though not necessarily capable of performing notably well over time. Trolls may be mostly an annoyance and a small tax, though they can alternatively be seen as serving a purpose by encouraging good contest design.
  • Miners may comprise specialized programs that only enter a sub-class of“performance” contests. Those are aimed at enhancement and outsourcing of well understood but computationally burdensome algorithms. Miners will typically use graphics card processing, specialized hardware or, sometimes, novel approximation algorithms. A specific variety of mining, one in which fast approximations for batch algorithms are lured, is considered below.
  • Readers are sentinels that specialize in converting the text of a contest description, and associated search results, into a series of n-gram prediction sub-contests, as discussed above.
  • Stalkers are similar to readers, but comb through personal location data rather than free text. Foot traffic is a predictor of virtually all commercial activity, but also social activity. As will be appreciated by those skilled in the art, many things can be inferred from collective movement. Stalkers need not be as sinister as the name implies. Cryptographic methods can be applied to preserve privacy and offer users of location tracking apps a better compromise between invasive use of data and the benefits they receive by giving it up.
  • Brokers typically cyborg, match contests with algorithms for commission on earnings. They can play an important role in private data applications as discussed below. Dealers help resell algorithms for external use, and can be used to implement a“push” model similar to analytics storefronts. Dealers may be human or cyborg. Algorithmic dealers might take a plain text description of a problem and suggest similar problems, relevant algorithms, historical performance and so forth
  • Head-hunters Closely associated with the dealing of algorithms is the recommendation of their human creators for full time employment, contracting work or task-specific private contests.
  • a hedge fund may arrive at a key statistic it believes is critical to predicting a sector, but asking the crowd to predict this may betray a known causality it previously considered a trade secret.
  • Figure 12 illustrates examples of methods for defense against misuse of data supplied to participants (or not supplied as the case may be).
  • Contamination of the data with a small number of erroneous data points may also deter unpaid commercial usage. These data points would not count in the assessment of participants, but they would come at a small price since calibrated models would have to account for this additional noise.
  • steganography can be used to make statistically insignificant yet traceable changes to the data, enabling a subsequent investigation to identify the precise participant who was the source of the stolen goods. Together with reasonable steps to know your data scientist (such as requiring login by Linked-In account with minimal vintage and details) this can provide a reasonable deterrent. Since the investigation or blocking of participants is a time series, a meta-contest can be run to assist in identifying and eliminating nefarious activity. All of the techniques mentioned in the following sections are stronger, and can also be used to address this concern.
  • transformation and obfuscation may be employed, though this may come at a price. It is unlikely to adversely impact“pure” model search entries (which will make their own
  • Bootstrapping can be used to drown out the actual data in a sea of artificial points. However, this requires the identification of invariants in the data or some preliminary estimation of a generative model. It is also possible to use methods applied where the data is considered“secret,” as described below, which are likely to be the most promising approaches.
  • bootstrapped public contests can be used as bait for robos and sentinels. They are designed to draw in a class of algorithms satisfying the signature of the class at hand, but also capable of adapting to new unseen training sets and time series without further human intervention. Sentinels entering this specific type of lure contest must agree to cloning and reuse of their algorithm on data they will never see.
  • such programs are developed in one or more standard languages and environments (standard library packages, for example, approved for the classified use and replicated in- house). In this way, sentinels can participate in private data contests without any information passing back to the authors of the s entinel. They have knowledge of the commercial arrangement but do not watch or participate in the private contest.
  • chumming can be viewed as analogous to human recruiting, and real time prediction contests may be viewed as recruiting exercises.
  • Real-time prediction contests may involve (a) recruiting human talent for use on private data; (b) recruiting relevant algorithms; (c) recruiting robos and sentinels which adapt and learn, and can be used on private data; (d) recruiting exogenous data (that predicts); and (e) recruiting uses (data that is predicted). Discussed below are a special case of chumming (inference) and details of an iterative scheme that successively improves the relevance of simulated data used to attract algorithms (boosting and synthetic data).
  • Robos are search algorithms intended to automate not only the estimation of parameters within a family but also comparisons across very different approaches. It may be desirable to leverage all this work in an online, time series setting. However, where time series analysis is identified with supervised learning in order to take advantage of tools like Caret, the solution found is likely to be inefficient. This is because the solution will typi cal l y need to be recomputed from scratch using the entire window of data each time, even though the window may only be updated incrementally, for instance with the addition of a single new data point and the removal of the least recent. In very few cases are online versions of the algorithm provided in the underlying libraries either for estimation or prediction.
  • participant A receives a question in contest 1;
  • participant A determines the computation required and posts as a question to contest 2 which they have established with a slightly tighter response time than contest 1;
  • participants B, C, D,. . . , Z respond to the contest 2 question; their results are aggregated, for example as the median of the top five best performers; and
  • participant A receives the aggregate result, and converts it back into an answer to contest 1.
  • participant A would rely on their own algorithm, perhaps at significant expense. They could not initially expect to rely solely on the sub-contest because the accuracy of responses would not be known. However over time they could gain confidence and eventually rely on sub-contest approximations for most of their answers. Their own, inefficient algorithm would then be relegated to a supervisory tool and used just enough to keep the sub contest participants honest.
  • participant A might decide, at their discretion, to disclose complete details of their desired calculation and even the offline code they use to evaluate the task. Quite often, for example, bottleneck computations can be separated from more proprietary logic. Complete transparency may encourage the development of clever transcription bots from one language to another. Another approach may take existing Python code and automatically add numba decorators. Or automatic differentiation libraries could be put to good effect. What is notable about this setup is that p articipant A is able to write code that doesn’t go out of date. The task of maintaining the best version of this algorithm, in the most suitable language, running on the most economic hardware, is left to the other participants. In this way human designers can be lifted to a higher, purer, mathematical level of abstraction to the extent that the task permits. But the same may be true of sentinel creation, as will now be discussed.
  • Chumming online algorithms can now be des cribed in a specific way: the prediction target will be the ground truth of a generative model. To make the di scussion more concrete, we revi sit the seeming dichotomy between inferential and predictive schools of thought with a specific example.
  • noisy measurements yt arrive at a sensor.
  • a statistician seeking to understand the system might posit a simple linear Gaussian system in which an underlying truth x following Brownian motion with time step variance Q is corrupted by white noise e / with variance R.
  • the generative model :
  • the statistician Due to the existence of the mathematical shortcut provided by the Kalman filter, the statistician has not only performed inference but also prediction. If sensor data were to be supplied that included the true values for x / and not just the noisy observations yt ⁇ the statistician would we very well placed to predict this ground truth. Indeed, even if it should turn out later that he or she was wrong about the noise distribution, he might nonetheless have “accidentally” arrived at the best linear estimator. In this sense the distinction between the two schools of thought can be dropped.
  • the statistician would probably want to verify the solution against lengthy simulations just to make sure no algebraic error had been introduced. To a degree, this process can continue.
  • the generative model may be altered as it pertains to the dynamics of the presumed underlying truth xt.
  • a third process may be introduced to accommodate regime changes, and so on.
  • the statistician will, in his efforts to find inference calculations for every reasonable generative model he finds plausible, be severely limited by his either his own dexterity, accumulated knowledge or ability to search literature in reasonable time. He may even run up against no-go theorems for conjugate families. In practice, there is a temptation to abandon the inferential approach at this point.
  • a careful workflow would typically include, at minimum: (1) implementing a simulation of the generative model (typically easy); (2) implementing an inference scheme for the same (almost always hard); (3) verifying correctness of the inference numerically for known parameters (tedious); (4) devising an estimation scheme for the parameters (almost always very hard); and (5) verifying in lengthy simulations that the estimation is consistent when the assumed generative model is used to generate synthetic data (again, tedious).
  • Contestants will be provided with measurements y t and judged against the underlying truth x t. However the ground truth will be quarantined for T » 1 lags so as to only be indirectly useful to participants in their choice of estimate x ⁇ . Thus at time t contestants will have observed y i, ... , y t -i but x ⁇ , ... , x t- t only. It is an assumption of this setup that, due to the decay in relevance of past x t values, they are forced to rely heavily on the actual noisy observations y t.
  • the statistician is inviting the crowd to develop an approximate inference and estimation algorithm that will work on his problem of interest so long as his generative model is a good representation of reality. By outsourcing approximate inference our statistician is able to work, once again, at the level of a generative model.
  • Reasons to persist with generative modeling There may be many reasons to persist with generative modeling. In the filtering example described above, the most accurate predictor of y t will not coincide with the best estimate of x t because the former will“chase” the predictable component of the error E t whereas the latter will not. Yet the concrete application in mind might demand an estimate of x t , not J7. Examples include market making where x t is a price time series.
  • Additional criteria deemed beneficial might be built into the contest.
  • An end user might be looking for a relatively smooth sequence of estimates oiy t and, once again, the best predictive estimate of y t may chase too often. Smoothness might be built directly into the competition scoring and compensation, encouraging participants to trade off the dual objectives.
  • the contestants’ entries might be mapped to an efficient frontier where the compromise between smoothness and accuracy is made apparent and a partial, rather than total ordering is applied. This would enable the customer to select a representative set of dominating algorithms, some smooth and not quite as accurate, others very accurate but not so smooth.
  • Figure 13 illustrates an indirect approach that avoids the hard work of producing an inference model from a generative model.
  • the crowd competes in a family of prediction contests in which observations ⁇ / are supplied but ground truth targets xt are quarantined for many lags or withheld entirely In traditional historical contests it may be necessary to consider the ability to interpret predictive models. In the inference contest that consideration is generally not a concern.
  • the generative model has already supplied the meaning.
  • Figure 14 illustrates a slightly different perspective on Figure 13 in which public inference contests bait good approximate inference algorithms, which are simultaneously used on private data.
  • Boosting The discussion above described the use of synthetic data contests as a means of attracting and shortlisting algorithms that can then be applied to private data. A class of examples were described using data generated from batch algorithms in need of online approximation, and more specifically the use of generative time series models for evaluating inference algorithms. A broader example will now be described that relies on the possibility of efficiently teaching recruited algorithms by comparing their performance on public data and private data their authors never see.
  • a contest designer should be cognizant of the relative amount of upfront versus ongoing estimation likely to be performed by participants’ hots. There are extremes at both ends of the spectrum.
  • One competitor may submit a constant model for a contest in which a daily estimate of temperature is required, that is to say, for avoidance of doubt, a function whose body is a single line of code“return 27.3.” The number might be valuable but the algorithm submitted clearly is not. Nonetheless the participant can push a new version of the code every day and win the contest, very much defeating the attempt by the sponsor to draw in algorithms that are reusable on private data.
  • Multi-task learning may be another driver of a prediction web where private use is important, and where low cost use drives adoption. Yet viewed this way, and predicated on sufficient incentive for the development and maintenance of highly effective multitask, few-shot and autonomous species, there may be no privacy issue. Data sensitivity issues only arise due to externalization of learning. With this caveat, we consider the chumming of non-leaming algorithms that are trained by their authors on a public, synthetic data set and simultaneously evaluated on a private commercial data set.
  • a specific algorithm is described that iterates on the synthetic data presented to participants based on performance predicting private training data they are not privy to. This concept may be referred herein to as the“boosting” of a synthetic training set. Those uncomfortable with the appropriation of the term boosting might prefer to refer to this as “active chumming”.
  • a synthetic data generation method is a requisite, and some are considered in the di scus sion below .
  • An iterative scheme for increasing the correlation between performance in public and private leaderboards may proceed by eliminating low correlation synthetic training points and regenerating more to take their place.
  • the procedure may be applied to time series contests where many synthetic time series are generated using different meta- parameters. We terminate some time series and birth others.
  • T ⁇ (0, 0), (0.1, 0.1),..., (1, 1) ⁇ ,
  • the private validation data comprises
  • V ⁇ (0.05, 0.05),..., (0.95, 0.95) ⁇
  • the coarse trained regression trees should fall towards the bottom of the leaderboard, with performance varying based on the decision boundaries. If (0.28, 0.34) are the edges straddling 0.3 after one tree model is fit in the simulated data then the tree prediction on the same point 0.3 will be the arithmetic average of the y values of only those synthetic training points whose x coordinate falls between 0.28 and 0.34. Only by accident will this be close to 0.3. More likely it will be higher. Trees whose edges straddle the training data abscissa symmetrically will tend to outperform those that are unbalanced.
  • the boosting algorithm will be particularly effective. If a collection of step functions dominate the upper echelons of the leader-board then points like (0.09, 0.1) and (0.11, 0.1) may actually record a high p j , encouraging a cycle in which the training points T achieved through boosting take on the collective appearance, at least locally to 0.1, 0.2 and so forth, of a step function. Models that are selected may not perform well on the validation set V at all. On the other hand if a healthier feedback sets in the contest will select mostly linear models, as desired.
  • probabilistic model for an underlying random variable such as a stock price
  • known constraints such as the price of an option on the same. Absent faster methods such as PDE or analytic solutions, one traditionally proceeds to draw samples from the probabilistic model for the purpose of pricing other securities. As an aside, the probabilistic model for this purpose would be under a pricing measure, and thus would not be intended to model real world probabilities.
  • a quasi-random sampling scheme can be used that automatically achieves the calibration while striving to retain smoothness of the distribution.
  • herding may refer to sampling data and coercing the empirical distribution of those samples towards desired properties.
  • a relatively recent line of research in machine learning goes by that name (see Max Welling, Herding Dynamic Weights for Partially Observed Random Field Models, Uai 2009 , pages 599-606, 2009; Max Welling, Herding dynamical weights to learn, Proceedings of the 26th Annual International Conference on Machine Learning -ICML 2009, pages 1-8, 2009).
  • herding creates quasi-random samples subject to constraints.
  • X a denotes a group whose cost relative to a desire moment is represented by f a , then once gain we can choose random samples X ⁇ . . . , X ⁇ sequentially. The next random sample can be chosen to-minimize a cost function representing weighted discrepancy across all the groups.
  • herding represents an approach to reconstruction of a joint distribution from lower dimensional projections (subject to the internal consistency of the n-margins themselves). It thereby hopes to address a generalization of the problem tackled by Copula theory, where only 1 -margins are considered and 2-margins are used in calibration.
  • the herding approach to synthetic data generation in high dimensions may be summarized: (1) estimate low dimensional margins (e.g. 3-margins) based on the real dataset, for example by using the empirical distribution; (2) use herding to simulate the joint distribution.
  • low dimensional margins e.g. 3-margins
  • Gaussian Copula made notorious during the global financial crisis. Simulation of synthetic data using a Gaussian Copula proceeds by first simulating a multivariate Gaussian vector Z and then applying a composition of a normal distributional transform to each margin (converting them to uniform) followed by the inverse distributional transform of the margin.
  • Gaussian Copula approach is philosophically similar to the herding approach:
  • Generative adversarial networks have proven extremely popular and may be implemented as adversarial contests.
  • a method for estimating generative models can be implemented that uses an adversarial process in which two models are trained simultaneously: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G.
  • the training procedure for G is to maximize the probability of D making a mistake.
  • This framework corresponds to a minimax two-player game.
  • the generative model G is pitted against an adversary: a discriminative model that learns to determine whether a sample is from the model distribution or the data distribution.
  • the generative model can be considered as trying to produce fake data and use it without detection.
  • the discriminative model can be thought of as trying to detect the fake data. Competition in this game drives both teams to improve their methods until the fake data is indistinguishable from the real data.
  • the adversarial modeling framework is preferably applied with both models comprising multilayer perceptrons.
  • p g a prior on input noise variables p z (z)
  • Gfe 0 a mapping to data space
  • D(x; 0d) a mapping to data space
  • D(v) represents the probability that* came from the data rather than p g.
  • D(v) represents the probability that* came from the data rather than p g.
  • We train D to maximize the probability of assigning the correct label to both training examples and samples from G.
  • G minimize log(l - D(G(z))). Additional details can be found at Ian J. Goodfellow et al., Generative Adversarial Nets, arXiv: l406.266lvl [stat.ML] (June 10, 2014).
  • participant in the family of bond contests may be greeted as follows: Welcome to the bond prediction contest. You will be asked to create an algorithm that responds, within one second, to a highly specific question about the future trading activity in a particular corporate bond.
  • the question payload will include historical trading in the bond in question, historical trading in some related bonds issued by the same company, and miscellaneous attribute data such as the time to maturity of the bond, the coupon, whether the bond is callable.
  • Those attributes are detailed in a data guide and their use illustrated in some template algorithms you are welcome to clone.
  • contest pages may also provide specialized instruction describing the parameters of the question they should expect. For example:
  • Figure 16 lists some other low sensitivity contests that can be run using the essentially public FINRA TRACE record as a means of defining question payloads and solutions.
  • Figure 16 shows an example of a listing of contests, relative commercial sensitivity of the data streams involved, and techniques employed.
  • Question payload An example of a listing of trades of a corporate bond is provided in Figure 17 (FINRA TRACE reporting of corporate bond trades).
  • the question payload can be built on this history by including generalized lags of the full TRACE history for all bonds from the same issuer.
  • the details of the lag generation are quite complex and can be provided to the participants inline.
  • the code for efficient generation of these lags can also be provided to participants so that they might better understand the construction.
  • exemplary embodiments of the invention can provide many advantages to data consumers. For example, real-time contests are reusable, as one contest can be used as a regressor for another. The immediate utility of a large array of interrelated, bi- temporally indexed random variables is clear.
  • timeliness Another advantage is timeliness.
  • Real-time contests combat the problem of timeliness, also identified as an important limitation of historical contests.
  • Today companies place emphasis on near real-time business intelligence and the data acquisition required for the same.
  • the creation time for a new collection of nodes on our lattice dictated by the amount of time it takes to accumulate enough data for participants to build reliable models, and then the time it takes to differentiate contestants. This compares favorably with typical time to market for quantitative models.
  • a contest can maintain its statistical relevance. Contestants are on the hook for that and must remain near the top of the leader-board to continue to be paid.
  • Ease of specification is another advantage. Any person can establish a contest.
  • the various components may be located at distant portions of a distributed network, such as a local area network, a wide area network, a telecommunications network, an intranet and/or the Internet.
  • a distributed network such as a local area network, a wide area network, a telecommunications network, an intranet and/or the Internet.
  • the components of the various embodiments may be combined into one or more devices, collocated on a particular node of a distributed network, or distributed at various locations in a network, for example.
  • the components of the various embodiments may be arranged at any location or locations within a distributed network without affecting the operation of the respective system.
  • Data and information maintained by the servers and personal computers shown by Figure 1 may be stored and cataloged in one or more databases, which may comprise or interface with a searchable database and/or a cloud database.
  • the databases may comprise, include or interface to a relational database.
  • Other databases such as a query format database, a Standard Query Language (SQL) format database, a storage area network (SAN), or another similar data storage device, query format, platform or resource may be used.
  • the databases may comprise a single database or a collection of databases.
  • the databases may comprise a file management system, program or application for storing and maintaining data and information used or generated by the various features and functions of the systems and methods described herein.
  • Communications network may be comprised of, or may interface to any one or more of, for example, the Internet, an intranet, a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a storage area network (SAN), a frame relay connection, an Advanced Intelligent Network (AIN) connection, a synchronous optical network (SONET) connection, a digital Tl, T3, El or E3 line, a Digital Data Service (DDS) connection, a Digital Subscriber Line (DSL) connection, an Ethernet connection, an Integrated Services Digital Network (ISDN) line, a dial-up port such as a V.90, a V.34 or a V.34bis analog modem connection, a cable modem, an Asynchronous Transfer Mode (ATM) connection, a Fiber Distributed Data Interface (FDDI) connection, a Copper Distributed Data Interface (CDDI) connection, or an optical/DWDM network.
  • ATM Asynchronous Transfer Mode
  • FDDI Fiber Distributed Data Interface
  • CDDI Copper Distributed Data Interface
  • Communications network 110 in Figure 1 may also comprise, include or interface to any one or more of a Wireless Application Protocol (WAP) link, a Wi-Fi link, a microwave link, a General Packet Radio Service (GPRS) link, a Global System for Mobile Communication (GSM) link, a Code Division Multiple Access (CDMA) link or a Time Division Multiple Access (TDMA) link such as a cellular phone channel, a Global Positioning System (GPS) link, a cellular digital packet data (CDPD) link, a Research in Motion, Limited (RIM) duplex paging type device, a Bluetooth radio link, or an IEEE 802.1 l-based radio frequency link.
  • WAP Wireless Application Protocol
  • GPRS General Packet Radio Service
  • GSM Global System for Mobile Communication
  • CDMA Code Division Multiple Access
  • TDMA Time Division Multiple Access
  • GPS Global Positioning System
  • CDPD cellular digital packet data
  • RIM Research in Motion, Limited
  • Communications network 110 may further comprise, include or interface to any one or more of an RS-232 serial connection, an IEEE-1394 (Firewire) connection, a Fibre Channel connection, an infrared (IrDA) port, a Small Computer Systems Interface (SCSI) connection, a Universal Serial Bus (USB) connection or another wired or wireless, digital or analog interface or connection.
  • an RS-232 serial connection an IEEE-1394 (Firewire) connection, a Fibre Channel connection, an infrared (IrDA) port, a Small Computer Systems Interface (SCSI) connection, a Universal Serial Bus (USB) connection or another wired or wireless, digital or analog interface or connection.
  • IEEE-1394 Firewire
  • Fibre Channel Fibre Channel connection
  • IrDA infrared
  • SCSI Small Computer Systems Interface
  • USB Universal Serial Bus
  • the communication network 110 may comprise a satellite communications network, such as a direct broadcast communication system (DBS) having the requisite number of dishes, satellites and transmitter/receiver boxes, for example.
  • the communications network may also comprise a telephone communications network, such as the Public Switched Telephone Network (PSTN).
  • PSTN Public Switched Telephone Network
  • communication network 110 may comprise a Personal Branch Exchange (PBX), which may further connect to the PSTN.
  • PBX Personal Branch Exchange
  • servers 120, 142, 146, 162, 166 and personal computing devices 122, 124, 140, 144, 160, 164 are shown in Figure 1, exemplary embodiments of the invention may utilize other types of communication devices whereby a user may interact with a network that transmits and delivers data and information used by the various systems and methods described herein.
  • the personal computing devices 122, 124, 140, 144, 160, 164 may include desktop computers, laptop computers, tablet computers, smart phones, and other mobile computing devices, for example.
  • the servers and personal computing devices may include a microprocessor, a microcontroller or other device operating under programmed control. These devices may further include an electronic memory such as a random access memory (RAM), electronically programmable read only memory (EPROM), other computer chip-based memory, a hard drive, or other magnetic, electrical, optical or other media, and other associated
  • the mobile device and personal computing device may be equipped with an integral or connectable liquid crystal display (LCD), electroluminescent display, a light emitting diode (LED), organic light emitting diode (OLED) or another display screen, panel or device for viewing and manipulating files, data and other resources, for instance using a graphical user interface (GET) or a command line interface (CLI).
  • the mobile device and personal computing device may also include a network-enabled appliance or another TCP/IP client or other device.
  • the personal computing devices 122, 124, 140, 144, 160, 164 may include various connections such as a cell phone connection, WiFi connection, Bluetooth connection, satellite network connection, and/or near field communication (NFC) connection, for example.
  • Figure 1 includes a number of servers and personal computing devices, each of which may include at least one programmed processor and at least one memory or storage device.
  • the memory may store a set of instructions.
  • the instructions may be either permanently or temporarily stored in the memory or memories of the processor.
  • the set of instructions may include various instructions that perform a particular task or tasks, such as those tasks described above. Such a set of instructions for performing a particular task may be characterized as a program, software program, software application, app, or software.
  • the modules described above may comprise software, firmware, hardware, or a combination of the foregoing.
  • each of the processors and/or the memories be physically located in the same geographical place. That is, each of the processors and the memories used in exemplary embodiments of the invention may be located in geographically distinct locations and connected so as to communicate in any suitable manner. Additionally, it is appreciated that each of the processor and/or the memory may be composed of different physical pieces of equipment. Accordingly, it is not necessary that the processor be one single piece of equipment in one location and that the memory be another single piece of equipment in another location. That is, it is contemplated that the processor may be two or more pieces of equipment in two or more different physical locations. The two distinct pieces of equipment may be connected in any suitable manner.
  • the memory may include two or more portions of memory in two or more physical locations.
  • a set of instructions is used in the processing of various embodiments of the invention.
  • the servers and personal computing devices in Figure 1 may include software or computer programs stored in the memory (e.g., non-transitory computer readable medium containing program code instructions executed by the processor) for executing the methods described herein.
  • the set of instructions may be in the form of a program or software or app.
  • the software may be in the form of system software or application software, for example.
  • the software might also be in the form of a collection of separate programs, a program module within a larger program, or a portion of a program module, for example.
  • the software used might also include modular programming in the form of object oriented programming. The software tells the processor what to do with the data being processed.
  • the instructions or set of instructions used in the implementation and operation of the invention may be in a suitable form such that the processor may read the instructions.
  • the instructions that form a program may be in the form of a suitable programming language, which is converted to machine language or object code to allow the processor or processors to read the instructions. That is, written lines of programming code or source code, in a particular programming language, are converted to machine language using a compiler, assembler or interpreter.
  • the machine language is binary coded machine instructions that are specific to a particular type of processor, i.e., to a particular type of computer, for example. Any suitable programming language may be used in accordance with the various embodiments of the invention.
  • the programming language used may include assembly language, Ada, APL, Basic, C, C++, COBOL, dBase, Forth, Fortran, Java, Modula-2, Pascal, Prolog, REXX, Visual Basic, and/or JavaScript.
  • assembly language Ada
  • APL APL
  • Basic Basic
  • C C
  • C++ COBOL
  • dBase dBase
  • Forth Forth
  • Fortran Java
  • Java Modula-2
  • Pascal Pascal
  • Prolog Prolog
  • REXX REXX
  • Visual Basic Visual Basic
  • JavaScript JavaScript
  • the instructions and/or data used in the practice of various embodiments of the invention may utilize any compression or encryption technique or algorithm, as may be desired.
  • An encryption module might be used to encrypt data.
  • files or other data may be decrypted using a suitable decryption module, for example.
  • the software, hardware and services described herein may be provided utilizing one or more cloud service models, such as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS), and/or using one or more deployment models such as public cloud, private cloud, hybrid cloud, and/or community cloud models.
  • SaaS Software-as-a-Service
  • PaaS Platform-as-a-Service
  • IaaS Infrastructure-as-a-Service
  • deployment models such as public cloud, private cloud, hybrid cloud, and/or community cloud models.
  • a variety of“user interfaces” may be utilized to allow a user to interface with the personal computing devices 122, 124, 140, 144, 160, 164.
  • a user interface may include any hardware, software, or combination of hardware and software used by the processor that allows a user to interact with the processor of the communication device.
  • a user interface may be in the form of a dialogue screen provided by an app, for example.
  • a user interface may also include any of touch screen, keyboard, voice reader, voice recognizer, dialogue screen, menu box, list, checkbox, toggle switch, a pushbutton, a virtual environment (e.g ., Virtual Machine
  • the user interface may be any system that provides communication between a user and a processor.
  • the information provided by the user to the processor through the user interface may be in the form of a command, a selection of data, or some other input, for example.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • Development Economics (AREA)
  • Physics & Mathematics (AREA)
  • Strategic Management (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Data Mining & Analysis (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Economics (AREA)
  • Software Systems (AREA)
  • Game Theory and Decision Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Technology Law (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Computational Mathematics (AREA)
  • Algebra (AREA)
  • Mathematical Analysis (AREA)
  • Human Resources & Organizations (AREA)
  • Operations Research (AREA)
  • Signal Processing (AREA)
  • Pure & Applied Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Mathematical Optimization (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

La présente invention concerne un système et un procédé mis en œuvre par ordinateur permettant de fournir des sciences de données en tant que service (DSaaS) à l'aide d'un concours de prédiction de données en temps réel. Des participants dans le concours de prédiction de données en temps réel sont autorisés à exécuter et soumettre des algorithmes, à utiliser des sources de données de tiers, et à utiliser des sous-concours afin de générer des prédictions de données destinées au concours de prédiction de données. Les participants dans le concours de prédiction de données peuvent être des êtres humains ou des robots logiciels. Une catégorie d'informations confidentielles de parrainage relatives à la prédiction de données est définie et maintenue comme étant confidentielle par le parraineur, tandis que divers procédés sont mis en œuvre afin d'obtenir des algorithmes et des données pertinents destinés à prédiction de données. Le sponsor reçoit des prédictions de données provenant des participants sur une base en temps réel ou presque en temps réel, calcule un score destiné aux prédictions de données, et compense les participants en fonction de leur score.
PCT/US2019/057870 2018-10-26 2019-10-24 Système et procédé de fourniture de sciences de données en tant que service WO2020086851A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16/172,212 US11562382B2 (en) 2016-11-11 2018-10-26 System and method for providing data science as a service
US16/172,212 2018-10-26

Publications (1)

Publication Number Publication Date
WO2020086851A1 true WO2020086851A1 (fr) 2020-04-30

Family

ID=70332080

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/057870 WO2020086851A1 (fr) 2018-10-26 2019-10-24 Système et procédé de fourniture de sciences de données en tant que service

Country Status (1)

Country Link
WO (1) WO2020086851A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220020042A1 (en) * 2020-07-17 2022-01-20 Sap Se Machine learned models for items with time-shifts

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110066544A1 (en) * 2005-08-16 2011-03-17 Hughes John M Systems and methods for providing investment opportunities
US20180137525A1 (en) * 2016-11-11 2018-05-17 Jpmorgan Chase Bank, N.A. System and Method for Providing Data Science as a Service

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110066544A1 (en) * 2005-08-16 2011-03-17 Hughes John M Systems and methods for providing investment opportunities
US20180137525A1 (en) * 2016-11-11 2018-05-17 Jpmorgan Chase Bank, N.A. System and Method for Providing Data Science as a Service

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
GOODFELLOW, I ET AL., GENERATIVE ADVERSARIAL NETS, 10 June 2014 (2014-06-10), pages 1, XP055572979, Retrieved from the Internet <URL:https://arxiv.org/pdf/1406.2661.pdf> [retrieved on 20191220] *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220020042A1 (en) * 2020-07-17 2022-01-20 Sap Se Machine learned models for items with time-shifts
US11562388B2 (en) * 2020-07-17 2023-01-24 Sap Se Machine learned models for items with time-shifts

Similar Documents

Publication Publication Date Title
US11562382B2 (en) System and method for providing data science as a service
Li et al. Crowd intelligence in AI 2.0 era
Athey The impact of machine learning on economics
Jansen Machine Learning for Algorithmic Trading: Predictive models to extract signals from market and alternative data for systematic trading strategies with Python
Cameron et al. Vietnam’s future digital economy–Towards 2030 and 2045
US11367088B2 (en) System and method for providing data science as a service
Wellman et al. Autonomous bidding agents: Strategies and lessons from the trading agent competition
US20180330281A1 (en) Method and system for developing predictions from disparate data sources using intelligent processing
MacKie-Mason et al. Automated markets and trading agents
Jansen Hands-On Machine Learning for Algorithmic Trading: Design and implement investment strategies based on smart algorithms that learn from data using Python
US8417614B1 (en) Investor personality tool
US20210209554A1 (en) Project governance
Stubbs Big data, big innovation: enabling competitive differentiation through business analytics
US11620706B2 (en) Trading platforms using market sentiment and dynamic risk assessment profiles
US8706761B1 (en) Information discovery based on investments in an invention network
Gejke A new season in the risk landscape: Connecting the advancement in technology with changes in customer behaviour to enhance the way risk is measured and managed
US12039604B2 (en) Dynamically-generated electronic database for portfolio selection
Calderon et al. The impact of digital infrastructure on African development
WO2020086851A1 (fr) Système et procédé de fourniture de sciences de données en tant que service
Zandi et al. A fuzzy group Electre method for electronic supply chain management framework selection
Kingsly Disruptive Technology: Blockchain: The Crystal Ball: Advancing Financial Trust, Inclusion, and Simplicity Through the Blockchain
Bhaduri et al. Advanced Business Analytics
Irfan et al. Causal Strategic Inference in a Game—Theoretic Model of Multiplayer Networked Microfinance Markets
Shrier et al. Prediction markets
Damlapinar Analytics of Life: Making Sense of Artificial Intelligence, Machine Learning and Data Analytics

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19875302

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19875302

Country of ref document: EP

Kind code of ref document: A1