WO2021037066A1 - 用于批量和实时特征计算的系统和方法 - Google Patents

用于批量和实时特征计算的系统和方法 Download PDF

Info

Publication number
WO2021037066A1
WO2021037066A1 PCT/CN2020/111412 CN2020111412W WO2021037066A1 WO 2021037066 A1 WO2021037066 A1 WO 2021037066A1 CN 2020111412 W CN2020111412 W CN 2020111412W WO 2021037066 A1 WO2021037066 A1 WO 2021037066A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature calculation
real
time
batch
data
Prior art date
Application number
PCT/CN2020/111412
Other languages
English (en)
French (fr)
Inventor
穆妮
王太泽
Original Assignee
第四范式（北京）技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 第四范式（北京）技术有限公司 filed Critical 第四范式（北京）技术有限公司
Priority to EP20857623.1A priority Critical patent/EP4024228A4/en
Priority to US17/639,125 priority patent/US11971882B2/en
Publication of WO2021037066A1 publication Critical patent/WO2021037066A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/211—Schema design and management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G06F16/2386—Bulk updating operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • the present disclosure generally relates to the field of data processing, and more specifically, to a system and method for batch and real-time feature calculation.
  • Machine learning is an inevitable product of the development of artificial intelligence research to a certain stage. It is committed to improving the performance of the system itself through computational means and using experience.
  • experience usually exists in the form of "data”.
  • a "model” can be generated from data. That is to say, by providing empirical data to the machine learning algorithm, it can be generated based on these empirical data Model; when facing new situations, use the trained model to get the corresponding prediction results.
  • a machine learning platform or system Based on such machine learning technology, a machine learning platform or system has been produced.
  • feature calculations are required, for example, by extracting features from the data to obtain a machine that includes various features Study samples.
  • tasks used to perform feature calculations may include offline tasks and online tasks.
  • Offline tasks can be used for batch feature calculations. For example, when a scheduled task is started, feature calculations for big data are performed in batches. After all feature calculations are completed, the final execution results are obtained for subsequent research. Offline tasks can target time-insensitive business scenarios. For example, in a banking business scenario, offline tasks can include predicting whether each user will purchase a wealth management product based on a batch of users' consumption behavior data in the previous six months.
  • Online tasks can be used for real-time feature calculations. For example, feature calculations can be performed on a certain amount of data in real time, and the obtained feature calculation results can be used for prediction operations. Online tasks can be targeted at time-sensitive business scenarios, for example, in an online shopping business scenario, related product recommendations are made based on the products that the user has recently viewed.
  • the batch feature calculation logic and the real-time feature calculation logic are not unified, and they are not the same or even conflict. In this case, the two feature calculation logics need to be set and maintained separately.
  • the offline database used for batch feature processing and the real-time database used for real-time feature processing are separated.
  • the data in these two databases are not related and need to be set separately and separately. Maintain these two databases.
  • Exemplary embodiments of the present disclosure are to provide a system and method for batch and real-time feature calculations, in various feature extraction scenarios, based on the associated offline database and real-time database and unified batch and real-time feature calculation logic. Script, perform feature extraction uniformly.
  • a system for batch and real-time feature calculation includes: a management server, which receives a script containing feature calculation logic, and in a batch feature calculation scenario, from registration To the batch feature calculation unit of the management server, select the batch feature calculation unit for batch feature calculation to start the batch feature calculation, and in the real-time feature calculation scenario, deploy the script to the real-time feature calculation unit registered to the management server to generate Calculate the plan in real time, and select the real-time feature calculation unit used when executing the real-time calculation plan from the real-time feature calculation units registered to the management server, wherein the script containing the feature calculation logic is for batch feature calculation logic and real-time feature calculation logic Have uniformity; one or more batch feature calculation units are registered to the management server when activated, and when selected to start batch feature calculation, a batch calculation task is generated, and the batch calculation task is sent to one or more Multiple distributed computing units; the one or more distributed computing units execute the batch computing tasks to perform batch feature calculations based on the script
  • a method for batch and real-time feature calculation includes: receiving a script containing feature calculation logic; in a batch feature calculation scenario, from registration to The batch feature calculation unit used for batch feature calculation is selected from the batch feature calculation unit of the management server to start the batch feature calculation, so that the selected batch feature calculation unit generates a batch calculation task, and sends the batch calculation task to one or more Multiple distributed computing units, so that the one or more distributed computing units execute the batch computing tasks to perform batch feature calculations based on the scripts and data stored in the offline database; in a real-time feature computing scenario, The script is deployed to the real-time feature calculation unit registered to the management server to generate a real-time calculation plan, and the real-time feature calculation unit used when executing the real-time calculation plan is selected from the real-time feature calculation units registered to the management server so as to be selected The real-time feature calculation unit of the real-time feature calculation unit performs real-time feature calculation based on the data stored in the real-time
  • a management server including at least one computing device and at least one storage device storing instructions, wherein, when the instructions are executed by the at least one computing device, Said at least one computing device executes the following steps of the method for batch and real-time feature calculation:
  • the computing unit receives the script containing the feature calculation logic; in the batch feature calculation scenario, select the batch feature calculation unit for batch feature calculation from the batch feature calculation unit registered to the management server to start the batch feature calculation, so that the selected batch feature
  • the computing unit generates batch computing tasks, and sends the batch computing tasks to one or more distributed computing units so that the one or more distributed computing units execute the batch computing tasks based on the script Perform batch feature calculation with the data stored in the offline database;
  • the real-time feature calculation scenario deploy the script to the real-time feature calculation unit registered to the management server to generate a real-time calculation plan, and from the real-time feature calculation unit registered to the management server Select the real-time feature calculation unit to be used when executing the real-time calculation plan, so that the selected real-time feature calculation unit performs real-time feature calculation based on the data stored in the real-time database and the script; wherein the script containing the feature calculation logic is for batch The feature calculation logic and the real-time feature calculation logic are unified.
  • the data stored in the offline database is
  • a computer-readable storage medium storing instructions, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the above method.
  • the system and method of the present disclosure can simultaneously support batch feature calculation and real-time feature calculation based on a unified script and associated two databases, and can solve the inconsistency of batch feature calculation logic and real-time feature calculation logic in machine learning scenarios and offline batch feature processing The problem of separation of database and real-time feature processing of real-time database.
  • the data in the offline database and the data in the real-time database are at least partially synchronized (for example, all data or changed data is synchronized regularly, or each database is updated according to the underlying log file, which can record each database
  • the two databases can have log files or share log files), so as to ensure that the data used in the online scene and the offline scene are consistent.
  • Initializing the real-time database can generate a data field structure corresponding to the real-time feature calculation in the real-time database and obtain the data field structure for the user to use as a reference for input data (for example, sample data), even when the user does not understand the script.
  • the acquired data field structure inputs the at least one piece of data that conforms to the data field structure corresponding to real-time feature calculation for real-time feature calculation; the user’s local raw data can be imported into an offline database for quick data input, and offline
  • the existing data in the database is used as the data for batch feature calculation to avoid repeated import of data. It can also execute at least one of judging whether the script has syntax errors, debugging intermediate running results, debugging final running results, and knowing the correctness of the input scripts and running results in time to avoid running wrong scripts and other processing that affects the efficiency and accuracy of feature calculations .
  • the window data obtained from the real-time database may include the at least one piece of data that conforms to the data field structure corresponding to the real-time feature calculation.
  • the script may be used to construct the fields of the window data (for example, field filtering), window size and other information, and the window data is based on The window slides to update in real time, so that real-time performance and computational efficiency can be guaranteed.
  • the execution status of batch feature calculation can be obtained at any time.
  • the calculation time of batch feature calculation is long, so that the batch feature calculation result can be used for offline training or batch estimation of the model; the calculation time of real-time feature calculation is short, so the real-time feature calculation result can be suitable for prediction based on training model, especially It is online prediction.
  • the script can be run by at least one of each unit and the server. When being run, the script can be executed through script conversion or analysis, so that the script has good compatibility.
  • Fig. 1 shows a schematic structural diagram of a system for batch and real-time feature calculation according to an exemplary embodiment of the present disclosure
  • FIG. 2 shows a timing diagram according to an exemplary embodiment of the present disclosure
  • FIG. 3 shows a timing diagram according to another exemplary embodiment of the present disclosure
  • FIG. 4 shows a flowchart of a method for batch and real-time feature calculation according to an exemplary embodiment of the present disclosure.
  • “including at least one of A and B” and “including A and/or B” include the following three parallel situations: (1) including A; (2) including B; (3) including A and B .
  • “perform at least one of step one and step two” means the following three parallel situations: (1) perform step one; (2) perform step two; (3) Perform steps one and two. That is to say, “A and/or B” can also be expressed as “at least one of A and B”, and “perform step one and/or step two" can also be expressed as "perform one of step one and step two.” At least one of”.
  • Exemplary embodiments may adopt a unified script and two associated databases to support batch feature calculation and real-time feature calculation at the same time.
  • the script here may include scripts written in a feature query language, etc., and may also include codes or scripts written in various computer languages.
  • Fig. 1 shows a schematic structural diagram of a system for batch and real-time feature calculation according to an exemplary embodiment of the present disclosure.
  • the system for batch and real-time feature calculation according to an exemplary embodiment of the present disclosure may include: a management server, one or more batch feature calculation units, one or more distributed computing units, one or more real-time Feature calculation unit, offline database, and real-time database.
  • the management server can receive the script containing the feature calculation logic.
  • the batch feature calculation unit used for batch feature calculation is selected from the batch feature calculation units registered to the management server to start the batch feature calculation, and the feature calculation is performed in real time.
  • deploy the script to the real-time feature calculation unit registered to the management server to generate a real-time calculation plan, and select the real-time feature calculation unit to be used when executing the real-time calculation plan from the real-time feature calculation units registered to the management server,
  • the script containing the feature calculation logic has unity for batch feature calculation logic and real-time feature calculation logic.
  • the one or more batch feature calculation units are registered to the management server when activated, and generate batch calculation tasks when selected to initiate batch feature calculation, and send the batch calculation tasks to the one or more Multiple distributed computing units.
  • the one or more distributed computing units execute the batch calculation task to perform batch feature calculation based on the script and data stored in the offline database.
  • the one or more real-time feature calculation units are registered to the management server when activated, and when selected to execute the real-time calculation plan, perform real-time feature calculation based on the data stored in the real-time database and the script.
  • the offline database may store data used for batch feature calculation (for example, all historical or raw data).
  • the real-time database can obtain and store data used for real-time feature calculation from an offline database.
  • the management server may be implemented as a database management system (DBMS) server or the like.
  • DBMS database management system
  • users can view various required results for the input script, for example, batch feature calculation results, real-time feature calculation results, data field structure, syntax error verification results, intermediate running results obtained through debugging, and/or final Run results, batch feature calculation execution status, etc.
  • the user can input a unified script without using different scripts according to different calculation scenarios.
  • the management server can also be implemented as each unit and/or database of the management system to implement data import operations and task scheduling.
  • the intermediate operation result and/or the final operation result in the present disclosure can also be expressed as at least one of the intermediate operation result and the final operation result.
  • the management server in the present disclosure can also be implemented as each unit and/or database of the management system, and can also be expressed as: the management server can also be implemented as at least one of each unit and database of the management system.
  • the batch feature computing unit can be implemented as a batch computing engine (for example, Spark), the distributed computing unit can be implemented as a distributed computing cluster, the real-time feature computing unit can be implemented as a real-time computing engine, and a real-time database can be implemented It is an in-memory database.
  • a batch computing engine for example, Spark
  • the distributed computing unit can be implemented as a distributed computing cluster
  • the real-time feature computing unit can be implemented as a real-time computing engine
  • a real-time database can be implemented It is an in-memory database.
  • the selection of batch feature computing units, the selection of distributed computing units, and/or the selection of real-time feature computing units can be implemented according to load balancing goals.
  • Batch computing tasks can be executed regularly or according to corresponding trigger instructions.
  • the real-time calculation plan may be executed according to a specific trigger condition (for example, the trigger condition when the data to be predicted (for example, sample data) is received online).
  • batch feature calculation units can be implemented according to the load balancing goal, and can also be expressed as: batch feature calculation At least one of the selection of the unit, the selection of the distributed computing unit, and the selection of the real-time feature computing unit can be achieved according to the load balancing goal.
  • the data in the offline database and the data in the real-time database are at least partially synchronized, where the real-time database and the offline database perform data synchronization according to a specific mechanism, for example, the real-time database and the offline database are timed or based on data changes in response to user triggers. Or perform data synchronization based on a master-slave synchronization mechanism.
  • Data synchronization can be performed based on log files, which are for real-time databases and/or offline databases and can record data changes in the corresponding databases.
  • real-time database and/or offline database in the present disclosure can also be expressed as at least one of the real-time database and the offline database.
  • the real-time database is an in-memory database.
  • the management server also initializes the real-time database to obtain the data field structure (for example, schema) corresponding to the real-time feature calculation.
  • data field structure for example, schema
  • the management server also receives the original data imported by the user and stores the original data in the offline database as data for batch feature calculation, or the management server also uses at least part of the data pre-stored in the offline database as used for batch feature calculation
  • the data In order to import data, an original table can be created in the designated namespace of the offline database, and the original table has a specific data field structure.
  • the management server can receive the local file uploaded by the user from the local to the management server, and create the original table based on the local file.
  • the input parameters of the data import include: the table name and the local file (the local file itself is uploaded) To the management server as an input parameter), the local file can be transferred to the offline database through the management server and stored in the offline database.
  • the input parameters can include: table name and pre-stored in the offline database The storage path of the data.
  • the management server may also verify whether the script has a syntax error, and/or debug at least a part of the intermediate running result and/or the final running result corresponding to the script based on at least a part of the data.
  • the verification result, intermediate running result, and/or final running result of whether there is a syntax error can be fed back to the user so that the user can change the script in time.
  • the management server is configured to verify whether the script has a syntax error, to debug at least a part of the intermediate running result corresponding to the script based on at least a part of data, and to debug based on at least a part of the data.
  • a part of the data is used to debug at least one of at least a part of the final execution result corresponding to the script.
  • At least one of the verification result, intermediate running result, and final running result of whether there is a syntax error can be fed back to the user, so that the user can change the script in time.
  • FIG. 2 shows a timing diagram according to an exemplary embodiment of the present disclosure.
  • each of the real-time computing engines referred to as real-time engines, multiple real-time engines can form a cluster
  • the management server can also initialize the real-time database, the real-time database generates a data field structure corresponding to the real-time feature calculation, and feeds back the data field structure corresponding to the real-time feature calculation to the management server.
  • the management server can also deploy the script to the registered real-time calculation engine to generate a real-time calculation plan, and the real-time calculation engine can feed back the deployment result to the management server.
  • the process of initializing the real-time database can be part of the deployment process.
  • a deployment name can be generated, and deployment information such as the data field structure can be viewed according to the deployment name and/or a real-time calculation plan can be executed.
  • a deployment name can be generated, and deployment information such as the data field structure can be viewed and/or a real-time calculation plan can be executed according to the deployment name, which can also be expressed as:
  • At least one of the following 3 steps can be performed during the deployment process:
  • the management server sends the at least one piece of data to the selected real-time feature calculation unit, wherein the at least one piece of data conforms to A data field structure corresponding to real-time feature calculation, wherein the real-time feature calculation unit requests window data from the real-time database when receiving the at least one piece of data, and performs real-time feature calculation based on the window data fed back by the real-time database and the script, wherein
  • the window data includes all or part of the data fields of the data of the real-time database, and the window data includes the at least one piece of data and a specific amount of data associated with the at least one piece of data in characteristics such as time.
  • the window data is updated in real time according to the sliding of the window; the window is used to process multiple rows of data in a real-time scenario.
  • the window is used to process multiple rows of data in a real-time scenario. For example, in the transaction information, using the user card number as the dimension, a window with a time length of 5 days can be constructed. Specify the maximum data length in the window.
  • Table 1 shows personal transaction data according to an exemplary embodiment of the present disclosure.
  • each column represents a data field; name represents the name, and the type is string; time represents the transaction time, and the type is timestamp; price represents the transaction amount, and the type is double-precision floating point (double).
  • the three rows of data shown in Table 1 can represent three transaction records.
  • the data in Table 1 corresponds to data table t1.
  • the following feature calculation script can be input to the management server:
  • the first line of script is used to construct the window.
  • the target data table is table t1.
  • the windows are aggregated according to the name field and sorted according to the time field.
  • the maximum amount of data in the window is 100, and the maximum time span in the window is 10 days.
  • the second line of script is used to calculate the discrete feature, which is calculated as the sum of the price field in the window.
  • Table 2 shows a data field structure according to an exemplary embodiment of the present disclosure.
  • the data field structure (schema) shown in Table 2 can be generated in the real-time database, and the management server can obtain the schema in the real-time database.
  • the deployment name for the above script can be d1.
  • the script can be deployed to all registered real-time computing engines. If the feedback deployment result does not include abnormal information, the deployment is successful. Then you can start to calculate real-time feature calculations.
  • real-time feature calculation you can enter at least one row of data that conforms to the schema as sample data, and obtain window data based on the sample data.
  • sample data and deployment name d1 When the sample data and deployment name d1 are entered, real-time feature calculation can be started.
  • Table 3 shows sample data according to an exemplary embodiment of the present disclosure. The sample data shown in Table 3 includes one row of data, which conforms to the schema.
  • Window data of the same window as the sample data can be acquired, and real-time feature calculations can be performed based on the acquired window data.
  • Table 4 shows window data according to an exemplary embodiment of the present disclosure, and the window data includes the sample data.
  • the calculation logic for real-time feature calculation comes from the script.
  • the script actually used may only include the first line of script or include the first line of script and the second line of script.
  • the number of lines of the input script may reach hundreds, thousands or even tens of thousands of lines.
  • the workload of script input is reduced, thereby improving feature calculation efficiency.
  • FIG. 3 shows a timing diagram according to another exemplary embodiment of the present disclosure.
  • the batch feature calculation unit implemented as a batch feature calculation engine (referred to as a batch engine) can register with the management server implemented as a DBMS server, and the management server feeds back the registration result to the batch engine .
  • At least one batch engine among multiple registered batch engines can be selected according to load balancing and the script can be run.
  • the batch engine can obtain the original data storage path from the offline database.
  • the batch engine can submit computing tasks to a distributed computing unit implemented as a distributed computing cluster (Hadoop), and the distributed computing cluster feeds back the submission results to the batch engine.
  • Hadoop distributed computing cluster
  • Batch computing tasks can be executed by distributed computing clusters to perform batch feature calculations based on scripts and data stored in offline databases, where data stored in offline databases can be obtained based on the original data storage path.
  • the management server can query the batch engine for the batch feature calculation execution status through the batch feature calculation execution status query request.
  • the batch feature calculation execution status can be provided to the batch engine by the distributed computing cluster and fed back to the management server by the batch engine.
  • the one or more distributed computing units acquire data stored in the offline database based on the data storage path of the offline database, and perform batch feature calculation based on the acquired data and the script, and execute the batch feature calculation
  • the status is sent to the batch feature calculation unit, where the management server obtains the batch feature calculation execution status from the batch feature calculation unit when receiving the batch feature calculation execution status query request.
  • the calculation result of batch feature calculation is used for model training or batch estimation, and the calculation result of real-time feature calculation is used for online prediction based on the trained model.
  • the exemplary embodiments of the present disclosure are applicable to both model training and online prediction based on the trained model.
  • the script when the script is run through the management server, the script is converted into batch feature calculation units, distributed calculation units, and/or real-time feature calculation units that can be executed, and when executed, realize what the script contains Code for feature calculation logic.
  • the above content can also be expressed as: when the script is run through the management server, the script is converted into at least one of a batch feature calculation unit, a distributed calculation unit, and a real-time feature calculation unit that can be executed. , And when executed, the code that implements the feature calculation logic contained in the script, or when the script is executed, at least one of the batch feature calculation unit, the distributed calculation unit, and the real-time feature calculation unit resolves the Describe the feature calculation logic contained in the script.
  • the batch feature calculation unit, the distributed calculation unit, and/or the real-time feature calculation unit parse out the feature calculation logic contained in the script.
  • the script has good compatibility.
  • the above content can also be expressed as: when the script is executed, at least one of the batch feature calculation unit, the distributed calculation unit, and the real-time feature calculation unit parses the feature calculation contained in the script logic.
  • batch and real-time feature calculations can be supported by a set of systems, batch feature calculation logic and real-time feature calculation logic can be unified by a set of scripts, and the data of the offline database and the real-time database are related (real-time database
  • the data can be obtained from offline databases) or even synchronized, which solves the problem of the inconsistency of batch feature calculation logic and real-time feature calculation logic in machine learning scenarios, and the separation of offline database for batch feature processing and real-time database for real-time feature processing.
  • On-line data and off-line data are inconsistent, and limited machine resources can be used to implement machine learning in different scenarios, improve resource utilization, and improve task execution efficiency through unified scripts.
  • FIG. 4 shows a flowchart of a method for batch and real-time feature calculation according to an exemplary embodiment of the present disclosure.
  • the method for batch and real-time feature calculation according to an exemplary embodiment of the present disclosure may include step 110 to step 150.
  • the management server receives the script containing the feature calculation logic; in step 120, the offline database stores the data used for batch feature calculation; in step 130, the real-time database acquires and stores the data used for real-time feature calculation from the offline database.
  • step 140 in the batch feature calculation scenario, the management server selects the batch feature calculation unit for batch feature calculation from the batch feature calculation units registered with the management server to start the batch feature calculation, when the batch feature calculation unit is When selecting to start batch feature calculation, the batch feature calculation unit generates a batch calculation task, sends the batch calculation task to one or more distributed calculation units, and is executed by the one or more distributed calculation units
  • the batch calculation task is to perform batch feature calculation based on the script and the data stored in the offline database;
  • step 150 in the real-time feature calculation scenario, the management server deploys the script to the real-time feature calculation unit registered to the management server To generate a real-time calculation plan, and select the real-time feature calculation unit used when executing the real-time calculation plan from the real-time feature calculation units registered to the management server.
  • the selected The real-time feature calculation unit performs real-time feature calculation based on the data stored in the real-time database and the script; wherein the script containing the feature calculation logic is unified with respect to the batch feature calculation logic and the real-time feature calculation logic.
  • the following introduces specific implementation manners for the management server to execute related steps in the method for batch and real-time feature calculation.
  • the management server includes at least one computing device and at least one storage device that stores instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform batch and The following steps of the method of real-time feature calculation:
  • the batch feature calculation scenario select the batch feature calculation unit used for batch feature calculation from the batch feature calculation units registered to the management server to start the batch feature calculation, so that the selected batch feature calculation unit generates the batch calculation task, and
  • the batch computing task is sent to one or more distributed computing units, so that the one or more distributed computing units execute the batch computing task to perform batches based on the script and the data stored in the offline database Feature calculation
  • the script deploys the script to the real-time feature calculation unit registered to the management server to generate a real-time calculation plan, and select the real-time feature used when executing the real-time calculation plan from the real-time feature calculation unit registered to the management server A calculation unit, so that the selected real-time feature calculation unit performs real-time feature calculation based on the data stored in the real-time database and the script;
  • the script containing the feature calculation logic has unity with respect to the batch feature calculation logic and the real-time feature calculation logic.
  • the data stored in the offline database is the data used for batch feature calculation; the data stored in the real-time database is the data obtained from the offline database. Data for real-time feature calculation.
  • the method further includes: performing data synchronization of at least part of the data based on a data change or based on a master-slave synchronization mechanism by the real-time database and the offline database regularly or in response to a user trigger.
  • the method further includes that the management server initializes the real-time database to obtain the data field structure corresponding to the real-time feature calculation.
  • the method further includes at least one of the following steps:
  • the management server receives the original data imported by the user and stores the original data in the offline database as data for batch feature calculation;
  • the management server uses at least a part of the data pre-stored in the offline database as data for batch feature calculation.
  • the real-time database is an in-memory database.
  • the method further includes: verifying by the management server whether the script has a syntax error, and/or debugging at least a part of the intermediate running result and/or the final running result corresponding to the script based on at least a part of the data.
  • the above content can also be expressed as that the method further includes at least one of the following steps:
  • the management server verifies whether the script has syntax errors
  • At least a part of the final running result corresponding to the script is debugged based on at least a part of the data.
  • the method further includes: when at least one piece of data for real-time feature calculation is received by the management server and the real-time calculation plan is executed, sending the at least one piece of data to the selected real-time feature calculation unit, wherein, The at least one piece of data conforms to the data field structure corresponding to the real-time feature calculation; the selected real-time feature calculation unit requests the real-time database for window data when receiving the at least one piece of data, and performs real-time based on the window data and the script Feature calculation, wherein the window data includes all or part of the data fields of the data of the real-time database.
  • the step of performing batch feature calculation includes: the one or more distributed computing units obtain data stored in the offline database based on the data storage path of the offline database, and execute the batch based on the obtained data and the script. Feature calculation, and sending the batch feature calculation execution status to the batch feature calculation unit, wherein the method further includes: when the management server receives the batch feature calculation execution status query request, the management server obtains the batch feature from the batch feature calculation unit Calculate the execution status.
  • the calculation result of batch feature calculation is used for model training or batch estimation, and the calculation result of real-time feature calculation is used for prediction based on the trained model, and the prediction includes online prediction.
  • the method further includes: when the script is executed by the management server, converting the script into a batch feature calculation unit, a distributed calculation unit, and/or a real-time feature calculation unit that can be executed and realized when executed The code of the feature calculation logic contained in the script.
  • the above content can also be expressed as: the script runs the script through the management server, and converts the script into at least one of a batch feature calculation unit, a distributed calculation unit, and a real-time feature calculation unit that can be executed. And when executed, it implements the code of the feature calculation logic contained in the script.
  • the feature calculation logic contained in the script is parsed by the batch feature calculation unit, the distributed calculation unit, and/or the real-time feature calculation unit.
  • the above content can also be expressed as: when the script is executed, at least one of the batch feature calculation unit, the distributed calculation unit, and the real-time feature calculation unit parses the feature calculation contained in the script logic.
  • the units or servers included in the system for batch and real-time feature calculation may be respectively configured as software, hardware, firmware, or any combination of the foregoing items that perform specific functions.
  • these units or servers can correspond to dedicated integrated circuits, can also correspond to pure software codes, and can also correspond to modules combining software and hardware.
  • one or more functions implemented by these units or servers may also be uniformly performed by components in a physical physical device (for example, a processor, a client, or a server, etc.).
  • the method for batch and real-time feature calculation can be implemented by a program recorded on a computer-readable medium.
  • a Computer-readable media for batch and real-time feature calculations may be multiple computer-readable media, and multiple computer-readable media are respectively deployed on the management server, offline database, real-time database, batch feature computing unit, distributed computing unit, and real-time feature computing unit in.
  • the management server receives the script containing the feature calculation logic; the offline database stores the data used for batch feature calculation; the real-time database from offline The database acquires and stores data for real-time feature calculation; in the batch feature calculation scenario, the management server selects the batch feature calculation unit used for batch feature calculation from the batch feature calculation units registered to the management server to start the batch feature calculation, When the batch feature calculation unit is selected to start the batch feature calculation, the batch feature calculation unit generates the batch calculation task, sends the batch calculation task to one or more distributed calculation units, and the one or more A distributed computing unit executes the batch computing tasks to perform batch feature calculations based on the scripts and data stored in the offline database; in the real-time feature computing scenario, the management server deploys data to the real-time feature computing unit registered to the management server The script is used to generate a real-time calculation plan, and the real-time feature calculation unit used when executing the real-time calculation plan is selected from the real
  • the The selected real-time feature calculation unit When the real-time feature calculation unit is selected to execute the real-time calculation plan, the The selected real-time feature calculation unit performs real-time feature calculation based on the data stored in the real-time database and the script; wherein, the script containing the feature calculation logic is unified with respect to the batch feature calculation logic and the real-time feature calculation logic.
  • the computer program in the above-mentioned computer-readable medium can be run in an environment deployed in computer equipment such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above-mentioned steps or More specific processing is performed when the above steps are performed. These additional steps and further processing contents have been described with reference to FIGS. 1 to 3, and will not be repeated here in order to avoid repetition.
  • each unit or server corresponds to the functional architecture of the computer program, so that the entire system It is called by a special software package (for example, lib library) to realize the corresponding function.
  • lib library for example, lib library
  • each unit or server included in the system for batch and real-time feature calculation may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof.
  • the program code or code segment used to perform the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that the processor can read and run the corresponding program Code or code segment to perform the corresponding operation.
  • the exemplary embodiments of the present disclosure may also be implemented as a computing device, the computing device includes a storage component and a processor, the storage component stores a set of computer-executable instructions, when the set of computer-executable instructions is used by the processor When executing, execute methods for batch and real-time feature calculations.
  • the computing device is different from the computing device in the aforementioned management server. Specifically, the computing device can execute the steps performed by any one of the above-mentioned management server, offline database, real-time database, batch feature computing unit, distributed computing unit, and real-time feature computing unit, and the calculation in the above-mentioned management server The device only performs the steps performed by the management server.
  • the computing device can be deployed in a server or a client, and can also be deployed on a node device in a distributed network environment.
  • the computing device may be a PC computer, a tablet device, a personal digital assistant, a smart phone, a web application, or other devices capable of executing the above set of instructions.
  • the computing device does not have to be a single computing device, and may also be any device or a collection of circuits that can execute the above-mentioned instructions (or instruction sets) individually or jointly.
  • the computing device may also be a part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
  • the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor.
  • the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, and the like.
  • Some operations described in the method for batch and real-time feature calculation according to exemplary embodiments of the present disclosure can be implemented by software, and some operations can be implemented by hardware. In addition, it can also be implemented by a combination of software and hardware. Way to achieve these operations.
  • the processor can run instructions or codes stored in one of the storage components, where the storage component can also store data. Instructions and data can also be sent and received via a network via a network interface device, where the network interface device can use any known transmission protocol.
  • the storage component can be integrated with the processor, for example, RAM or flash memory is arranged in an integrated circuit microprocessor or the like.
  • the storage component may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system.
  • the storage component and the processor may be operatively coupled, or may communicate with each other, for example, through an I/O port, a network connection, or the like, so that the processor can read files stored in the storage component.
  • the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the computing device may be connected to each other via a bus and/or network.
  • a video display such as a liquid crystal display
  • a user interaction interface such as a keyboard, a mouse, a touch input device, etc.
  • a computing device for batch and real-time feature calculation may include a storage component and a processor, wherein the storage component stores a set of computer-executable instructions, and when the computer can When the execution instruction set is executed by the processor, the following steps are executed: the management server receives the script containing the feature calculation logic; the offline database stores the data for batch feature calculation; the real-time database acquires and stores it from the offline database for use Real-time feature calculation data; in the batch feature calculation scenario, the management server selects the batch feature calculation unit used for batch feature calculation from the batch feature calculation units registered to the management server to start the batch feature calculation.
  • the batch feature calculation unit When the batch feature calculation unit is When selecting to start batch feature calculation, the batch feature calculation unit generates a batch calculation task, sends the batch calculation task to one or more distributed calculation units, and is executed by the one or more distributed calculation units
  • the batch calculation task is to perform batch feature calculation based on the script and the data stored in the offline database; in the real-time feature calculation scenario, the management server deploys the script to the real-time feature calculation unit registered to the management server to generate real-time calculations Plan, and select the real-time feature calculation unit used when executing the real-time calculation plan from the real-time feature calculation units registered to the management server.
  • the selected real-time feature calculation unit Real-time feature calculation is performed based on the data stored in the real-time database and the script; wherein the script containing the feature calculation logic has unity for batch feature calculation logic and real-time feature calculation logic.
  • the system and method for batch and real-time feature calculation provided by the present disclosure can support batch feature calculation and real-time feature calculation at the same time based on a unified script and two associated databases, and can solve batch feature calculation logic and real-time feature calculation in machine learning scenarios The problem of inconsistent logic and separation of offline database for batch feature processing and real-time database for real-time feature processing.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Business, Economics & Management (AREA)
  • Economics (AREA)
  • Human Resources & Organizations (AREA)
  • Strategic Management (AREA)
  • Marketing (AREA)
  • Development Economics (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Game Theory and Decision Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本公开提供了一种用于批量和实时特征计算的系统和方法，所述系统包括：管理服务器、一个或更多个批量特征计算单元、一个或更多个分布式计算单元、一个或更多个实时特征计算单元、离线数据库、以及实时数据库。所述管理服务器接收包含特征计算逻辑的脚本，在批量特征计算场景下，启动批量特征计算，并且在实时特征计算场景下，部署所述脚本以产生实时计算计划，其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性，实时数据库从离线数据库获取并存储用于实时特征计算的数据。根据本公开，可通过所述脚本统一批量特征计算逻辑和实时特征计算逻辑。

Description

用于批量和实时特征计算的系统和方法
本申请要求申请号为201910789592.2，申请日为2019年8月26日，名称为“用于批量和实时特征计算的系统和方法”的中国专利申请的优先权，其中，上述申请公开的内容通过引用结合在本申请中。
技术领域
本公开总体说来涉及数据处理领域，更具体地讲，涉及一种用于批量和实时特征计算的系统和方法。
背景技术
随着海量数据的出现，人们倾向于使用机器学习技术来从数据中挖掘出价值。机器学习是人工智能研究发展到一定阶段的必然产物，其致力于通过计算的手段，利用经验来改善系统自身的性能。在计算机系统中，“经验”通常以“数据”形式存在，通过机器学习算法，可从数据中产生“模型”，也就是说，将经验数据提供给机器学习算法，就能基于这些经验数据产生模型；在面对新的情况时，利用训练好的模型来得到相应的预测结果。
基于这样的机器学习技术，产生了机器学习平台或系统。在机器学习平台或系统中，不论是训练机器学习模型的阶段，还是利用机器学习模型进行预估的阶段，都需要进行特征计算，例如，通过对数据进行特征抽取来得到包括各种特征的机器学习样本。
在现有的机器学习平台或系统中，用于执行特征计算的任务可包括线下任务和线上任务。
线下任务可用于批量特征计算，例如，当计划任务被启动时，批量的执行大数据的特征计算，待全部特征计算完成后，获取最终的执行结果用于后续的研究。线下任务可针对时间不敏感的业务场景，例如，在银行业务场景中，线下任务可包括根据一批用户前六个月的消费行为数据，预测各个用户是否购买理财产品。
线上任务可用于实时特征计算，例如，可实时的对一定数量的数据进行特征计算，获取的特征计算结果可用于预测操作。线上任务可针对时间敏感的业务场景，例如，在线上购物业务场景中，根据用户在最近浏览的商品进行相关产品推荐。
然而，在现有的机器学习平台或系统中，批量特征计算逻辑和实时特征计算逻辑是不统一的，两者不相同甚至存在冲突。在这种情况下，需要对两种特征计算逻辑分别进行设置和维护。
另外，在现有的机器学习平台或系统中，用于批量特征处理的离线数据库和用于实时特征处理的实时数据库是分离的，这两种个数据库中的数据不 关联，需要分别设置并分别维护这两种数据库。
发明内容
本公开的示例性实施例在于提供一种用于批量和实时特征计算的系统和方法，以在各种特征抽取场景下，基于关联的离线数据库和实时数据库以及统一了批量和实时特征计算逻辑的脚本，统一的执行特征抽取。
根据本公开的示例性实施例，提供了一种用于批量和实时特征计算的系统，其中，所述系统包括：管理服务器，接收包含特征计算逻辑的脚本，在批量特征计算场景下，从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，并且在实时特征计算场景下，向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性；一个或更多个批量特征计算单元，当被启动时注册到管理服务器，并且当被选择以启动批量特征计算时产生批量计算任务，并且将所述批量计算任务发送给一个或更多个分布式计算单元；所述一个或更多个分布式计算单元，执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；一个或更多个实时特征计算单元，当被启动时注册到管理服务器，并且当被选择以执行实时计算计划时，基于实时数据库存储的数据和所述脚本进行实时特征计算；离线数据库，存储用于批量特征计算的数据；实时数据库，从离线数据库获取并存储用于实时特征计算的数据。
根据本公开的另一示例性实施例，提供了一种用于批量和实时特征计算的方法，其中，所述方法包括：接收包含特征计算逻辑的脚本；在批量特征计算场景下，从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，以使被选择的批量特征计算单元产生批量计算任务，将所述批量计算任务发送给一个或更多个分布式计算单元，以使所述一个或更多个分布式计算单元执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；在实时特征计算场景下，向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并且从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，以使被选择的实时特征计算单元基于实时数据库存储的数据和所述脚本进行实时特征计算；其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性，离线数据库存储的数据为用于批量特征计算的数据；实时数据库存储的数据为从离线数据库获取的用于实时特征计算的数据。
根据本公开的另一示例性实施例，提供了一种包括至少一个计算装置和至少一个存储指令的存储装置的管理服务器，其中，所述指令在被所述至少 一个计算装置运行时，促使所述至少一个计算装置执行用于批量和实时特征计算的方法的以下步骤：
接收包含特征计算逻辑的脚本；在批量特征计算场景下，从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，以使被选择的批量特征计算单元产生批量计算任务，将所述批量计算任务发送给一个或更多个分布式计算单元，以使所述一个或更多个分布式计算单元执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；在实时特征计算场景下，向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并且从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，以使被选择的实时特征计算单元基于实时数据库存储的数据和所述脚本进行实时特征计算；其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性，离线数据库存储的数据为用于批量特征计算的数据；实时数据库存储的数据为从离线数据库获取的用于实时特征计算的数据。
根据本公开的另一示例性实施例，提供了一种存储指令的计算机可读存储介质，其中，当所述指令被至少一个计算装置运行时，促使所述至少一个计算装置执行如上所述的方法。
本公开的系统和方法可基于统一的脚本和关联的两种数据库同时支持批量特征计算和实时特征计算，可解决机器学习场景下批量特征计算逻辑和实时特征计算逻辑不统一以及批量特征处理的离线数据库和实时特征处理的实时数据库分离的问题。
另外，离线数据库中的数据和实时数据库中的数据至少部分同步(例如，定时同步全部数据或有变化的数据，或者，根据底层日志文件来更新每个数据库，所述底层日志文件可记录各个数据库中的数据的变化，两种数据库可分别具有日志文件也可共用日志文件)，从而保证在线场景和离线场景使用的数据一致。初始化实时数据库可在实时数据库中产生与实时特征计算对应的数据字段结构并且获取所述数据字段结构以供用户作为输入数据(例如，样本数据)的参考，即使在用户不理解脚本时也可根据获取的所述数据字段结构输入符合与实时特征计算对应的数据字段结构的所述至少一条数据，以供实时特征计算；可将用户本地的原始数据导入离线数据库便于快速输入数据，还可将离线数据库中已有数据作为用于批量特征计算的数据以避免数据重复导入。还可以执行判断脚本是否存在语法错误、调试中间运行结果、调试最终运行结果之中的至少一个，以及时获知输入的脚本的正确性以及运行结果避免运行错误脚本等处理影响特征计算效率和准确性。从实时数据库获取的窗口数据可包括符合与实时特征计算对应的数据字段结构的所述至少一条数据，脚本可用于构造窗口数据的字段(例如，进行字段筛选)、窗口大小等信息，窗口数据根据窗口滑动进行实时更新，从而即可保证实时性又可保证计算效率。在离线(批量)特征计算的过程中，可随时获取批量特征计算执行 状态。批量特征计算计算时间长，从而批量特征计算结果可适用于用于模型的离线训练或批量预估；实时特征计算的计算时间短，从而实时特征计算结果可适用于基于训练的模型的预测，尤其是在线预测。脚本可被各个单元以及服务器中的至少一个运行，在被运行时，可通过脚本转化或解析而使得脚本可被执行，从而脚本具有良好的兼容性。
将在接下来的描述中部分阐述本公开总体构思另外的方面和优点，还有一部分通过描述将是清楚的，或者可以经过本公开总体构思的实施而得知。
附图说明
通过下面结合示例性地示出实施例的附图进行的描述，本公开示例性实施例的上述和其他目的和特点将会变得更加清楚，其中：
图1示出根据本公开的示例性实施例的用于批量和实时特征计算的系统的结构示意图；
图2示出根据本公开的示例性实施例的时序图；
图3示出根据本公开的另一示例性实施例的时序图；
图4示出根据本公开的示例性实施例的用于批量和实时特征计算的方法的流程图。
具体实施方式
现将详细参照本公开的实施例，所述实施例的示例在附图中示出，其中，相同的标号始终指的是相同的部件。以下将通过参照附图来说明所述实施例，以便解释本公开。在此需要说明的是，在本公开中出现的“若干项之中的至少一项”均表示包含“该若干项中的任意一项”、“该若干项中的任意多项的组合”、“该若干项的全体”这三类并列的情况。在本公开中出现的“和/或”均表示被其连接的前后两项或多项中的至少一项。例如，“包括A和B之中的至少一个”、“包括A和/或B”即包括如下三种并列的情况：(1)包括A；(2)包括B；(3)包括A和B。又例如，“执行步骤一和步骤二之中的至少一个”、“执行步骤一和/或步骤二”即表示如下三种并列的情况：(1)执行步骤一；(2)执行步骤二；(3)执行步骤一和步骤二。也就是说，“A和/或B”也可被表示为“A和B之中的至少一个”，“执行步骤一和/或步骤二”也可被表示为“执行步骤一和步骤二之中的至少一个”。
为了解决机器学习场景下批量特征计算和实时特征计算分离(例如，批量特征计算逻辑和实时特征计算逻辑不统一)以及批量特征处理的离线数据库和实时特征处理的实时数据库分离的问题，本公开的示例性实施例可采用统一的脚本和关联的两种数据库以同时支持批量特征计算和实时特征计算。这里的脚本可包括通过特征查询语言等编写的脚本，还可包括通过各种计算机语言编写的代码或脚本。
图1示出根据本公开的示例性实施例的用于批量和实时特征计算的系统的结构示意图。根据本公开的示例性实施例的用于批量和实时特征计算的系 统可包括：管理服务器、一个或更多个批量特征计算单元、一个或更多个分布式计算单元、一个或更多个实时特征计算单元、离线数据库、以及实时数据库。
管理服务器可接收包含特征计算逻辑的脚本，在批量特征计算场景下，从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，并且在实时特征计算场景下，向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性。所述一个或更多个批量特征计算单元，当被启动时注册到管理服务器，并且当被选择以启动批量特征计算时产生批量计算任务，并且将所述批量计算任务发送给所述一个或更多个分布式计算单元。所述一个或更多个分布式计算单元，执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算。所述一个或更多个实时特征计算单元，当被启动时注册到管理服务器，并且当被选择以执行实时计算计划时，基于实时数据库存储的数据和所述脚本进行实时特征计算。所述离线数据库可存储用于批量特征计算的数据(例如，全部历史或原始数据)。所述实时数据库可从离线数据库获取并存储用于实时特征计算的数据。
作为示例，管理服务器可被实现为数据库管理系统(DBMS)服务器等。通过管理服务器，用户可针对输入的脚本来查看各种需要的结果，例如，批量特征计算结果、实时特征计算结果、数据字段结构、语法错误验证结果、通过调试获得的中间运行结果和/或最终运行结果、批量特征计算执行状态等。在批量特征计算场景下和在实时特征计算场景下，用户可输入统一的脚本，而无需根据不同计算场景使用不同脚本。管理服务器还可被实现为管理系统的各个单元和/或数据库，以实现数据导入操作和任务调度等。
在此需要说明的是，本公开中的中间运行结果和/或最终运行结果，还可以表述为中间运行结果和最终运行结果之中的至少一个。本公开中的管理服务器还可被实现为管理系统的各个单元和/或数据库，还可以表述为：管理服务器还可被实现为管理系统的各个单元和数据库之中的至少一个。
作为示例，批量特征计算单元可被实现为批量计算引擎(例如，Spark)，分布式计算单元可被实现为分布式计算集群，实时特征计算单元可被实现为实时计算引擎，实时数据库可被实现为内存数据库。
作为示例，批量特征计算单元的选择、分布式计算单元的选择和/或实时特征计算单元的选择，可依据负载均衡目标来实现。批量计算任务可定时被执行或者根据相应的触发指令而被执行。实时计算计划可根据特定触发条件(例如，在线接收到待预测数据(例如，样本数据)时的触发条件)而被执行。
在此需要说明的是，本公开中的批量特征计算单元的选择、分布式计算单元的选择和/或实时特征计算单元的选择，可依据负载均衡目标来实现，还 可以表述为：批量特征计算单元的选择、分布式计算单元的选择和实时特征计算单元的选择之中的至少一个，可依据负载均衡目标来实现。
作为示例，离线数据库中的数据和实时数据库中的数据至少部分同步，其中，实时数据库与离线数据库按照特定的机制执行数据同步，例如，实时数据库与离线数据库定时或响应于用户触发而基于数据变化或者基于主从同步机制进行数据同步。数据同步可基于日志文件来进行，所述日志文件针对实时数据库和/或离线数据库，可记载相应数据库的数据变化情况。通过数据同步，可保证在批量特征计算场景下和在实时特征计算场景下数据的一致性，使得在使用同样的脚本和数据进行批量特征计算和实时特征计算可获得同样的计算结果。
在此需要说明的是，本公开中的实时数据库和/或离线数据库，还可以表述为实时数据库和离线数据库之中的至少一个。
作为示例，实时数据库是内存数据库。
作为示例，管理服务器还初始化实时数据库，以获取与实时特征计算对应的数据字段结构(例如，schema)。
作为示例，管理服务器还接收用户导入的原始数据并且将原始数据存储到离线数据库中作为用于批量特征计算的数据，或者管理服务器还将预先存入离线数据库的至少一部分数据作为用于批量特征计算的数据。为了进行数据导入，可在离线数据库的指定命名空间下创建原始表，该原始表具有特定数据字段结构。在导入原始数据时，管理服务器可接收用户从本地上传到管理服务器的本地文件，并基于所述本地文件来创建原始表，数据导入的输入参数包括：表名称和本地文件(本地文件本身被上传到管理服务器作为输入参数)，本地文件可通过管理服务器被转送给离线数据库并且被存储到离线数据库中。当使用预先存入离线数据库的数据作为用于批量特征计算的数据时，可基于离线数据库中的数据存储路径来实现，在这种情况下，输入参数可包括：表名称以及预先存入离线数据库的数据的存储路径。
作为示例，管理服务器还可验证所述脚本是否存在语法错误，和/或基于至少一部分数据来调试与所述脚本对应的至少一部分中间运行结果和/或最终运行结果。是否存在语法错误的验证结果、中间运行结果和/或最终运行结果可反馈给用户，以便用户可及时更改脚本。
在此需要说明的是，上述内容还可以表述为：管理服务器被配置为用于验证脚本是否存在语法错误、用于基于至少一部分数据来调试与脚本对应的至少一部分中间运行结果、用于基于至少一部分数据来调试与脚本对应的至少一部分最终运行结果之中的至少一个。是否存在语法错误的验证结果、中间运行结果和最终运行结果之中的至少一个结果可反馈给用户，以便用户可及时更改脚本。
图2示出根据本公开的示例性实施例的时序图。如图2中所示，实时计算引擎(简称为实时引擎，多个实时引擎可形成集群)中的每个实时计算引擎在被启动时可向被实现为DBMS服务器的管理服务器进行注册，管理服务 器向实时计算引擎反馈注册结果。管理服务器还可初始化实时数据库，实时数据库产生与实时特征计算对应的数据字段结构，并向管理服务器反馈与实时特征计算对应的数据字段结构。管理服务器还可将脚本部署到注册的实时计算引擎，以产生实时计算计划，实时计算引擎可将部署结果反馈给管理服务器。初始化实时数据库的过程可属于部署过程的一部分。在部署过程中，可产生部署名称，可根据部署名称查看所述数据字段结构等部署信息和/或执行实时计算计划。
在本公开中，在部署过程中，可产生部署名称，可根据部署名称查看所述数据字段结构等部署信息和/或执行实时计算计划，还可以表述为：
在部署过程中可以执行以下3个步骤中的至少一个步骤：
产生部署名称；
根据部署名称查看所述数据字段结构等部署信息；
执行实时计算计划。
作为示例，管理服务器还在接收到用于实时特征计算的至少一条数据并且实时计算计划被执行时，将所述至少一条数据发送给被选择的实时特征计算单元，其中，所述至少一条数据符合与实时特征计算对应的数据字段结构，其中，实时特征计算单元在接收到所述至少一条数据时向实时数据库请求窗口数据，并且基于实时数据库反馈的窗口数据以及所述脚本进行实时特征计算，其中，所述窗口数据包含实时数据库的数据的全部或部分数据字段，所述窗口数据包括所述至少一条数据以及与所述至少一条数据在时间等特征上存在关联的特定数量的数据。作为示例，窗口数据根据窗口滑动而实时更新；窗口用于实时场景下的多行数据的处理，例如，在交易信息中，以用户卡号为维度，可构造一个时间长度为5天的窗口，可指定窗口内的最大数据长度。
以下结合表1至表4来描述实时特征计算过程，表1示出根据本公开的示例性实施例的个人交易数据。在表1中，每一列表示一个数据字段；name表示姓名，类型为字符串型(string)；time表示交易时间，类型为时间戳(timestamp)；price表示交易金额，类型为双精度浮点型(double)。表1示出的三行数据可表示三条交易记录。表1中的数据对应数据表t1。
表1
name:string time:timestamp price:double
Wtz 1556152031000 1.0
Wtz 1556065631000 2.0
Wtz 1555979231000 3.0
可向管理服务器输入如下的特征计算脚本：
w＝window(t1,"name","time",100,10d)
f＝continuous(sum(w.price))
第一行脚本用于构造窗口，针对的数据表是表t1，窗口按照name字段 聚合且按照time字段排序，窗口内最大数据量为100，窗口内最大时间跨度为10天。第二行脚本用于计算离散特征，计算为窗口内price字段之和。
表2示出根据本公开的示例实施例的数据字段结构。可在实时数据库中产生表2所示数据字段结构(schema)，并由管理服务器获取实时数据库中的schema。
表2
字段名称 字段类型
name string
time timestamp
price double
针对以上脚本的部署名称可以是d1。可将脚本部署至全部已注册的实时计算引擎。如果反馈的部署结果不包括异常信息，则部署成功。随后可开始计算实时特征计算。
在实时特征计算时，可输入至少一行符合schema的数据作为样本数据，根据样本数据获取窗口数据。当输入样本数据和部署名称d1时可开始实时特征计算。表3示出根据本公开的示例性实施例的样本数据。表3中示出的样本数据包括一行数据，该数据符合schema。
表3
name:string time:timestamp price:double
Wtz 1556238431000 4.0
可获取与所述样本数据同一窗口的窗口数据，并基于获取的窗口数据进行实时特征计算，表4示出了根据本公开的示例性实施例的窗口数据，该窗口数据包括所述样本数据。
表4
name:string time:timestamp price:double
Wtz 1556152031000 1.0
Wtz 1556065631000 2.0
Wtz 1555979231000 3.0
Wtz 1556238431000 4.0
用于实时特征计算的计算逻辑来自于所述脚本。实际使用的脚本可仅包括第一行脚本或者包含第一行脚本和第二行脚本。当然，当进行大量特征的抽取时，输入的脚本的行数可能达到成百、上千甚至上万行。在这种情况下，由于不需要分别针对批量特征计算场景和实时特征计算场景的脚本，因此减少脚本输入的工作量，从而可提高特征计算效率。
图3示出根据本公开的另一示例性实施例的时序图。如图3中所示，当被启动时，被实施为批量特征计算引擎(简称为批量引擎)的批量特征计算单元可向被实施为DBMS服务器的管理服务器注册，管理服务器向批量引擎反馈注册结果。可根据负载均衡选择多个已经注册的批量引擎中的至少一个批量引擎并且运行脚本。批量引擎可从离线数据库获取原始数据存储路径。批量引擎可向实施为分布式计算集群(Hadoop)的分布式计算单元提交计算任务，分布式计算集群向批量引擎反馈提交结果。批量计算任务可由分布式计算集群来执行，以基于脚本和离线数据库存储的数据进行批量特征计算，其中，离线数据库存储的数据可基于原始数据存储路径而获取。管理服务器可通过批量特征计算执行状态查询请求向批量引擎查询批量特征计算执行状态，批量特征计算执行状态可由分布式计算集群提供给批量引擎，并由批量引擎反馈给管理服务器。
作为示例，所述一个或更多个分布式计算单元基于离线数据库的数据存储路径来获取离线数据库存储的数据，并且基于获取的数据、所述脚本来执行批量特征计算，并且将批量特征计算执行状态发送给批量特征计算单元，其中，管理服务器在接收到批量特征计算执行状态查询请求时，从批量特征计算单元获取批量特征计算执行状态。
作为示例，批量特征计算的计算结果用于模型训练或批量预估，实时特征计算的计算结果用于基于训练的模型进行在线预测。在这种情况下，本公开的示例性实施例既适用于模型训练也适用于基于训练的模型的在线预测。
作为示例，当所述脚本通过管理服务器被运行时，所述脚本被转化为批量特征计算单元、分布式计算单元和/或实时特征计算单元能够执行且当被执行时实现所述脚本所包含的特征计算逻辑的代码。
在本公开中，上述内容还可以表述为：当所述脚本通过管理服务器被运行时，所述脚本被转化为批量特征计算单元、分布式计算单元和实时特征计算单元之中的至少一个能够执行、且当被执行时实现所述脚本所包含的特征计算逻辑的代码，或者当所述脚本被运行时，批量特征计算单元、分布式计算单元和实时特征计算单元之中的至少一个解析出所述脚本中所包含的特征计算逻辑。
作为示例，当所述脚本被运行时，批量特征计算单元、分布式计算单元和/或实时特征计算单元解析出所述脚本中所包含的特征计算逻辑。在这种情况下，脚本具有良好的兼容性。
在本公开中，上述内容还可以表述为：当所述脚本被运行时，批量特征计算单元、分布式计算单元和实时特征计算单元之中的至少一个解析出所述脚本中所包含的特征计算逻辑。
根据本公开的示例性实施例，可通过一套系统支持批量和实时特征计算，通过一套脚本统一了批量特征计算逻辑和实时特征计算逻辑，并且离线数据库和实时数据库的数据存在关联(实时数据库中的数据可从离线数据库获取)甚至同步，从而解决了机器学习场景下批量特征计算逻辑和实时特征计算逻 辑不统一以及批量特征处理的离线数据库和实时特征处理的实时数据库分离的问题，避免线上数据和线下数据不一致，可利用有限的机器资源实现不同场景下的机器学习，提高资源利用率，还可通过统一的脚本提高任务执行效率。
图4示出根据本公开的示例性实施例的用于批量和实时特征计算的方法的流程图。如图4中所示，根据本公开的示例性实施例的用于批量和实时特征计算的方法可包括步骤110至步骤150。在步骤110，由管理服务器接收包含特征计算逻辑的脚本；在步骤120，由离线数据库存储用于批量特征计算的数据；在步骤130，由实时数据库从离线数据库获取并存储用于实时特征计算的数据；在步骤140，在批量特征计算场景下，由管理服务器从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，当批量特征计算单元被选择以启动批量特征计算时，由批量特征计算单元产生批量计算任务，将所述批量计算任务发送给一个或更多个分布式计算单元，并且由所述一个或更多个分布式计算单元执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；在步骤150，在实时特征计算场景下，由管理服务器向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并且从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，当实时特征计算单元被选择以执行实时计算计划时，由被选择的实时特征计算单元基于实时数据库存储的数据和所述脚本进行实时特征计算；其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性。
下面介绍管理服务器执行用于批量和实时特征计算的方法中相关步骤的具体的实施方式。
作为示例，管理服务器包括至少一个计算装置和至少一个存储指令的存储装置的管理服务器，其中，所述指令在被所述至少一个计算装置运行时，促使所述至少一个计算装置执行用于批量和实时特征计算的方法的以下步骤：
接收包含特征计算逻辑的脚本；
在批量特征计算场景下，从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，以使被选择的批量特征计算单元产生批量计算任务，将所述批量计算任务发送给一个或更多个分布式计算单元，以使所述一个或更多个分布式计算单元执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；
在实时特征计算场景下，向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并且从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，以使被选择的实时特征计算单元基于实时数据库存储的数据和所述脚本进行实时特征计算；
其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性，离线数据库存储的数据为用于批量特征计算的数据； 实时数据库存储的数据为从离线数据库获取的用于实时特征计算的数据。
作为示例，所述方法还包括：由实时数据库与离线数据库定时或响应于用户触发而基于数据变化或者基于主从同步机制进行至少部分数据的数据同步。
作为示例，所述方法还包括管理服务器初始化实时数据库，以获取与实时特征计算对应的数据字段结构。
作为示例，所述方法还包括以下步骤中的至少一个步骤：
管理服务器接收用户导入的原始数据并且将原始数据存储到离线数据库中作为用于批量特征计算的数据；
管理服务器将预先存入离线数据库的至少一部分数据作为用于批量特征计算的数据。
作为示例，实时数据库是内存数据库。
作为示例，所述方法还包括：由管理服务器验证所述脚本是否存在语法错误，和/或，基于至少一部分数据来调试与所述脚本对应的至少一部分中间运行结果和/或最终运行结果。
在本公开中，上述内容还可以表述为，所述方法还包括以下步骤中的至少一个步骤：
由管理服务器验证所述脚本是否存在语法错误；
基于至少一部分数据来调试与所述脚本对应的至少一部分中间运行结果；
基于至少一部分数据来调试与所述脚本对应的至少一部分最终运行结果。
作为示例，所述方法还包括：由管理服务器在接收到用于实时特征计算的至少一条数据并且实时计算计划被执行时，将所述至少一条数据发送给被选择的实时特征计算单元，其中，所述至少一条数据符合与实时特征计算对应的数据字段结构；由被选择的实时特征计算单元在接收到所述至少一条数据时向实时数据库请求窗口数据，并且基于窗口数据以及所述脚本进行实时特征计算，其中，所述窗口数据包含实时数据库的数据的全部或部分数据字段。
作为示例，进行批量特征计算的步骤包括：由所述一个或更多个分布式计算单元基于离线数据库的数据存储路径来获取离线数据库存储的数据，并且基于获取的数据、所述脚本来执行批量特征计算，并且将批量特征计算执行状态发送给批量特征计算单元，其中，所述方法还包括：在管理服务器接收到批量特征计算执行状态查询请求时，由管理服务器从批量特征计算单元获取批量特征计算执行状态。
作为示例，批量特征计算的计算结果用于模型训练或批量预估，实时特征计算的计算结果用于基于训练的模型的预测，所述预测包括在线预测。
作为示例，所述方法还包括：当所述脚本通过管理服务器被运行时，将所述脚本转化为批量特征计算单元、分布式计算单元和/或实时特征计算单元能够执行且当被执行时实现所述脚本所包含的特征计算逻辑的代码。
在本公开中，上述内容还可以表述为：所述脚本通过管理服务器运行所 述脚本，将所述脚本转化为批量特征计算单元、分布式计算单元和实时特征计算单元之中的至少一个能够执行且当被执行时实现所述脚本所包含的特征计算逻辑的代码。
作为示例，当所述脚本被运行时，由批量特征计算单元、分布式计算单元和/或实时特征计算单元解析出所述脚本中所包含的特征计算逻辑。
在本公开中，上述内容还可以表述为：当所述脚本被运行时，批量特征计算单元、分布式计算单元和实时特征计算单元之中的至少一个解析出所述脚本中所包含的特征计算逻辑。
应该理解，根据本公开示例性实施例的用于批量和实时特征计算的方法的具体实现方式可参照结合图1至图3描述的相关具体实现方式来实现，在此不再赘述。
根据本公开示例性实施例的用于批量和实时特征计算的系统所包括的单元或服务器可被分别配置为执行特定功能的软件、硬件、固件或上述项的任意组合。例如，这些单元或服务器可对应于专用的集成电路，也可对应于纯粹的软件代码，还可对应于软件与硬件相结合的模块。此外，这些单元或服务器所实现的一个或多个功能也可由物理实体设备(例如，处理器、客户端或服务器等)中的组件来统一执行。
应理解，根据本公开示例性实施例的用于批量和实时特征计算的方法可通过记录在计算可读介质上的程序来实现，例如，根据本公开的示例性实施例，可提供一种用于批量和实时特征计算的计算机可读介质。作为示例，提供的计算机可读介质可以为多个计算机可读介质，多个计算机可读介质分别部署在管理服务器、离线数据库、实时数据库、批量特征计算单元、分布式计算单元以及实时特征计算单元中。
其中，在每个计算机可读介质上记录有用于执行以下方法对应步骤的计算机程序：由管理服务器接收包含特征计算逻辑的脚本；由离线数据库存储用于批量特征计算的数据；由实时数据库从离线数据库获取并存储用于实时特征计算的数据；在批量特征计算场景下，由管理服务器从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，当批量特征计算单元被选择以启动批量特征计算时，由批量特征计算单元产生批量计算任务，将所述批量计算任务发送给一个或更多个分布式计算单元，并且由所述一个或更多个分布式计算单元执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；在实时特征计算场景下，由管理服务器向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并且从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，当实时特征计算单元被选择以执行实时计算计划时，由被选择的实时特征计算单元基于实时数据库存储的数据和所述脚本进行实时特征计算；其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性。
上述计算机可读介质中的计算机程序可在诸如客户端、主机、代理装置、 服务器等计算机设备中部署的环境中运行，应注意，所述计算机程序还可用于执行除了上述步骤以外的附加步骤或者在执行上述步骤时执行更为具体的处理，这些附加步骤和进一步处理的内容已经参照图1至图3进行了描述，这里为了避免重复将不再进行赘述。
应注意，根据本公开示例性实施例的用于批量和实时特征计算的系统可完全依赖计算机程序的运行来实现相应的功能，即，各个单元或服务器与计算机程序的功能架构相应，使得整个系统通过专门的软件包(例如，lib库)而被调用，以实现相应的功能。
另一方面，根据本公开示例性实施例的用于批量和实时特征计算的系统所包括的各个单元或服务器也可以通过硬件、软件、固件、中间件、微代码或其任意组合来实现。当以软件、固件、中间件或微代码实现时，用于执行相应操作的程序代码或者代码段可以存储在诸如存储介质的计算机可读介质中，使得处理器可通过读取并运行相应的程序代码或者代码段来执行相应的操作。
例如，本公开的示例性实施例还可以实现为计算装置，该计算装置包括存储部件和处理器，存储部件中存储有计算机可执行指令集合，当所述计算机可执行指令集合被所述处理器执行时，执行用于批量和实时特征计算的方法。
在此需要说明的是，该计算装置与上述的管理服务器中的计算装置不同。具体来说，该计算装置可以执行上述的管理服务器、离线数据库、实时数据库、批量特征计算单元、分布式计算单元以及实时特征计算单元中的任意一个所执行的步骤，而上述管理服务器中的计算装置仅执行管理服务器所执行的步骤。
具体说来，所述计算装置可以部署在服务器或客户端中，也可以部署在分布式网络环境中的节点装置上。此外，所述计算装置可以是PC计算机、平板装置、个人数字助理、智能手机、web应用或其他能够执行上述指令集合的装置。
这里，所述计算装置并非必须是单个的计算装置，还可以是任何能够单独或联合执行上述指令(或指令集)的装置或电路的集合体。计算装置还可以是集成控制系统或系统管理器的一部分，或者可被配置为与本地或远程(例如，经由无线传输)以接口互联的便携式电子装置。
在所述计算装置中，处理器可包括中央处理器(CPU)、图形处理器(GPU)、可编程逻辑装置、专用处理器系统、微控制器或微处理器。作为示例而非限制，处理器还可包括模拟处理器、数字处理器、微处理器、多核处理器、处理器阵列、网络处理器等。
根据本公开示例性实施例的用于批量和实时特征计算的方法中所描述的某些操作可通过软件方式来实现，某些操作可通过硬件方式来实现，此外，还可通过软硬件结合的方式来实现这些操作。
处理器可运行存储在存储部件之一中的指令或代码，其中，所述存储部 件还可以存储数据。指令和数据还可经由网络接口装置而通过网络被发送和接收，其中，所述网络接口装置可采用任何已知的传输协议。
存储部件可与处理器集成为一体，例如，将RAM或闪存布置在集成电路微处理器等之内。此外，存储部件可包括独立的装置，诸如，外部盘驱动、存储阵列或任何数据库系统可使用的其他存储装置。存储部件和处理器可在操作上进行耦合，或者可例如通过I/O端口、网络连接等互相通信，使得处理器能够读取存储在存储部件中的文件。
此外，所述计算装置还可包括视频显示器(诸如，液晶显示器)和用户交互接口(诸如，键盘、鼠标、触摸输入装置等)。计算装置的所有组件可经由总线和/或网络而彼此连接。
根据本公开示例性实施例的用于批量和实时特征计算的方法所涉及的操作可被描述为各种互联或耦合的功能块或功能示图。然而，这些功能块或功能示图可被均等地集成为单个的逻辑装置或按照非确切的边界进行操作。
例如，如上所述，根据本公开示例性实施例的用于批量和实时特征计算的计算装置可包括存储部件和处理器，其中，存储部件中存储有计算机可执行指令集合，当所述计算机可执行指令集合被所述处理器执行时，执行下述步骤：由管理服务器接收包含特征计算逻辑的脚本；由离线数据库存储用于批量特征计算的数据；由实时数据库从离线数据库获取并存储用于实时特征计算的数据；在批量特征计算场景下，由管理服务器从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，当批量特征计算单元被选择以启动批量特征计算时，由批量特征计算单元产生批量计算任务，将所述批量计算任务发送给一个或更多个分布式计算单元，并且由所述一个或更多个分布式计算单元执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；在实时特征计算场景下，由管理服务器向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并且从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，当实时特征计算单元被选择以执行实时计算计划时，由被选择的实时特征计算单元基于实时数据库存储的数据和所述脚本进行实时特征计算；其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性。
以上描述了本公开的各示例性实施例，应理解，上述描述仅是示例性的，并非穷尽性的，本公开不限于所披露的各示例性实施例。在不偏离本公开的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。因此，本公开的保护范围应该以权利要求的范围为准。
工业实用性
本公开提供的用于批量和实时特征计算的系统和方法可基于统一的脚本和关联的两种数据库同时支持批量特征计算和实时特征计算，可解决机器学习场景下批量特征计算逻辑和实时特征计算逻辑不统一以及批量特征处理的离线数据库和实时特征处理的实时数据库分离的问题。

Claims (30)

  1. 一种用于批量和实时特征计算的系统，其中，所述系统包括：
    管理服务器，接收包含特征计算逻辑的脚本，在批量特征计算场景下，从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，并且在实时特征计算场景下，向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性；
    一个或更多个批量特征计算单元，当被启动时注册到管理服务器，并且当被选择以启动批量特征计算时产生批量计算任务，并且将所述批量计算任务发送给一个或更多个分布式计算单元；
    所述一个或更多个分布式计算单元，执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；
    一个或更多个实时特征计算单元，当被启动时注册到管理服务器，并且当被选择以执行实时计算计划时，基于实时数据库存储的数据和所述脚本进行实时特征计算；
    离线数据库，存储用于批量特征计算的数据；
    实时数据库，从离线数据库获取并存储用于实时特征计算的数据。
  2. 根据权利要求1所述的系统，其中，所述离线数据库中的数据和所述实时数据库中的数据至少部分同步，
    其中，所述实时数据库与所述离线数据库定时或响应于用户触发而基于数据变化或者基于主从同步机制进行数据同步。
  3. 根据权利要求2所述的系统，包括以下至少一项：
    所述实时数据库是内存数据库；
    所述管理服务器还初始化所述实时数据库，以获取与实时特征计算对应的数据字段结构；
    所述管理服务器还接收用户导入的原始数据并且将原始数据存储到离线数据库中作为用于批量特征计算的数据，或者所述管理服务器还将预先存入所述离线数据库的至少一部分数据作为用于批量特征计算的数据。
  4. 根据权利要求1所述的系统，包括以下至少一项：
    所述管理服务器还验证所述脚本是否存在语法错误；
    所述管理服务器基于至少一部分数据来调试与所述脚本对应的至少一部分中间运行结果和最终运行结果之中的至少一个。
  5. 根据权利要求3所述的系统，其中，所述管理服务器还在接收到用于实时特征计算的至少一条数据并且实时计算计划被执行时，将所述至少一条数据发送给被选择的实时特征计算单元，其中，所述至少一条数据符合与实时特征计算对应的数据字段结构，
    其中，所述实时特征计算单元在接收到所述至少一条数据时向实时数据库请求窗口数据，并且基于窗口数据以及所述脚本进行实时特征计算，
    其中，所述窗口数据包含所述实时数据库的数据的全部或部分数据字段。
  6. 根据权利要求1所述的系统，其中，所述一个或更多个分布式计算单元基于所述离线数据库的数据存储路径来获取所述离线数据库存储的数据，并且基于获取的数据、所述脚本来执行批量特征计算，并且将批量特征计算执行状态发送给批量特征计算单元，
    其中，所述管理服务器在接收到批量特征计算执行状态查询请求时，从所述批量特征计算单元获取批量特征计算执行状态。
  7. 根据权利要求1所述的系统，其中，批量特征计算的计算结果用于模型训练或批量预估，实时特征计算的计算结果用于基于训练的模型的预测，所述预测包括在线预测。
  8. 根据权利要求1所述的系统，其中，当所述脚本通过所述管理服务器被运行时，所述脚本被转化为所述批量特征计算单元、所述分布式计算单元和所述实时特征计算单元之中的至少一个能够执行且当被执行时实现所述脚本所包含的特征计算逻辑的代码。
  9. 根据权利要求1所述的系统，其中，当所述脚本被运行时，所述批量特征计算单元、所述分布式计算单元和所述实时特征计算单元之中的至少一个解析出所述脚本中所包含的特征计算逻辑。
  10. 一种用于批量和实时特征计算的方法，其中，所述方法包括：
    接收包含特征计算逻辑的脚本；
    在批量特征计算场景下，从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，以使被选择的批量特征计算单元产生批量计算任务，将所述批量计算任务发送给一个或更多个分布式计算单元，以使所述一个或更多个分布式计算单元执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；
    在实时特征计算场景下，向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并且从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，以使被选择的实时特征计算单元基于实时数据库存储的数据和所述脚本进行实时特征计算；
    其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性，离线数据库存储的数据为用于批量特征计算的数据；实时数据库存储的数据为从离线数据库获取的用于实时特征计算的数据。
  11. 根据权利要求10所述的方法，其中，
    所述所述实时数据库与所述离线数据库定时或响应于用户触发而基于数据变化或者基于主从同步机制进行至少部分数据的数据同步。
  12. 根据权利要求11所述的方法，其中，所述方法还包括以下步骤中的至少一个步骤：
    初始化所述实时数据库，以获取与实时特征计算对应的数据字段结构；
    接收用户导入的原始数据并且将原始数据存储到所述离线数据库中作为用于批量特征计算的数据，或者将预先存入所述离线数据库的至少一部分数据作为用于批量特征计算的数据。
  13. 根据权利要求11所述的方法，其中，所述实时数据库是内存数据库。
  14. 根据权利要求10所述的方法，其中，所述方法还包括以下步骤中的至少一个步骤：
    验证所述脚本是否存在语法错误；
    基于至少一部分数据来调试与所述脚本对应的至少一部分中间运行结果；
    基于至少一部分数据来调试与所述脚本对应的至少一部分最终运行结果。
  15. 根据权利要求12所述的方法，其中，所述方法还包括：
    在接收到用于实时特征计算的至少一条数据并且实时计算计划被执行时，将所述至少一条数据发送给被选择的实时特征计算单元，以使被选择的实时特征计算单元在接收到所述至少一条数据时向实时数据库请求窗口数据，并且基于窗口数据以及所述脚本进行实时特征计算，
    其中，所述至少一条数据符合与实时特征计算对应的数据字段结构，所述窗口数据包含所述实时数据库的数据的全部或部分数据字段。
  16. 根据权利要求10所述的方法，其中，所述一个或更多个分布式计算单元基于所述离线数据库的数据存储路径来获取所述离线数据库存储的数据，并且基于获取的数据、所述脚本来执行批量特征计算，并且将批量特征计算执行状态发送给所述批量特征计算单元；
    其中，所述方法还包括：在接收到批量特征计算执行状态查询请求时，从所述批量特征计算单元获取批量特征计算执行状态。
  17. 根据权利要求10所述的方法，其中，批量特征计算的计算结果用于模型训练或批量预估，实时特征计算的计算结果用于基于训练的模型的预测，所述预测包括在线预测。
  18. 根据权利要求10所述的方法，其中，所述方法还包括：
    运行所述脚本，将所述脚本转化为所述批量特征计算单元、所述分布式计算单元和所述实时特征计算单元之中的至少一个能够执行、且当被执行时实现所述脚本所包含的特征计算逻辑的代码。
  19. 根据权利要求10所述的方法，其中，
    当所述脚本被运行时，所述批量特征计算单元、所述分布式计算单元和所述实时特征计算单元之中的至少一个解析出所述脚本中所包含的特征计算逻辑。
  20. 一种包括至少一个计算装置和至少一个存储指令的存储装置的管理服务器，其中，所述指令在被所述至少一个计算装置运行时，促使所述至少一个计算装置执行用于批量和实时特征计算的方法的以下步骤：
    接收包含特征计算逻辑的脚本；
    在批量特征计算场景下，从注册到管理服务器的批量特征计算单元中选择用于批量特征计算的批量特征计算单元以启动批量特征计算，以使被选择 的批量特征计算单元产生批量计算任务，将所述批量计算任务发送给一个或更多个分布式计算单元，以使所述一个或更多个分布式计算单元执行所述批量计算任务，以基于所述脚本和离线数据库存储的数据进行批量特征计算；
    在实时特征计算场景下，向注册到管理服务器的实时特征计算单元部署所述脚本以产生实时计算计划，并且从注册到管理服务器的实时特征计算单元中选择在执行实时计算计划时使用的实时特征计算单元，以使被选择的实时特征计算单元基于实时数据库存储的数据和所述脚本进行实时特征计算；
    其中，包含所述特征计算逻辑的脚本针对批量特征计算逻辑和实时特征计算逻辑具有统一性，离线数据库存储的数据为用于批量特征计算的数据；实时数据库存储的数据为从离线数据库获取的用于实时特征计算的数据。
  21. 根据权利要求20所述的管理服务器，其中，
    所述实时数据库与所述离线数据库定时或响应于用户触发而基于数据变化或者基于主从同步机制进行至少部分数据的数据同步。
  22. 根据权利要求21所述的管理服务器，其中，所述指令在被所述至少一个计算装置运行时，促使所述至少一个计算装置执行以下至少一个步骤中的至少一个步骤：
    初始化所述实时数据库，以获取与实时特征计算对应的数据字段结构；
    接收用户导入的原始数据并且将原始数据存储到所述离线数据库中作为用于批量特征计算的数据，或者将预先存入所述离线数据库的至少一部分数据作为用于批量特征计算的数据。
  23. 根据权利要求21所述的管理服务器，其中，所述实时数据库是内存数据库。
  24. 根据权利要求20所述的管理服务器，其中，所述指令在被所述至少一个计算装置运行时，促使所述至少一个计算装置执行以下步骤中的至少一个步骤：
    验证所述脚本是否存在语法错误；
    基于至少一部分数据来调试与所述脚本对应的至少一部分中间运行结果；
    基于至少一部分数据来调试与所述脚本对应的至少一部分最终运行结果。
  25. 根据权利要求22所述的管理服务器，其中，所述指令在被所述至少一个计算装置运行时，促使所述至少一个计算装置执行以下步骤：
    在接收到用于所述实时特征计算的至少一条数据并且实时计算计划被执行时，将所述至少一条数据发送给被选择的实时特征计算单元，以使被选择的实时特征计算单元在接收到所述至少一条数据时向实时数据库请求窗口数据，并且基于窗口数据以及所述脚本进行实时特征计算，
    其中，所述至少一条数据符合与实时特征计算对应的数据字段结构，所述窗口数据包含所述实时数据库的数据的全部或部分数据字段。
  26. 根据权利要求20所述的管理服务器，其中，所述一个或更多个分布式计算单元基于所述离线数据库的数据存储路径来获取所述离线数据库存储的数据，并且基于获取的数据、所述脚本来执行批量特征计算，并且将批量 特征计算执行状态发送给所述批量特征计算单元，
    其中，所述指令在被所述至少一个计算装置运行时，促使所述至少一个计算装置执行以下步骤：在接收到批量特征计算执行状态查询请求时，从所述批量特征计算单元获取批量特征计算执行状态。
  27. 根据权利要求20所述的管理服务器，其中，批量特征计算的计算结果用于模型训练或批量预估，实时特征计算的计算结果用于基于训练的模型的预测，所述预测包括在线预测。
  28. 根据权利要求20所述的管理服务器，其中，所述指令在被所述至少一个计算装置运行时，促使所述至少一个计算装置执行以下步骤：
    运行所述脚本，将所述脚本转化为所述批量特征计算单元、所述分布式计算单元和所述实时特征计算单元之中的至少一个能够执行、且当被执行时实现所述脚本所包含的特征计算逻辑的代码。
  29. 根据权利要求20所述的管理服务器，其中，当所述脚本被运行时，所述批量特征计算单元、所述分布式计算单元和所述实时特征计算单元之中的至少一个解析出所述脚本中所包含的特征计算逻辑。
  30. 一种存储指令的计算机可读存储介质，其中，当所述指令被至少一个计算装置运行时，促使所述至少一个计算装置执行如权利要求10-19中任意一项权利要求所述的方法。
PCT/CN2020/111412 2019-08-26 2020-08-26 用于批量和实时特征计算的系统和方法 WO2021037066A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20857623.1A EP4024228A4 (en) 2019-08-26 2020-08-26 SYSTEM AND METHOD FOR CALCULATING BATCH AND REAL-TIME CHARACTERISTICS
US17/639,125 US11971882B2 (en) 2019-08-26 2020-08-26 System and method for batch and real-time feature computation

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910789592.2A CN110502579A (zh) 2019-08-26 2019-08-26 用于批量和实时特征计算的系统和方法
CN201910789592.2 2019-08-26

Publications (1)

Publication Number Publication Date
WO2021037066A1 true WO2021037066A1 (zh) 2021-03-04

Family

ID=68589459

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/111412 WO2021037066A1 (zh) 2019-08-26 2020-08-26 用于批量和实时特征计算的系统和方法

Country Status (4)

Country Link
US (1) US11971882B2 (zh)
EP (1) EP4024228A4 (zh)
CN (1) CN110502579A (zh)
WO (1) WO2021037066A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113360581A (zh) * 2021-06-04 2021-09-07 北京自如信息科技有限公司 数据处理方法、装置及存储介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12026104B2 (en) * 2019-03-26 2024-07-02 Rambus Inc. Multiple precision memory system
CN110502579A (zh) 2019-08-26 2019-11-26 第四范式（北京）技术有限公司 用于批量和实时特征计算的系统和方法
CN111651524B (zh) * 2020-06-05 2023-10-03 第四范式（北京）技术有限公司 利用机器学习模型进行线上预测的辅助实现方法及装置
CN117555944A (zh) * 2020-12-24 2024-02-13 第四范式（北京）技术有限公司 特征计算的批请求处理方法、装置、电子设备及存储介质
CN112597121A (zh) * 2020-12-25 2021-04-02 北京知因智慧科技有限公司 逻辑脚本的处理方法、装置、电子设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8156119B2 (en) * 2009-01-19 2012-04-10 Microsoft Corporation Smart attribute classification (SAC) for online reviews
US20150278706A1 (en) * 2014-03-26 2015-10-01 Telefonaktiebolaget L M Ericsson (Publ) Method, Predictive Analytics System, and Computer Program Product for Performing Online and Offline Learning
CN105488662A (zh) * 2016-01-07 2016-04-13 北京歌利沃夫企业管理有限公司 一种基于双向推荐的在线招聘系统
CN109597842A (zh) * 2018-12-14 2019-04-09 深圳前海微众银行股份有限公司 数据实时计算方法、装置、设备及计算机可读存储介质
CN109918048A (zh) * 2018-12-27 2019-06-21 北京奇艺世纪科技有限公司 目标对象提取方法、装置、系统及计算机可读存储介质
CN110502579A (zh) * 2019-08-26 2019-11-26 第四范式（北京）技术有限公司 用于批量和实时特征计算的系统和方法

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3876107A1 (en) * 2013-03-15 2021-09-08 Factual Inc. Apparatus, systems, and methods for analyzing movements of target entities
US20140297744A1 (en) * 2013-04-02 2014-10-02 Microsoft Corporation Real-time supplement of segmented data for user targeting
US20170076323A1 (en) * 2015-09-11 2017-03-16 Adobe Systems Incorporated Matching devices with entities using real-time data and batch-processed data
US10387798B2 (en) 2015-12-16 2019-08-20 Accenture Global Solutions Limited Machine for development of analytical models
US10733531B2 (en) 2016-01-27 2020-08-04 Bonsai AI, Inc. Artificial intelligence engine having an architect module
WO2017156624A1 (en) * 2016-03-14 2017-09-21 Rubikloud Technologies Inc. Method and system for persisting data
CN106469202A (zh) * 2016-08-31 2017-03-01 杭州探索文化传媒有限公司 一种影视大数据平台的数据分析方法
US10078337B1 (en) * 2017-07-14 2018-09-18 Uber Technologies, Inc. Generation of trip estimates using real-time data and historical data
CN108008942B (zh) * 2017-11-16 2020-04-07 第四范式（北京）技术有限公司 对数据记录进行处理的方法及系统
CN108228861B (zh) * 2018-01-12 2020-09-01 第四范式（北京）技术有限公司 用于执行机器学习的特征工程的方法及系统
CN108519914B (zh) * 2018-04-09 2021-10-26 腾讯科技（深圳）有限公司 大数据计算方法、系统和计算机设备
CN111949349B (zh) * 2018-08-21 2024-09-20 第四范式（北京）技术有限公司 统一地执行特征抽取的方法及系统
CN109656914A (zh) * 2018-11-07 2019-04-19 上海前隆信息科技有限公司 线上线下混合的风控建模训练和生产发布方法及系统
CN109767255B (zh) * 2018-12-06 2023-05-12 东莞团贷网互联网科技服务有限公司 一种通过大数据建模实现智能运营及精准营销的方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8156119B2 (en) * 2009-01-19 2012-04-10 Microsoft Corporation Smart attribute classification (SAC) for online reviews
US20150278706A1 (en) * 2014-03-26 2015-10-01 Telefonaktiebolaget L M Ericsson (Publ) Method, Predictive Analytics System, and Computer Program Product for Performing Online and Offline Learning
CN105488662A (zh) * 2016-01-07 2016-04-13 北京歌利沃夫企业管理有限公司 一种基于双向推荐的在线招聘系统
CN109597842A (zh) * 2018-12-14 2019-04-09 深圳前海微众银行股份有限公司 数据实时计算方法、装置、设备及计算机可读存储介质
CN109918048A (zh) * 2018-12-27 2019-06-21 北京奇艺世纪科技有限公司 目标对象提取方法、装置、系统及计算机可读存储介质
CN110502579A (zh) * 2019-08-26 2019-11-26 第四范式（北京）技术有限公司 用于批量和实时特征计算的系统和方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4024228A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113360581A (zh) * 2021-06-04 2021-09-07 北京自如信息科技有限公司 数据处理方法、装置及存储介质

Also Published As

Publication number Publication date
CN110502579A (zh) 2019-11-26
US20220292083A1 (en) 2022-09-15
EP4024228A4 (en) 2022-11-02
EP4024228A1 (en) 2022-07-06
US11971882B2 (en) 2024-04-30

Similar Documents

Publication Publication Date Title
WO2021037066A1 (zh) 用于批量和实时特征计算的系统和方法
KR102493449B1 (ko) 엣지 컴퓨팅 테스트 방법, 장치, 전자 장치 및 컴퓨터 판독 가능 매체
US10901961B2 (en) Systems and methods for generating schemas that represent multiple data sources
US10387798B2 (en) Machine for development of analytical models
KR101621137B1 (ko) 아파치 하둡을 위한 로우 레이턴시 쿼리 엔진
US8321450B2 (en) Standardized database connectivity support for an event processing server in an embedded context
US8387076B2 (en) Standardized database connectivity support for an event processing server
JP5298117B2 (ja) 分散コンピューティングにおけるデータマージング
US9659012B2 (en) Debugging framework for distributed ETL process with multi-language support
US20160292591A1 (en) Streamlined analytic model training and scoring system
US11928130B2 (en) Automated performing of replication tasks in a multiple database system
US10949218B2 (en) Generating an execution script for configuration of a system
AU2017327824B2 (en) Data integration job conversion
US20190361999A1 (en) Data analysis over the combination of relational and big data
US12001273B2 (en) Software validation framework
US20190213007A1 (en) Method and device for executing the distributed computation task
US11853196B1 (en) Artificial intelligence driven testing
CN111523676B (zh) 辅助机器学习模型上线的方法及装置
Wang et al. Flint: A platform for federated learning integration
WO2020038376A1 (zh) 统一地执行特征抽取的方法及系统
US20220107817A1 (en) Dynamic System Parameter for Robotics Automation
US20170161359A1 (en) Pattern-driven data generator
US12009991B1 (en) Artificial aging of digital twin representing infrastructure
US20240330136A1 (en) Artificial aging of digital twin to debug infrastructure
Маньюнь et al. Classification of Meteorological Satellite Ground System Applications

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20857623

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020857623

Country of ref document: EP

Effective date: 20220328