CN115803715A

CN115803715A - Intelligent process routing in a partitioned database management system

Info

Publication number: CN115803715A
Application number: CN202080102917.6A
Authority: CN
Inventors: 傅忱; 蔡乐; 李飞飞
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2020-08-28
Filing date: 2020-08-28
Publication date: 2023-03-14
Also published as: WO2022041143A1

Abstract

A database system may include at least one load balancing node and a plurality of compute nodes. The load balancing node may receive a distributed transaction that includes a stored procedure and select a prediction function for the stored procedure from a plurality of prediction functions. The load balancing node may extract one or more input parameters from a stored process and determine a compute node from the plurality of compute nodes based at least in part on the one or more input parameters using a prediction function. The load balancing node may then forward the stored procedure to the compute node for processing of the stored procedure.

Description

Intelligent process routing in a partitioned database management system

Background

With the ever increasing amount of data stored in various industries, such as corporate data, shopping data, personal data, etc., partitioned distributed database management systems (PDBMS) running on a cloud or server network are becoming increasingly popular. In a PDBM, the data table is divided horizontally into multiple parts, commonly referred to as data slices. Each data slice is a collection of rows and is independently hosted and replicated. Slices may be moved, split, or merged to improve performance and resiliency.

Transactions involving data hosted on a single node (i.e., a database server) are referred to as local transactions. These transactions are essentially indistinguishable from transactions in a conventional single database management system. On the other hand, a transaction involving data on multiple nodes is a global transaction, and a complex process called two-phase commit (2 PC) is required to be performed when a commit operation is performed. Thus, global transactions are much slower than local transactions.

For better performance, a database administrator (DBA) will typically specify data partitioning and placement policies so that most transactions can be performed locally. For example, a database of an e-commerce application may include a warehouse table and a client table, among others. If most of the orders submitted to the e-commerce application can be completed by the local warehouse, the database administrator may choose to partition the table according to geographic location so that rows representing the co-located warehouse and customer are in the same slice of their respective tables. The database administrator may also specify that data slices of the co-located client table and data slices of the bin table are hosted on the same node. In this way, most orders can be serviced by performing local transactions, resulting in better performance.

Such database partitioning work may achieve optimal performance if the PDBMS can compute the data, i.e., perform transactions directly on the nodes hosting or storing the data. However, it has proven difficult to do so for the following reasons: 1) Transactions are preferably executed by a single node, and it is feasible, but often costly, to abort the transaction and restart the transaction on another node; 2) It is difficult to predict the node where the relevant data is located before the transaction begins, especially when most transactions involve multiple queries. Thus, the node performing the transaction does not typically host the data related to the transaction, but rather needs to forward the query request to the actual hosting node, thus incurring high communication costs. At worst, transferring query results and instructions between the control node and other nodes further incurs communication costs and time, as the transaction has been assigned to a node that needs to act as a control node to coordinate the execution or processing of multiple queries contained in the transaction by other nodes, send commit instructions to these other nodes to complete the 2-phase commit, and collect query results from the other nodes that are relevant to the execution or processing of the multiple queries, and send the query results to the client device of the database administrator for presentation or viewing.

Disclosure of Invention

This summary section introduces a simplified concept of intelligent process routing in a partitioned database management system, which is further described in the detailed description below. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

This disclosure describes example implementations of intelligent process routing in a partitioned database management system. In implementations, one or more computing devices may perform static program analysis on a stored procedure to determine one or more input parameters. The one or more computing devices may also obtain stored data that includes a plurality of value sets for one or more input parameters of the stored procedure and an identification of the plurality of data shards. In an implementation, the one or more computing devices may train a classification model using a plurality of value sets as inputs and corresponding identifications of a plurality of data slices as outputs, and set the trained classification model as a prediction function for mapping a set of new values of one or more input parameter settings of a stored procedure to corresponding data slices.

Drawings

The detailed description is set forth with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference symbols in different drawings indicates similar or identical items.

FIG. 1 illustrates an example environment in which a database system can be used.

FIG. 2 illustrates an example compute node in more detail.

Fig. 3 illustrates an example load balancing node in more detail.

Fig. 4 shows an example monitoring node in more detail.

FIG. 5 illustrates an example method of determining a prediction function.

FIG. 6 illustrates an example method of processing a distributed database transaction.

Detailed Description

SUMMARY

As described above, existing partitioned distributed database systems employ a load balancing policy, and cannot deterministically or strategically assign a distributed database transaction to a particular node that includes one or more data tables involved in the database transaction, and the node to which the database transaction is assigned may need to act as a control or coordinating node, coordinating other nodes to process the distributed database transaction and synchronously commit the distributed database transaction, including the data tables involved in the distributed database transaction. This not only increases the communication cost and time of data and instructions transmitted between nodes, but also wastes processing resources of the control or coordinating nodes that may not be needed for distributed database transactions.

This disclosure describes example database systems. In an implementation, a database system may include a load balancing node, a plurality of compute nodes, and a monitoring node. In implementations, a database system can be associated with a plurality of processes that are pre-registered and stored in a distributed database associated with the database system. Each stored procedure may include one or more database queries that access data shards of one or more database tables located or distributed in one or more compute nodes of the database system. In implementations, the stored procedure can include one or more input parameters that can act as placeholders to receive input values from a user for executing a database transaction. In an implementation, the database system can assign a unique identification to each stored process of the plurality of stored processes.

In implementations, the database system may include or store one or more prediction functions. In an implementation, a prediction function of the one or more prediction functions may be used for one or more stored procedures of the plurality of stored procedures. In an implementation, each prediction function may be tagged with a respective identification, and a database system or load balancing node may include a first data structure (e.g., a table or map) that associates or stores each of one or more prediction functions with a respective identification. In addition, the database system may also include a second data structure (e.g., a table or map) that associates or stores the identification of the one or more stored procedures with the identification of the corresponding prediction functions.

In an implementation, a prediction function may be configured to predict or attempt to map a stored procedure to an identification of a data slice, the stored procedure being of a stored procedure for which an input parameter has a particular value, the data slice may include data corresponding to at least one query of the stored procedure. Alternatively, the prediction function may be configured to predict or attempt to map a stored procedure to an identification of a compute node, the stored procedure being a stored procedure having a particular value of an input parameter, the compute node may include data corresponding to at least one query of the stored procedure.

In implementations, a database system may receive a stored procedure from a client device, the stored procedure having input values for input parameters of the stored procedure. The database system may select or determine a prediction function for the stored procedure from the one or more prediction functions based on a second data structure that associates a plurality of stored procedures with the one or more prediction functions. In an implementation, the database system may determine or extract input values for one or more input parameters of a stored procedure through static program analysis, and determine a compute node to which the stored procedure is forwarded or assigned using the selected or determined prediction function (and the mapping function if the prediction function outputs an identification of a data slice, rather than an identification of a compute node determined from the extracted input values).

In implementations, the one or more prediction functions can include one or more classification models obtained by the database system through training based on historical data collected by the plurality of compute nodes. In implementations, the historical data of the prediction function may include, but is not limited to, a plurality of value sets of respective one or more input parameters of one or more stored procedures associated with the prediction function, an identification of a data slice or compute node, and the like. In implementations, the one or more classification models can include, but are not limited to, a neural network model, a deep learning model, a decision tree model, and the like. In implementations, the database system may update or retrain one or more prediction functions periodically or when the prediction error reaches a predetermined error threshold.

As described above, an exemplary database system may receive a stored procedure having input values for input parameters and predict or determine (determine to which the stored procedure is forwarded or allocated) a computing node, based on a prediction function, that may store data fragments for a data table involved in at least one query for the stored procedure, thereby potentially avoiding sending the stored procedure to a computing node that does not store any data fragments for the process query or access, thereby reducing resource waste, communication costs, and processing delays caused by additional demand resulting from interactions between the computing node and other computing nodes storing data fragments for the stored procedure query or access.

In implementations, the functions described herein as being performed by a database system may be performed by a plurality of separate units or services. Further, although in the examples described herein, the database system may be implemented as a combination of software and hardware implemented and distributed across multiple devices, in other examples, the database system may be implemented and distributed as a service provided over one or more computing devices on a network and/or cloud computing architecture.

This application describes many different embodiments and implementations. The following sections describe example frameworks suitable for practicing various implementations. Next, this application describes example systems, devices, and processes for implementing a database system.

Example Environment

FIG. 1 illustrates an example environment 100 that can be used to implement a database system. The environment 100 may include a database system 102. In an implementation, database system 102 may include a plurality of servers or computing nodes 104-1, 104-2, \8230; \8230, 104-N (collectively referred to as computing nodes 104). The computing nodes 104 may communicate data with each other over the network 106. In an implementation, database system 102 may also include at least one load balancing node 108 for distributing workload to servers or compute nodes 104. In an implementation, one or more servers or computing nodes 104 may be designated or used as at least one load balancing node 108. In an implementation, database system 102 may also include a monitoring node 110.

In implementations, each server or computing node 104 may be implemented as any of a variety of computing devices, not limited to a desktop computer, a notebook or portable computer, a handheld device, a netbook, an internet appliance, a tablet or tablet computer, a mobile device (e.g., a mobile phone, a personal digital assistant, a smartphone, etc.), a server computer, etc., or a combination thereof.

The network 106 may be a wireless network or a wired network, or a combination thereof. The network 106 may be a collection of independent networks interconnected with one another to serve as a single large network (e.g., the internet or an intranet). Examples of such independent networks include, but are not limited to, telephone networks, cable networks, local Area Networks (LANs), wide Area Networks (WANs), and Metropolitan Area Networks (MANs). Further, the independent network may be a wireless network or a wired network, or a combination thereof. A wired network may include electrical carrier connections (e.g., communication cables, etc.) and/or optical carriers or connections (e.g., fiber optic connections, etc.). The wireless network may include, for example, a WiFi network, other radio frequency networks (e.g., bluetooth, zigbee, etc.), and so forth.

In an implementation, environment 100 may also include a client device 112. Client device 112 may be implemented as any of a variety of computing devices, not limited to a desktop computer, a notebook or portable computer, a handheld device, a netbook, an internet appliance, a tablet or tablet computer, a mobile device (e.g., a mobile phone, a personal digital assistant, a smartphone, etc.), a server computer, etc., or a combination thereof.

In an implementation, database system 102 may receive a request from client device 112 to process a distributed database transaction. For example, a user 114 of a client device 112 (e.g., a database administrator, etc.) may submit multiple query requests directed to one or more data tables to database system 102 as a distributed database transaction. In response to receiving the request, database system 102 may determine, based on the routing instructions, at least one computing node that includes the data portion of the data table involved in the distributed database transaction, send the distributed database transaction to the at least one computing node for processing, and return a result of processing of the distributed database transaction to client device 112.

Example partition database

In an implementation, database system 102 may also include one or more databases partitioned or distributed among multiple compute nodes 104. By way of example and not limitation, a partitioned or distributed database may include one or more partitioned or partitioned data tables, each of which is horizontally partitioned into portions, referred to as data shards or simply shards. In implementations, each shard can include a set of rows and can be hosted and replicated independently in one or more computing nodes 104. Further, the shards may be moved, split, or merged to improve performance and resiliency of the database system 102.

In an implementation, a transaction involving data hosted on a single computing node 104 or server (e.g., a database server) may be referred to as a local transaction. Local transactions are essentially indistinguishable from transactions in a traditional monolithic DBMS. On the other hand, transactions involving data on multiple compute nodes, i.e., global transactions, require a complex process called 2-phase commit (2 PC) at commit time. Thus, global transactions are much slower to process than local transactions.

In implementations, the data table may be horizontally partitioned based on the value of a column or a combination of multiple columns. In the latter case, the values from each column may form a tuple, and for partitioning purposes, the columns may be considered as a single logical column. Without loss of generality, for simplicity, it is assumed that the table is partitioned based on partition lists.

In implementations, rows in a partition column having the same value may be placed in the same data segment or slice. For example, an e-commerce application may include a database having a table of clients, including columns such as streets, cities, states, zip codes, and the like. These columns may form the address of the client. A user 114 (e.g., a database administrator) may partition the client table using columns associated with zip codes alone such that all rows in the client table having the same zip code are located in the same data segment or slice.

In implementations, the database system 102 may also include or provide a partition function, part, for each data table _table (k) → plate _ id, where k is called partition key, is the value of the partition column. In an implementation, the output of this function may be the identification (i.e., ID) of the tile including all rows whose partition columns have values equal to the partition key (i.e., k).

In implementations, database system 102 may allow user 114 to specify one or more partitions and placement policies for data stored in a partitioned database. Continuing with the example of the e-commerce application described above, a repository table may be included in addition to the client table. Since most orders for the e-commerce application may need to be completed from a local repository, the user 114 may decide to partition the database based on geographic location. In this case, the zip code for the warehouse location and the zip code for the customer address may be treated as corresponding partition columns for the warehouse table and the customer table, respectively. Further, the user 114 may instruct the database system 102 to allow rows representing warehouses and customers at the same zip code to be hosted or stored in the same node so that most order processing may trigger local transactions. In other words, part _warehouse (k) And part _customer (k) Both functions have zip codes as zone keys and satisfy

Where place () is the placement function.

In implementations, the partition function and the placement function may be implemented by querying metadata, which includes a mapping from partition keys to shard IDs, and a mapping from shard IDs to compute nodes when the corresponding shards are hosted or stored. The metadata may be replicated to some or all of the computing nodes 104 and/or load balancing nodes 108 in the database system 102.

In some implementations, partition functions may not exist or may be difficult to construct due to the potentially large amount of data involved and the need for pre-storage. In implementations, database system 102 can include or provide a prediction function predict for a stored procedure _proc (parameters) → plate _ id, in which the parameters represent values entered by the user for the input parameters of the stored procedure. In an implementation, the output of this function may be a shard-ID, i.e. may include an identification (i.e. ID) of the slice to which the data line queried or accessed by the at least one query request of the stored procedure belongs. After obtaining the identification of the shard that may include the data row queried or accessed by at least one query request for the stored procedure, database system 102 may use a placement function as described above to obtain an identification of the compute node to which the stored procedure may be assigned or transmitted.

Example computing node

Fig. 2 shows the compute node 104 in detail. In an implementation, the computing node 104 may include, but is not limited to, one or more processors 202, input/output (I/O) interfaces 204, and/or network interfaces 206, and memory 208. In an implementation, some of the functionality of the compute node 104 may be implemented using hardware, such as an ASIC (i.e., application specific integrated circuit), FPGA (i.e., field programmable gate array), and/or other hardware.

In an implementation, the processor 202 may be configured to execute instructions stored in the memory 208 and/or received from the I/O interface 204 and/or the network interface 206. In an implementation, processor 202 may be implemented as one or more hardware processors including, for example, a microprocessor, a special purpose instruction set processor, a Physical Processing Unit (PPU), a Central Processing Unit (CPU), a graphics processing unit, a digital signal processor, a tensor processor. Additionally or alternatively, the functions described herein may be performed, at least in part, by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a Chip Systems (SOCs), complex Programmable Logic Devices (CPLDs), and the like.

Memory 208 may include a computer-readable medium that may be in the form of volatile memory, such as Random Access Memory (RAM), and/or non-volatile memory, such as Read Only Memory (ROM) or flash RAM. Memory 208 is an example of a computer-readable medium.

Computer-readable media may include volatile or nonvolatile types of removable or non-removable media, which may implement the storage of information using any method or technology. The information may include computer readable instructions, data structures, program modules or other data. Examples of a computer-readable medium include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other internal storage technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that may be used to store information that may be accessed by a computing device. As defined herein, a computer-readable medium does not include any transitory medium, such as modulated data signals and carrier waves.

Although in this example only hardware components in the computing node 104 are described, in other cases the computing node 104 may also include other hardware components and/or other software components, such as program elements, to execute instructions stored in the memory 208 to perform various operations. For example, the computing node 104 may also include a local or partitioned database 210 for storing data tables and other program data 212. By way of example and not limitation, a computing node 104 may store data segments or slices of one or more data tables in a local or partitioned database 210. In an implementation, one or more data tables may be partitioned and distributed according to the respective partition keys of different computing nodes 104.

Example load-balancing nodes

Fig. 3 shows the load balancing node 108 in more detail. In an implementation, load balancing node 108 may include, but is not limited to, one or more processors 302, input/output (I/O) interfaces 304, and/or network interfaces 306, and memory 308. In an implementation, some of the functions of the load balancing node 108 may be implemented using hardware, such as an ASIC (i.e., an application specific integrated circuit), an FPGA (i.e., a field programmable gate array), and/or other hardware.

In an implementation, the processor 302 may be configured to execute instructions stored in the memory 308 and/or received from the I/O interface 304 and/or the network interface 306. In an implementation, the processor 302 may be implemented as one or more hardware processors including, for example, a microprocessor, a special purpose instruction set processor, a Physical Processing Unit (PPU), a Central Processing Unit (CPU), a graphics processing unit, a digital signal processor, a tensor processing unit. Additionally or alternatively, the functions described herein may be performed, at least in part, by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a Chip Systems (SOCs), complex Programmable Logic Devices (CPLDs), and the like.

Memory 308 may include a computer-readable medium that may be in the form of volatile memory, such as Random Access Memory (RAM), and/or non-volatile memory, such as Read Only Memory (ROM) or flash RAM. Memory 308 is an example of a computer-readable medium as described previously.

Although in this example only hardware components in load balancing node 108 are described, in other cases, load balancing node 108 may also include other hardware components and/or other software components, such as program elements, for executing instructions stored in memory 308 to perform various operations. For example, load balancing node 108 may also include mapping table 310 and other program data 312. In an implementation, mapping table 310 may include a mapping from a combination of data table information and partition key information to information of a compute node that includes a data segment or slice of a data table corresponding to the partition key. By way of example and not limitation, given a name of a data table and a value of a partition key, load balancing node 108 may determine an identification or address of a compute node that includes a data segment or slice of the data table corresponding to the value of the partition key.

In an implementation, the load balancing node 108 may obtain the mapping table 310 in advance, for example, by receiving broadcast information from multiple computing nodes 104. In implementations, the broadcast information for each computing node 104 may include, but is not limited to, information about data segments or slices of the data table corresponding to particular values of partition keys included or stored in the respective computing node 104.

Additionally or alternatively, in some implementations, the load balancing node 108 may be associated with a mapping device (which may be a server or computing device provided in the database system 102). The mapping means may collect information from the plurality of computing nodes 104 regarding the data segments or slices of the data table included or stored in each computing node 104 corresponding to the particular value of the partition key, for example, by broadcasting the information from the plurality of computing nodes 104 as described above. In this case, the load balancing node 108 may send the information (e.g., values) of the partition keys and the information (e.g., names) of the data tables obtained from the distributed database transaction to a mapping device, which maps the information of the data tables and the information of the partition keys to information (e.g., identifications or addresses) of the compute nodes that include the data segments or slices of the data tables that correspond to the partition keys. The load balancing node 108 may then receive information of the compute nodes from the mapping device, thereby reducing the workload and complexity of the load balancing node while load balancing enabling the load balancing node to continue processing incoming requests or data packets.

In an implementation, the load balancing node 108 may include one or more predetermined load balancing policies. By way of example and not limitation, the one or more predetermined load balancing policies may include: the distributed database transactions received (e.g., from client device 112) are assigned to the computing nodes in a random manner, in a round-robin manner, with the least amount of current work, according to a mapping of the IP address of the client device to the computing nodes, or the like.

In an implementation, the load balancing node 108 may also include an identification database 314. The identification database 314 may include or store, for example, unique identifications (e.g., unique names, etc.) of stored procedures registered and stored in the database 102, unique identifications of prediction functions, and the like. In an implementation, the load balancing node 108 may also include a relational database 316. The relational database 316 may include or store, for example, relationships between unique identifications of stored procedures and unique identifications of prediction functions, and the like. In implementations, the load balancing node 108 may also include a predictor database 318. Predictor database 318 may include or store a plurality of prediction functions, which may be queried or called, for example, by identification of the prediction function.

Example monitoring node

Fig. 4 shows the monitoring node 110 in more detail. In an implementation, monitoring node 110 may include, but is not limited to, one or more processors 402, input/output (I/O) interfaces 404, and/or network interfaces 406, and memory 408. In an implementation, some of the functions of the monitoring node 110 may be implemented using hardware, such as an ASIC (i.e., an application specific integrated circuit), an FPGA (i.e., a field programmable gate array), and/or other hardware.

In an implementation, the processor 402 may be configured to execute instructions stored in the memory 408 and/or received from the I/O interface 404 and/or the network interface 306. In an implementation, processor 402 may be implemented as one or more hardware processors including, for example, a microprocessor, a special purpose instruction set processor, a Physical Processing Unit (PPU), a Central Processing Unit (CPU), a graphics processing unit, a digital signal processor, a tensor processing unit. Additionally or alternatively, the functions described herein may be performed, at least in part, by one or more hardware logic components. By way of example, and not limitation, types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), and the like.

Memory 408 may include a computer-readable medium, which may be in the form of volatile memory, such as Random Access Memory (RAM), and/or non-volatile memory, such as Read Only Memory (ROM) or flash RAM. Memory 408 is an example of computer-readable media as described previously.

Although in this example only hardware components in monitoring node 110 are described, in other cases monitoring node 110 may also include other hardware components and/or other software components, such as program elements, to execute instructions stored in memory 408 to perform various operations. For example, the monitoring node 110 may further include an information database 410 and other program data 412. In implementations, the information database 410 may store data received from the plurality of computing nodes 104 and the load balancing node 108, which may include, but is not limited to, the following information: the respective number of times each stored procedure from load balancing node 108 is individually invoked or used by a user, the identification of missed procedures and respective number of times missed from each compute node 104, the respective input and output data sets (including the plurality of value sets of one or more input parameters of the stored procedure and the identification of the plurality of data slices of each of the plurality of prediction functions), and so forth.

Example method

FIG. 5 shows a schematic diagram depicting an example method of determining a prediction function. FIG. 6 shows a schematic diagram depicting an example method of processing a distributed database transaction. The methods of fig. 5 and 6 may be, but need not be, implemented in the environment of fig. 1 and using the compute nodes, load balancing nodes, and monitoring nodes of fig. 2-4. For ease of explanation, method 500 and method 600 are described with reference to fig. 1-3. However, the

methods

500 and 600 may alternatively be implemented in other environments and/or using other systems.

The method 500 and the method 600 are described in the general context of computer-executable instructions. Generally, computer-executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, and the like that perform particular functions or implement particular abstract data types. Further, each of the example methods is illustrated as a collection of blocks in a logical flow graph, which represents a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof. The order in which the method is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method, or an alternative method. Moreover, individual blocks of the method may be omitted without departing from the spirit and scope of the subject matter described herein. In the context of software, blocks represent computer instructions, which, when executed by one or more processors, perform the recited operations. In the context of hardware, some or all of the blocks may represent Application Specific Integrated Circuits (ASICs) or other physical components that perform the operations.

Referring back to FIG. 5, at block 502, the monitoring node 110 may select a stored procedure.

In implementations, database system 102 may include multiple processes stored in database system 102. These processes are written in a query language (e.g., the SQL language) and registered in database system 102 for invocation by a user. In an implementation, a stored procedure may include one or more query requests to access or update data of one or more data partitions of one or more database tables located in one or more compute nodes of database system 102. In an implementation, a query request for a stored procedure may include a partition key and an input parameter that receives an input value from a user using the stored procedure. In an implementation, the stored procedure may include a parameter list of input parameters, which may be used as a placeholder to receive input values for the input parameters of the stored procedure from a user. In other implementations, the storage process may not include a parameter list of input parameters that are distributed among the storage processes that need to be searched.

In an implementation, monitoring node 110 may obtain one or more stored procedures that are newly added or registered in database system 102 and for which there is no prediction function or for which there has not been a prediction function trained. The monitoring node 110 may select a stored procedure from one or more stored procedures newly added or registered in the database system 102. In an implementation, the monitoring node 110 may select to newly add or register a stored procedure after the stored procedure has been used or invoked by a user of the database system 102 a predetermined number of times (e.g., 1,000 times, 10,000 times, etc.) and/or a predetermined period of time (e.g., one week, two weeks, etc.). This allows a sufficient amount of training data to be available to train the prediction function against the stored procedure.

Additionally or alternatively, the monitoring node 110 may select a stored procedure from a plurality of stored procedures associated with the prediction function to retrain periodically (e.g., every week, every two weeks, every month, etc.). In implementations, the frequency with which a stored procedure is selected can depend on the frequency with which a user invokes or uses the stored procedure in database system 102. By way of example and not limitation, the more frequently a stored procedure is invoked or used, the more frequently the stored procedure is selected for retraining.

Additionally or alternatively, the monitoring node 110 may monitor the performance of a plurality of stored processes in the database system 102 and determine whether to update or retrain the stored processes based on the performance of the stored processes. In implementations, the performance of the stored procedure may include, but is not limited to, an error rate of a prediction function used to map the stored procedure, and the like. For example, load balancing node 108 may count the number of times each stored procedure is used or invoked by a user within a preset time interval (e.g., every week, every two weeks, etc.), and store the identification of each stored procedure and the corresponding number of uses or invocations in a data structure (e.g., a table, etc.). Further, each computing node 104 may track one or more storage processes assigned to the computing node that encountered a miss in the respective computing node 104 (i.e., data that one or more storage processes requested access or update was not present in the computing node 104). Each computing node 104 may store an identification of such stored procedures and the corresponding number of misses for the stored procedures in a particular data structure (e.g., a list, etc.) for a preset time interval.

In an implementation, the monitoring node 110 may receive information from the load balancing node 108 storing a respective number of times a process was individually invoked or used by a user, and further collect from each computing node 104 an identification of missed processes and the respective number of misses. Monitoring node 110 may then determine a respective error rate for each stored procedure (e.g., determine a duty cycle for using or invoking the stored procedure that resulted in the miss). Monitoring node 110 may compare the error rate of each stored procedure to a predetermined error threshold and select the prediction function for the stored procedure with the highest error rate and an error rate above the predetermined error threshold to update or retrain.

At block 504, the monitoring node 110 may determine one or more input parameters for training a prediction function associated with the stored procedure.

In implementations, upon selecting a stored procedure, the monitoring node 110 may determine one or more input parameters that may be used to train a prediction function for the stored procedure. In an implementation, the stored procedure may include a parameter list including at least one input parameter, and the monitoring node 110 may use the at least one input parameter included in the parameter list as a candidate input parameter for training the prediction function for the stored procedure.

Additionally or alternatively, the monitoring node 110 may perform static program analysis to extract one or more candidate input parameters from a stored procedure. Static program analysis, which may also be referred to as compile-time analysis, is an algorithm that analyzes program code before the program code runs to predict the runtime behavior of the program code. The monitoring node 110 may employ static program analysis to determine which input parameters decide on or affect the data slice in which one or more query requests in the storage process will operate (e.g., access or update). In an implementation, monitoring node 110 may identify one or more variables that are used as one or more partition keys in the storage process. For example, for a SELECT query request during storage, the monitoring node 110 may find the distributed tables in the FROM clause and identify variables that may be used for comparison with the various partition columns of the distributed tables. Similar methods can also be used for UPDATE, INSERT, and DELETE query requests.

For example, a stored procedure, such as the paymentproc, can be expressed as follows. In this example, the first UPDATE is used to UPDATE the target table, i.e., the bin table, and compare the variable in _ w _ zip with the corresponding partition column, warehousjzip.

In an implementation, the monitoring node 110 may compute a transitive closure of all variables that may affect the value of a partition key in one or more query requests of a stored procedure. As described above, the monitoring node 110 may employ data flow analysis to estimate possible values for variables of data flow between procedural points reachable according to the control flow graph. To locate all variables whose values may affect the partition key of a query request during storage, monitor node 110 may trace back the values of the variables of the data flow starting at the partition key of the query request until all links passed through from the starting point are traversed. In an implementation, the monitoring node 110 may perform such operations using a graph traversal algorithm. In an implementation, the monitoring node 110 may set variables to candidate input parameters after retrieving those variables whose values may affect the partition keys of the query request. The monitoring node 110 may repeat the above operations for all partition keys of all query requests in the stored procedure to obtain a set of candidate input parameters that may affect the partition keys of all query requests of the stored procedure. In an implementation, the monitoring node 110 may set these candidate input parameters of the partition key that may affect the query request of the stored procedure as one or more input parameters for training the prediction function associated with the stored procedure.

At block 506, the monitoring node 110 may obtain historical data, including the identity of the plurality of data shards and the plurality of value sets of the one or more input parameters of the stored procedure.

In an implementation, the monitoring node 110 may obtain historical data from a plurality of computing nodes 104 and load balancing nodes 108. In an implementation, the historical data may include, but is not limited to, a plurality of value sets for one or more input parameters of the stored procedure and an identification of a plurality of data shards. In an implementation, each data slice of the plurality of data slices may include a corresponding data segment that is evaluated by the storage process when the one or more input parameters assume a respective one of a plurality of value sets.

At block 508, the monitoring node 110 may train the classification model using the multiple value sets as inputs and corresponding identifications of the multiple data patches as outputs.

In an implementation, the monitoring node 110 may train a classification model based on historical data, the lightning protection model to learn or mimic the behavior of the prediction function of the stored procedure. In an implementation, the monitoring node 110 may employ supervised training to train and obtain a classification model. In implementations, the classification model may include, but is not limited to, a neural network model, a deep learning model, a decision tree model, and the like.

By way of example and not limitation, in one example, a neural network model is used as a classification model to learn or model the behavior of a prediction function of a stored process. In this example, one or more input parameters of the stored procedure obtained or determined at block 504 may be used as inputs (or input features) to the neural network model, and the historical values (or sets of values) of the one or more parameters are training values for the input features. In implementations, each stored procedure can include one or more query requests and can perform operations (e.g., access, update, etc.) on one or more data shards of one or more database tables. In this case, the monitoring node 110 may select the most frequently operated data shard from among one or more data shards of one or more database tables. In implementations, the data shards that operate most frequently for a storage process may include, but are not limited to, the data shards that have the most number of lines accessed or affected by query requests in the storage process, and the like. In an implementation, the monitoring node 110 may treat the most frequently operated data slices as the corresponding outputs (or labels) of the neural network model. In an implementation, such feature-tag pairs may be obtained by monitoring the execution of stored procedures, recording individual input parameters and corresponding data slices accessed and corresponding line numbers accessed or affected, and so forth. The monitoring node 110 may then train the neural network model using conventional training or learning algorithms.

At block 510, the monitoring node 110 may set the trained classification model as a prediction function for mapping a set of new values for one or more input parameters of the stored procedure to corresponding data slices.

In an implementation, after training of the classification model is completed, the monitoring node 110 may use the trained classification model as a predictive function for the stored procedure. In an implementation, the monitoring node 110 may send a prediction function to the load balancing node 108, such that the load balancing node 108 may use the prediction function to map a stored process having any new set of input values set for one or more input parameters to a corresponding data slice (e.g., an identification of the data slice) that may store data that is to be accessed or updated by at least one query request of the stored process.

In an implementation, the monitoring node 110 may continue to receive new data associated with a new set of values for one or more input parameters for each stored process and an identification of data shards that respectively include corresponding data segments evaluated by each stored process when the one or more input parameters assume the new set of values that are periodically obtained from the plurality of computing nodes 104 and the load balancing node 108. The monitoring node 110 may select a stored procedure and retrain the prediction function for the stored procedure according to the operations as described in blocks 502-510 based on the new data and the previously stored data.

Although in the above operations, the prediction function is described as being trained for and associated with each stored procedure, in some cases, the prediction function may be trained for and associated with a set of stored procedures. In this case, the inputs (i.e., input parameters) of the prediction function may include a combination of candidate input parameters obtained from each stored procedure in the set of stored procedures as described in block 504, and the output of the predictor may include an identification of data slices that are likely to store or include data to be queried or updated by the respective stored procedure. In an implementation, the monitoring node 110 may automatically group stored procedures with similar candidate input parameters or group stored procedures according to a user's request.

Referring back to fig. 6, at block 602, the load balancing node 108 may receive a distributed database transaction that includes at least one query process.

In an implementation, the database system 102 or the load balancing node 108 of the database system 102 may receive a distributed database transaction including at least one query process from the client device 112. In an implementation, the at least one query process may include one or more query requests that access or manipulate data of one or more data shards of one or more database tables located in one or more computing nodes (e.g., computing node 104) of database system 102.

At block 604, the load balancing node 108 may determine whether the at least one query procedure is a pre-stored procedure.

In implementations, the query process can be a process that is pre-registered and stored in database system 102 (i.e., a stored process), or a process that is not registered in database system 102 but written by user 114 in some query language (e.g., the SQL language) according to his/her needs. If the query process is a registered stored process, the values of one or more input parameters for the stored process may be further provided by user 114 when the stored process is sent to database system 102. In an implementation, the one or more input parameters may include one or more potential partitioning key values associated with one or more query requests included in the stored procedure.

In implementations, a number of stored procedures may be registered and stored in database system 102 and may be invoked by user 114. In order to distinguish between different stored procedures, each stored procedure may be assigned a unique identification that may uniquely represent the corresponding stored procedure. The unique identification of the stored procedure may include, but is not limited to, a unique name of the stored procedure, a unique index of the stored procedure, and the like. In implementations, the stored procedure may include one or more input parameters or objects through which the user 114 may provide input values to the stored procedure. For example, in the stored procedure of the above example (proxy _ proc), the stored procedure may be associated with a unique name, namely proxy _ proc, that may uniquely identify the stored procedure and distinguish the stored procedure from other stored procedures in database system 102. Example last name, the stored procedure may also include a parameter list including a plurality of input parameters provided or input by the user 114 (i.e., in _ w _ identerger, in _ w _ zip integer, in _ c _ identerger, in _ c _ zip integer, in _ vector _ amountdecimal)

In an implementation, a determination is made by the load balancing node 10 as to whether the received query procedure is a registered stored procedure, the load balancing node 10 determining whether the received query procedure is a stored procedure by determining whether the query procedure is associated with an identification (e.g., name, etc.) and the identification is registered or found in a data structure (e.g., table, list, etc.) that stores the identification of stored procedures registered in the database system 102. Continuing with the above example, load balancing node 108 may determine whether the received query procedure is associated with a name (e.g., proxy _ proc) and, if so, further determine whether the name is registered or found in a data structure storing an identification of stored procedures registered in database system 102. If it is determined that the name is registered or found, load balancing node 108 may determine that the received query procedure is a registered stored procedure. Alternatively, load balancing node 108 may determine that the received query procedure is not a registered stored procedure if the received query procedure is not associated with any name, or the name with which the received query procedure is associated is not registered or found in a data structure storing the identification of stored procedures registered in database system 102.

At block 606, the load balancing node 108 may employ a predetermined load balancing policy to assign the distributed database transaction to the compute nodes in response to determining that the received query process is not a registered storage process.

In an implementation, if it is determined that the received query process is not a registered stored process, the load balancing node 108 may employ a predetermined load balancing policy to distribute distributed database transactions to computing nodes of the plurality of computing nodes 104 included in the database system 102. By way of example and not limitation, the predetermined load balancing policy may include: the distributed database transaction may be assigned to the computing node randomly, in a round robin fashion, with the least amount of current work, based on the IP address of the client device from which the distributed database transaction came, etc.

At block 608, in response to determining that the received query procedure is a registered storage procedure, the load balancing node 108 may determine or select a prediction function for the storage procedure from a plurality of prediction functions.

In implementations, after determining that the received query procedure is a registered storage procedure, the load balancing node 108 may determine or select a prediction function for the storage procedure. In implementations, the database system 102 may include or store multiple prediction functions, and each prediction function may be configured to perform a prediction or mapping on one or more stored procedures. In implementations, the load balancing node 108 may determine or select a prediction function for the stored procedure based at least in part on an identification (e.g., name or index) of the stored procedure.

In an implementation, the load balancing node 108 or database system 102 may include or store a data structure (e.g., a table, etc.) that includes mappings between a plurality of stored procedures and a plurality of prediction functions, such as mappings between identities of the plurality of stored procedures and identities of the plurality of prediction functions, or mappings between identities of the plurality of stored procedures and storage locations of the plurality of prediction functions, etc. The load balancing node 108 may determine or select a prediction function for the stored procedure from a data structure comprising mappings between the plurality of stored procedures and the plurality of prediction functions based at least in part on the identification of the stored procedure.

By way of example and not limitation, load balancing node 108 may determine or obtain an identification of a stored procedure, obtain an identification of a prediction function based on the identification of the stored procedure and the mapping relationship, and select the prediction function among a plurality of prediction functions based on the identification of the prediction function.

In an implementation, each prediction function may include a classification model trained based on a plurality of partition key values and corresponding identities of a plurality of partitions, the plurality of partitions including data segments corresponding to the plurality of partition key values in a database table, and the plurality of partitions may be stored in a plurality of compute nodes, respectively. By way of example and not limitation, the classification model may include a neural network model.

At block 610, the load balancing node 108 may extract or determine input values for one or more input parameters of the stored procedure.

In an implementation, the stored procedure may include a parameter list through which the user 114 may enter values for one or more input parameters of the stored procedure. The load balancing node 108 may extract or determine input values for one or more input parameters of the stored procedure from the parameter list. In an implementation, the load balancing node 108 may use static program analysis to extract or determine input values for one or more input parameters of the stored procedure.

At block 612, the load balancing node 108 may determine a compute node to which the stored procedure is sent or allocated based at least in part on the prediction function and input values of one or more input parameters of the stored procedure.

In implementations, after determining the prediction function and the input values for the one or more input parameters of the stored procedure, the load balancing node 108 may determine the computing node to which the stored procedure is sent or distributed. In an implementation, the load balancing node 108 may apply a prediction function to input values of one or more input parameters of the stored procedure to produce an output of the prediction function. In implementations, the output of the prediction function may be a tag related to a location of the data potentially or potentially accessible or queried by at least one query request of the stored procedure. In an implementation, the tag may include an identification of a data fragment of a database table that includes or stores data that is potentially or potentially accessed or queried by at least one query request of a stored process. In an implementation, load balancing node 108 may then employ a placement function to map the identity of the data slice to the identity of the computing node that includes or stores the data slice.

In an implementation, each prediction function of the plurality of prediction functions may be combined with a partition function to form a respective combined prediction-partition function, and given the same identification as the respective prediction function. In this case, the load balancing node 108 may directly use the combined prediction-partition function to obtain the identity of the compute node to which the storage process is assigned.

At block 614, the load balancing node may send the stored procedure to the compute node.

After determining the identity of the compute node, the load balancing node may send the stored process to the compute node to allow the compute node to process the stored process and/or to coordinate and manage the processing of the stored process as a control node.

Although the method blocks described above are described as being performed in a particular order, in some implementations, some or all of the method blocks may be performed in other orders or in parallel.

Conclusion

Although implementations have been described in language specific to structural features and/or methodological acts, it is to be understood that the claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed subject matter. Additionally or alternatively, some or all of the operations may be implemented by one or more ASICs, FPGAs, or other hardware.

The present disclosure may be further understood using the following clauses.

Clause 1: a method implemented by a load balancing node, the method comprising: receiving a distributed transaction comprising a stored procedure; selecting a prediction function for the stored procedure from a plurality of prediction functions; extracting one or more input parameters from a stored procedure; determining a computing node based at least in part on the one or more input parameters using the prediction function; and forwarding the stored procedure to the computing node for processing of the stored procedure.

Clause 2: the method of clause 1, wherein the one or more input parameters include one or more partitioning key values associated with one or more query requests included in the stored procedure.

Clause 3: the method of clause 1, wherein selecting the prediction function for the stored procedure from the plurality of prediction functions comprises: determining an identity of the stored procedure; obtaining the identifier of the prediction function according to the identifier and the mapping relation of the storage process; and selecting the prediction function from the plurality of prediction functions based on the identification of the prediction function.

Clause 4: the method of clause 1, wherein extracting the one or more input parameters from the stored procedure comprises: extracting the one or more input parameters from the parameter list of the stored procedure or from the stored procedure using static program analysis.

Clause 5: the method of clause 1, wherein the stored procedures comprise stored procedures that were previously registered and stored in the database system prior to being invoked.

Clause 6: the method of clause 1, wherein determining the computing node based at least in part on the one or more input parameters comprises: determining, based on the one or more input parameters, an identity of a data slice associated with at least one query request included in the storage process using the prediction function; and determining the identification of the computing node by using a placement function based on the identification of the data fragment, wherein the computing node stores the data fragment.

Clause 7: the method of clause 1, wherein the prediction function comprises a classification model trained based on a plurality of partition key values and corresponding identifications of a plurality of shards comprising respective data segments in a database table corresponding to the plurality of partition key values, the plurality of shards being stored separately in a plurality of compute nodes.

Clause 8: the method of clause 7, wherein the classification model comprises a neural network model.

Clause 9: a computer-readable medium storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising: performing a static program analysis on the stored procedure to determine one or more input parameters; obtaining stored data comprising a plurality of sets of values for the one or more input parameters of the stored procedure, and an identification of a plurality of data shards; training a classification model by taking the value sets as input and corresponding identifications of the data fragments as output; and setting the trained classification model as a prediction function for mapping a set of new values of the one or more input parameters of the stored procedure to corresponding data slices.

Clause 10: the computer-readable medium of clause 9, wherein each data slice of the plurality of data slices includes a corresponding data segment that is evaluated by the stored procedure when the one or more input parameters assume a respective set of values of the plurality of sets of values.

Clause 11: the computer-readable medium of clause 9, wherein performing the static program analysis on the stored procedure to determine the one or more input parameters comprises: identifying a variable representing a partition key in at least one query request included in the stored procedure; and determining one or more input parameters of the variables using a graph traversal algorithm.

Clause 12: the computer-readable medium of clause 9, wherein the classification model comprises a neural network model.

Clause 13: the computer-readable medium of clause 9, further comprising: and sending the classification model to a load balancing node.

Clause 14: the computer-readable medium of clause 9, further comprising: receiving, from a plurality of computing nodes, new data associated with a new set of values for the one or more input parameters of the stored process, and an identification of a data slice that includes corresponding data segments evaluated by the stored process when the one or more input parameters assume the new set of values.

Clause 15: the computer-readable medium of clause 14, further comprising: retraining the classification model based at least in part on the stored data and the new data.

Clause 16: a system, comprising: one or more processors; and a memory storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the acts of: performing a static program analysis on the stored procedure to determine one or more input parameters; obtaining stored data comprising a plurality of sets of values for the one or more input parameters of the stored procedure, and an identification of a plurality of data shards; training a classification model using the plurality of value sets as inputs and corresponding identifications of the plurality of data patches as outputs; and setting the trained classification model as a prediction function for mapping a set of new values of the one or more input parameters of the stored procedure to corresponding data slices.

Clause 17: the system of clause 16, wherein each data slice of the plurality of data slices includes a corresponding data segment that is evaluated by the storage process when the one or more input parameters assume a respective set of values of the plurality of sets of values.

Clause 18: the system of clause 16, wherein performing the static program analysis on the stored procedure to determine the one or more input parameters comprises: identifying a variable representing a partition key in at least one query request included in the stored procedure; and determining one or more input parameters of the variables using a graph traversal algorithm.

Clause 19: the system of clause 16, wherein the classification model comprises a neural network model.

Clause 20: the system of clause 16, wherein the actions further comprise: receiving, from a plurality of computing nodes, new data associated with a new value set of the one or more input parameters of the stored process, and an identification of data shards that respectively include corresponding data segments evaluated by the stored process when the one or more input parameters assume the new value set; and retraining the classification model based at least in part on the stored data and the new data.

Claims

1. A method implemented by a load balancing node, the method comprising:

receiving a distributed transaction comprising a stored procedure;

selecting a prediction function for the stored procedure from a plurality of prediction functions;

extracting one or more input parameters from the stored procedure;

determining a computing node based at least in part on the one or more input parameters using the prediction function; and

forwarding the stored procedure to the compute node to process the stored procedure.

2. The method of claim 1, wherein the one or more input parameters include one or more partitioning key values associated with one or more queries included in the stored procedure.

3. The method of claim 1, wherein selecting the prediction function for the stored procedure from the plurality of prediction functions comprises:

determining an identity of the stored procedure;

obtaining the identifier of the prediction function according to the identifier and the mapping relation of the storage process; and

selecting the prediction function from the plurality of prediction functions based on the identification of the prediction function.

4. The method of claim 1, wherein extracting the one or more input parameters from the stored procedure comprises:

extracting the one or more input parameters from a parameter list of the stored procedure or from the stored procedure using static program analysis.

5. The method of claim 1, wherein the stored procedures comprise procedures that are pre-registered and stored in a database system prior to being invoked.

6. The method of claim 1, wherein determining the compute node based at least in part on the one or more input parameters using the prediction function comprises:

determining, with the prediction function, an identification of a data slice associated with at least one query included in the storage process based on the one or more input parameters; and

and determining the identification of the computing node by using a placement function based on the identification of the data fragment, wherein the computing node stores the data fragment.

7. The method of claim 1, wherein the prediction function comprises a classification model trained based on a plurality of partition key values and corresponding identities of a plurality of shards comprising respective data segments in a database table corresponding to the plurality of partition key values, the plurality of shards being stored in a plurality of compute nodes, respectively.

8. The method of claim 7, wherein the classification model comprises a neural network model.

9. A computer-readable medium storing executable instructions, wherein the executable instructions, when executed by one or more processors, cause the one or more processors to perform the acts of:

performing a static program analysis on the stored procedure to determine one or more input parameters;

obtaining stored data comprising a plurality of sets of values for the one or more input parameters of the stored procedure, and an identification of a plurality of data shards;

taking the value sets as input and corresponding identifications of the data fragments as output, and training a classification model; and

setting the trained classification model as a prediction function for mapping a set of new values of the one or more input parameters of the stored procedure to corresponding data slices.

10. The computer-readable medium of claim 9, wherein each data slice of the plurality of data slices includes a corresponding data segment that is evaluated by the stored procedure when the one or more input parameters assume a respective set of values of the plurality of sets of values.

11. The computer-readable medium of claim 9, wherein performing static program analysis on the stored procedure to determine one or more input parameters comprises:

identifying a variable representing a partition key in at least one query included in the stored procedure; and

one or more input parameters of the variables are determined using a graph traversal algorithm.

12. The computer-readable medium of claim 9, wherein the classification model comprises a neural network model.

13. The computer-readable medium of claim 9, further comprising: and sending the classification model to a load balancing node.

14. The computer-readable medium of claim 9, further comprising:

receiving, from a plurality of computing nodes, new data associated with a new value set of the one or more input parameters of the stored process, and an identification of data shards that respectively include corresponding data segments evaluated by the stored process when the one or more input parameters assume the new value set.

15. The computer-readable medium of claim 14, further comprising: retraining the classification model based at least in part on the stored data and the new data.

16. A system, comprising:

one or more processors; and

a memory storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:

obtaining stored data comprising a plurality of value sets for the one or more input parameters of the stored procedure and an identification of a plurality of data shards;

17. The system of claim 16, wherein each data slice of the plurality of data slices comprises a corresponding data segment that is evaluated by the storage process when the one or more input parameters assume a respective set of values of the plurality of sets of values.

18. The system of claim 16, wherein performing static program analysis on the stored procedure to determine the one or more input parameters comprises:

19. The system of claim 16, wherein the classification model comprises a neural network model.

20. The system of claim 16, wherein the actions further comprise:

receiving, from a plurality of computing nodes, new data associated with a new set of values for the one or more input parameters of the stored process and an identification of a data shard that respectively includes corresponding data segments evaluated by the stored process when the one or more input parameters assume the new set of values; and

retraining the classification model based at least in part on the stored data and the new data.