JP7083901B2 - Dark Roch Realization Method, Equipment, Computation Node and System - Google Patents

Description

(Mutual reference of related applications)
This application is filed on the basis of a Chinese patent application dated December 29, 2017, with an application number of 201711499192.5, and claims the priority of the Chinese patent application and all the contents of the Chinese patent application. Is incorporated herein by reference.

The present application relates to distributed data processing techniques, and is particularly limited to, but not limited to, dark clone implementation methods, devices, computing nodes and systems based on distributed databases.

More and more applications are adopting Dark Roche policies for updates to reduce and thus avoid the impact of system updates on user usage. Dark roche (also called canary release) is a release method that allows a smooth transition between black and white. Dark clones can be A / B tested, that is, some users will continue to use product function A and some others will begin to use product function B. If the user has no objection to the product function B, the range is gradually expanded and all users are transferred to the product function B. Dark Roche can guarantee the stability of the entire system and control its impact by finding and adjusting problems early on.

In related technologies, commonly used dark clone policies involve an application introducing the corresponding traffic into the original service system and the new service system, respectively, depending on the list of official and trial users. Such a method adds a new service system and at the same time adds a corresponding set of functional modules such as a configuration center. As described above, the application needs to maintain both the old and new service codes corresponding to the original service system and the new service system in order to retain both the old and new data structures and their data contents.

Currently, the system implements redundancy mainly on the application side and the server side, and rarely participates in the database, and even if it is involved in the database, the trial user and the official user use the same database, while Adapt to service modifications by adding redundant fields. For example, in one table, the official user needs to use the three fields A, B, and C, and the trial user uses the three fields A, D, and E. Given that you need to carry an official user and a trial user in the same database, this table must contain five fields, Field A, Field B, Field C, Field D, and Field E, which is clear. In addition, the field D and the field E are redundant for the data of the official user, and the fields B and C are redundant for the data of the trial user. Thus, when a formal application accesses, it must be restricted to access only formal data, and fields D and E must be masked, and when a trial application accesses, it is only trial data. It is necessary to restrict access and mask fields B and C, which limits the service, complicates the modification of the trial service, is disadvantageous for late maintenance, and rolls the version. It is also disadvantageous to the back.

In order to solve the above technical problems, the present application provides a dark roach realization method, an apparatus, a calculation node and a system capable of easily and effectively achieving the purpose of the dark roach and reducing the amount of service change.

This application is
To process the received application request and determine that the received application request is a trial request,
It analyzes the trial request based on the preset partition setting information and determines whether the trial request is the first trial request requesting the trial data or the second trial request requesting the trial data and the official data. To decide and
The part of the first trial request or the second trial request that requests the data of the trial user is transferred to the trial data node where the trial user corresponding to the trial request is located and executed, and the second trial request is executed. Provides a dark clone implementation method including transferring and executing the part requesting the data of the official user to the official data node where the official user is located.

The present application further provides a computer-readable storage medium, which, when executed by a processor, stores a computer program that realizes any one of the steps of the dark clone implementation method in the embodiment of the present application.

The present application further comprises a dark cloned realization apparatus comprising a processor and, when executed by said processor, a computer-readable storage medium in which instructions for realizing any one of the dark clone realized methods of the embodiments of the present application are stored. offer.

This application is
A filtering module configured to process incoming application requests and determine that the received application request is a trial request,
Analyzes the trial request based on the loaded partition configuration information and determines whether the trial request is the first trial request requesting trial data or the second trial request requesting trial data and formal data. With an analysis module configured to
In the first trial request or the second trial request, the part requesting the data of the trial user is transferred to the trial data node where the trial user corresponding to the trial request is located and executed, and in the second trial request, Further provided is a dark clone implementation device including a transfer module configured to transfer and execute a portion requesting data of a formal user to a formal data node in which the formal user is located.

The present application further provides a compute node comprising the dark clone realization device according to any one of the embodiments of the present application.

This application is
A data node cluster in a distributed database that contains two or more data nodes and is configured to carry user data.
Containing two or more compute nodes, including a compute node cluster in a distributed database configured to provide services externally, data is distributed across multiple data nodes by partitioning information, said compute nodes of the present application. Further provided is a darker node realization system comprising the darker node realization device according to any one of the embodiments.

The technical solution of the present application is to process the received application request, determine that the received application request is a trial request, analyze the trial request based on the loaded partitioning information, and perform the trial request. Is a first trial request requesting trial data or a second trial request requesting trial data and formal data, and based on preset trial modification setting information. Rewriting the part requesting the data of the trial user in the first trial request or the second trial request, and the part requesting the data of the trial user in the rewritten first trial request or the second trial request. Transfer to the trial data node where the trial user corresponding to the trial request is located and execute it, and transfer the part of the second trial request that requests the data of the official user to the official data node where the official user is located. Including to do. The technical solution according to the present application distinguishes between a trial user and an official user in the database layer, the purpose of dark clone is easily and effectively achieved, and the amount of service change is reduced.

Other features and advantages of the invention will be obvious by description in the specification below or will be understood by the practice of the invention. The object and other advantages of the present invention can be realized and obtained by the structure specifically pointed out in the specification, claims and drawings.

The drawings are intended to provide a further understanding of the technical solutions of the present application, which form part of the specification and interpret the technical solutions of the present application together with the embodiments of the present application. Does not limit the solution
It is a flowchart of the dark clone realization method which concerns on this application. It is a schematic diagram of the Example which realizes the data partition and the heterogeneous table of the same name which concerns on this application. It is a schematic diagram of the Example of the user data partition which supports the dark crocodile which concerns on this application. It is a schematic diagram of the example of the data redistribution of the distributed database which concerns on this application. It is a schematic diagram of the example of the statistical processing of the trial request report which concerns on this application. It is a flowchart of the Example of the clearing process which concerns on this application. It is a structural schematic diagram of the dark rochi realization apparatus which concerns on this application. It is a schematic diagram of the configuration example of the calculation node which concerns on this application. It is a schematic diagram of the whole architecture which realizes a dark clone based on the distributed database which concerns on this application. It is a structural schematic diagram of the dark rochi realization apparatus which concerns on this application. It is a schematic diagram of the 1st Example which realizes the dark crocodile which concerns on this application. It is a schematic diagram of the 2nd Example which realizes the dark crocodile which concerns on this application. It is a schematic diagram of the 3rd Example which realizes the dark crocodile which concerns on this application. It is a schematic diagram of the 4th Example which realizes the dark crocodile which concerns on this application.

In order to further clarify the objectives, technical solutions and advantages of the present application, examples of the present application will be described in detail below with reference to the drawings. If they do not collide, the embodiments of the present application and the features of the embodiments can be arbitrarily combined with each other.

FIG. 1 is a flowchart of a dark clone realization method according to the present application, and as shown in FIG. 1, includes the following.

Step 100: The compute node processes the received application request and determines that the received application request is a trial request.

In one embodiment, if the received application request is determined to be a trial request, then the method of the present application is:
It further includes transferring and executing the formal request to the formal data node where the formal user corresponding to the formal request is located.

In one embodiment, in this step, the compute node is a formal request that the received application request is a trial request or requests formal data by filtering the received application request by IP whitelist. Distinguish whether there is.

In one embodiment, the IP whitelist comprises a formal request IP whitelist and a trial request IP whitelist, and filtering the received application requests by the IP whitelist can be done.
Formal Request To determine that an application request from a formal application database client in the IP whitelist is a formal request,
Trial Requests Determining that an application request from a trial application database client in the IP whitelist is a trial request,
Includes discarding an application request from a non-whitelist IP, i.e., which does not belong to the formal application database client in the formal request IP whitelist and does not belong to the trial application database client in the trial request IP whitelist.

The present application filters rogue application requests by IP whitelist.

Prior to this step, to ensure that the IP whitelist of the distributed database performs IP filtering on database client requests in this step.
In collaboration with the application, it provides traffic guidance, for example, identifies the request including identification information, etc., and identifies the request of the trial user and the official user as different IP database clients (usually the application, not the user's client). Further includes guiding to (deployed on the server). There are many specific implementation forms, and the scope of protection of the present application is not limited, and the description thereof will be omitted here.

Step 101: In the first trial request that analyzes the trial request based on the preset partition setting information and requests the trial data, or in the second trial request that requests the trial data and the formal data. Decide if there is.

In one embodiment, the partitioning information includes, but is not limited to, any combination of List partition, ER partition, Hash partition, Range partition, etc. Are combined and distinguished.

In one embodiment, prior to the present application,
Utilizing the data partitioning function of the distributed database based on the preset partition setting information to distinguish the trial user data and the official user data at the database layer, that is, to distinguish the trial user from the official user. Including further. Further, in the present application, by using the same partition setting information, all related data of the same user have the same distribution rule, that is, the same partition setting information is adopted for the data distribution of the same user. Guarantee. The same user can be involved in many tables, but these related tables will have the same partitioning information.

In one embodiment, parsing the trial request based on the partitioning information preset in step 101 specifically determines the table required by the query based on the trial request and partitioning. It includes querying the partitioning rules of the table informedly and determining the data node corresponding to the trial request according to the partitioning rules.

As an example, assuming that the syntax of the trial request is select * from Where id = 3, the table T is range partitioned by id based on the preset partition setting information, and id1 to id10 are data. Assuming that it is distributed to node 1 and it is known that id11 to id20 are distributed to data node 2, the data of the trial request is distributed to data node 1, that is, the trial request is distributed to data node 1. Analysis shows that it can be sent and executed.

Step 103: The part of the first trial request or the second trial request that requests the data of the trial user is transferred to the trial data node where the trial user corresponding to the trial request is located and executed, and the second trial is executed. The part of the request that requests the formal user's data is transferred to the formal data node where the formal user is located and executed.

In one embodiment, for a second trial request that involves both the trial user and the official user at the same time, for example, for a statistical report, the trial user is located by putting the requests together after the data execution is completed. It further includes summarizing some execution results sent to the data node and some execution results sent to the formal data node where the formal user is located.

In one embodiment, after step 101 and before step 103, the dark clone realization method according to the present application is:
Step 102: Further includes rewriting the part of the first trial request or the second trial request that requests the data of the trial user based on the preset trial modification setting information.

The distributed database supports heterogeneous tables of the same name, which are tables with the same name but different data structures, and in order to meet the demands on the data structure and service logic of the trial service, in one embodiment, before the present application. ,
It also includes rewriting the table structure of the trial user database based on the trial modification setting information so that the data table structure used by the trial user and the official user are different. The table structure of the trial user's table and the table structure of the official user's table are different, but they still use the table with the same name.

For example, there are two syntaxes in the request, the first syntax is select A, B, C from Where id = 3, and the second syntax is select A, B, C from Where id = 13. The table T is range-partitioned by id, id1 to id10 are distributed to the data node 1, and id11 to id20 are distributed to the data node 2, based on the preset partition setting information. Assuming you know
If the trial modification setting information is not added to the first syntax, there is no need to rewrite the SQL, and the first syntax is sent to the partition data node 1 as it is, and the second syntax is addressed to. Assuming that the trial modification setting information of D = A + B and E = 10 * C is added, B in the original syntax is replaced with DA, C in the original syntax is replaced with E / 10, and finally. The syntax sent to the partition data node 2 is rewritten as select A, DA, E / 10 from Where id = 13.

That is, it is determined whether or not rewriting is necessary depending on which table the trial modification setting information is involved in and which field is involved, and if the field is not involved in the modification, it is not necessary to rewrite the SQL.

In this way, access support to heterogeneous databases using the same syntax of the service is realized, and there are no cases where the dimensions do not match. For example, the fields in the original table are fields A, B and C, and the trial user's fields are fields A, D and E. If D = A + B and E = 10 * C, the trial user's table is actually field A, field (A + B) and field (10 * C) according to the rewriting method of the present application, and this dimension is the same as before. We can guarantee that you can roll back easily and you will not lose any information.

FIG. 2 is a schematic diagram of an embodiment that realizes a data partition and a heterogeneous table having the same name according to the present application. When a distributed database is not used, all the data in the same table is set in one data node and stored in the table. All data rows are stored, and as shown on the left side of FIG. 2, 9 rows of data are set in the table T on the single node. When using a distributed database, data is distributed among three data nodes, data node 1 is set to lines 1 to 4, data node 2 is set to lines 5 to 8, and the data node is set. Assuming that the 9th line is set in 3, the data distribution method in FIG. 2 can be realized by various policies.

For example, if the List partition is adopted, the data can be distributed to different data nodes by the enumeration method, for example, set1 {1, 2, 3, 4}, set2 {5, 6, 7, 8}, set3 {9}. Can be done
Further, for example, if a Range partition is adopted, data can be distributed to different data nodes according to a defined range, for example, set1 {1 to 4}, set2 {5 to 8}, and set3 {9 to ∞}.
Further, for example, another partition method, for example, a Hash partition, a related partition, or the like is adopted.

In one embodiment, the table structure can be modified if one expects the structure of the table T on different data nodes to be different. For the table T on the data node 2 and the data node 3 in FIG. 2, the table structure on the data node 2 is changed to T (A, by manipulating the table structure by alter, for example, by setting alter table set C = BC. B, BC), for example, alter table set C = B + C, so that the table structure on the data node 3 can be modified to T (A, B, B + C).

The modification of the table structure is performed after the table of the calculation node is created, and the original table structure T (A, B, C) is still shown to the calculation node by the entire data node. That is, in the case of a distributed database, the table structure is shown to the outside by the compute node, the table T in the formal database and the trial database is the whole, and the table surface structure shown to the outside is T regardless of any trial modification. It remains (A, B, C).

FIG. 3 is a schematic diagram of an example of a user data partition that supports dark clones according to the present application, and as shown in FIG. 3, it is necessary to set a trial user selection policy, that is, a distributed database partition policy. For example, a user with an ID of 1 to 100 can be set as a trial user according to a range, which belongs to the Range distribution, and the data is divided into official data and trial data. All data of the same user is distributed to the same database, tables T1, table T2, table T3 and table T4 related to the official user are all distributed to the official data node, and tables T1 and tables related to the trial user are distributed. To ensure that T2, table T3 and table T4 are all distributed to the trial data nodes, the same distribution rule is set for the related tables.

For the related table of the trial user, modify the structure of the corresponding table according to the demand of the trial service, and as shown in the trial library of FIG. 4, the table name matches the table name in the official data node, but the table. The structure is different. Regarding the modification of the table structure, for example, to change the table T (A, B, C) of the official library to the table T (A, B, BC) of the trial library, the SQL rewrite setting of the corresponding table (for example, C = BC) should be added.

The official user query is related only to the official library (data in the broken line frame on the left side), and the trial user query is involved in the trial library and the official library (data in the dashed line frame on the right side).

The trial user statically routes and screens the greylist for SQL requests that are not involved in the greylist crowd, then sends the SQL syntax as is to the formal library, as shown in basic information table 2 in FIG. be. The trial user rewrites the corresponding SQL in response to the SQL request involved in the graylist crowd, then sends it to the trial node and executes it. For example, according to the rewrite setting, the original select C from T is selected B + C from. After rewriting to T, send to the trial node.

Due to the use of data partitioning techniques for distributed databases, it is inevitable that the data will be divided into fine sharding, distributed across data node clusters, and the hotspot data of the service will appear during execution. In one embodiment, the present application is to ensure better utilization of the device, which can cause the single data node to become overstressed by concentrating on the loaded partition.
Further including redistributing the data to equalize the load, redistributing the data to equalize the load can move some hotspot data to additional partitions or from a heavily loaded data node. Includes transitioning to a lightly loaded data node.

Repartitioning of user partitions can be achieved by the data redistribution function of the distributed database. By using the data redistribution function, it is possible to transition data based on the default partition setting information when the service does not stop. As shown in FIG. 4, the original data is distributed to four data nodes, but due to the high stress of the data node 4, the data of the data node 4 is redistributed, for example, some data is added to the additional data. You can transition to node 5.

In one embodiment, the method of the present application further comprises modifying the table structure of the corresponding trial table depending on the application characteristics of the trial user. For example, in the original database, a certain field M is in the unit of "original", but what the service logic shows to the user is in the unit of "10,000 yuan". In that case, the trial modification modifies the field in the database to M / 10000.

Taking Table T as an example, it is assumed that the fields in the corresponding formal database include fields A, B and C, and the fields in the corresponding trial database include fields A, B and D, and the configuration rule is D. Assuming = C + B, the trial modification setting information that is a rewrite rule for the syntax including the field C in the application request, for example, the general-purpose SQL request, is C = DB. Specific examples are shown in Table 1.

In this way, when the compute node receives a general-purpose SQL request, it first determines the data partition involved based on the preset partition setting information, and if it is involved with the users of the official database and the trial database at the same time, it is SQL. It is determined whether or not rewriting is necessary according to the rewriting rule, and finally, the rewritten syntax is transmitted to the data node which is the corresponding database (DB).

In one embodiment, the method of the present application
If the user determines that the trial application can be officially published, it modifies the data structures of all the original official databases to the data structures of the trial database, that is, updates the data structures of the original official database.
If the user decides that the trial application does not apply, he rolls back the data structure of the trial database to the data structure of the official database, that is, rolls back the data structure of the trial database.

For the above update or rollback scene, the update may be executed by generating an automation script for the SQL setting rule, or it may be realized by introducing data and transitioning the user.

Taking Table T as an example, it is assumed that the fields in the corresponding formal database contain fields A, B and C, the fields in the corresponding trial database contain fields A, B and D, and the configuration rule is Assuming D = C + B, the update and rollback scripts are shown in Table 2.

Taking Table T1 as an example, it is assumed that the fields in the corresponding formal database include fields A, B and C, and the fields in the corresponding trial database include fields A, B and D, and the correspondence in table T2. It is assumed that the fields in the formal database include fields E, F and G, and the setting rule is T1. D = T1. A + T1. B + T2. Assuming G, the update and rollback scripts are as shown in Table 3.

Requests that simultaneously involve the trial user and the official user are generally statistical report processing, and FIG. 5 is a schematic diagram of an example of statistical processing of the trial request report according to the present application, as shown in FIG. Requests involving the trial user and the official user at the same time are generally the processing of statistical reports, and the official database and the trial database share the same report generator, for example, reports generated by the same storage program. Has the same structure as the allocation report, the compute node is responsible for calling the report generator for the formal database and the trial database, and introduces different parameters into the report generator by loading the SQL rewrite rules. , Matching the table structure of heterogeneous tables with the same name in each database, the compute node is responsible for putting together the allocation reports that are the reports generated by each database.

As shown in FIG. 5, taking the storage process as an example, the same storage process exists in both the formal database and the trial database, and when the compute node issues a report generation instruction, it is based on the table involved in the report. Trigger different rewrite rules and introduce different parameters, for example, for table T, batch process field C in the storage process of the formal database and process the result of field E / 10 in the storage process of the trial database. do.

As shown in FIG. 5, taking a statistical storage process as an example, the storage process statistics the number of fields in the table that are greater than 100. In the official database, it is introduced directly in the field C, but in the trial database, it is necessary to introduce E / 10 in order to realize accurate statistics.

FIG. 6 is a flowchart of an embodiment of the clearing process according to the present application, and as shown in FIG. 6, the clearing program in Dark Loch mainly has only one copy of the clearing file, and the invoice record table is a formal database and. It is distributed across the trial database and handles two issues where the structure of the clearing record table in the official database and the trial database may be different.

Create a clearing temporary table in the official database and trial database, and match the table structure of the temporary table with the clearing data file derived from the clearing system.
The compute node introduces the entries in the clearing data file into the corresponding temporary table according to the partitioning rules of the official user (entry 1, entry 2 and entry 3 in Figure 6) and the trial user (entry 4 and entry 5 in Figure 6). To do and
A compute node calls a clearing program (eg, a storage process) in each database and loads SQL rewrite rules to introduce different parameters into the clearing program in different partitioned nodes and match different clearing tables with the same name in the database. (By rewriting, adjust the syntax in the request so that it resembles the structure of the table in the database).
After the clearing of each assigned database is completed, the compute node summarizes the clearing results.

FIG. 7 is a schematic configuration diagram of the dark clone realization device according to the present application, and as shown in FIG. 7, the device is
A filtering module configured to process incoming application requests and determine that the received application request is a trial request,
It analyzes the trial request based on the preset partition setting information and determines whether the trial request is the first trial request requesting the trial data or the second trial request requesting the trial data and the official data. An analysis module configured to determine and
In the first trial request or the second trial request, the part requesting the data of the trial user is transferred to the trial data node where the trial user corresponding to the trial request is located and executed, and in the second trial request, It includes at least a transfer module configured to transfer and execute the part that requests the data of the formal user to the formal data node where the formal user is located.

In one embodiment, the apparatus according to the present application is
Based on the preset trial modification setting information, the part of the first trial request or the second trial request that requests the data of the trial user is rewritten, and the rewritten first trial request or second trial request is performed. Further includes a processing module configured to output to the transfer module.

In one embodiment, the filtering module further determines that the received application request is a formal request requesting formal data, so that the formal request is sent to the formal data node where the formal user corresponding to the formal request is located. It is configured to transfer and execute.

In one embodiment, the filtering module filters the received application request by an IP whitelist so that the received application request is a trial request requesting trial data or a formal request requesting formal data. It is configured to distinguish between. More specifically
The IP whitelist is divided into two parts, the formal request IP whitelist and the trial request IP whitelist. The application request from the formal application database client in the formal request IP whitelist is the formal request, and the trial request IP whitelist. The application request from the trial application database client in is a trial request, and the application request from the non-whitelist IP is discarded.

In one embodiment, the analysis module determines the table required by the query based on the trial request, queries the partitioning rules for the table based on the partitioning information, and depending on the partitioning rules, the trial. It is configured to determine the data node that corresponds to the request.

In one embodiment, the analysis module also utilizes the data partitioning feature of the distributed database to distinguish between trial user data and official user data at the database tier, based on preset partitioning information. That is, it is configured to distinguish between a trial user and an official user. The application also guarantees that all relevant data of the same user will have the same distribution rule using the same partitioning information.

In one embodiment, the processing module is further configured to rewrite the table structure of the trial user database based on the trial modification setting information so that the data table structure used by the trial user and the official user are different.

In one embodiment, the processing module is further configured to modify the table structure of the corresponding trial table, depending on the application characteristics of the trial user.

In one embodiment, the processing module also places trial data in which the trial user is located after the execution of the data is complete for a second trial request that involves the trial user and the official user at the same time, such as a statistical report. It is configured to combine some execution results sent to the node and some execution results sent to the official data node where the official user is located.

In one embodiment, the processing module is further configured to redistribute the data to equalize the load, and redistributing the data to equalize the load adds some hotspot data. Includes moving to a partition or transitioning from a heavily loaded data node to a lightly loaded data node.

In one embodiment, the processing module further
If the user determines that the trial application can be officially published, the data structure of all the original official databases is modified to the data structure of the trial database, that is, the data structure of the original official database is updated and the user However, if it determines that the trial application does not apply, it is configured to roll back the data structure of the trial database to the data structure of the official database, that is, to roll back the data structure of the trial database.

In an actual application, the functions realized by each means in the above-mentioned darkness realization device are the central processing unit (CPU, Central Processing Unit), the microprocessor (MPU, MicroProcessor Unit), and the digital signal in the darkness realization device. It can be realized by a processor (DSP, Digital Signal Processor), a field programmable gate array (FPGA, Field Programmable Gate Array), or the like.

FIG. 8 is a schematic diagram of a configuration embodiment of a calculation node according to the present application, and as shown in FIG. 8, includes an IP white list, a static routing module, and a data rewriting module, and the IP white list is specifically shown in FIG. 7. The static routing module is specifically realized as any one of the analysis modules in FIG. 7, and the data rewriting module is specifically realized as any one of FIG. 7. It is realized like a processing module.

When the data node is started, the compute node loads the IP whitelist configuration information by the IP whitelist, the data partition configuration information by the static routing module, and the SQL rewrite rule configuration information by the SQL data rewrite module. .. The operating principle of the calculation node according to the present application includes the following.

The IP whitelist module of the data compute node filters the whitelist for all received external application requests, and the whitelist contains two parts: the official request IP whitelist and the trial request IP whitelist. The SQL request sent from the non-whitelist IP is discarded, the request sent from the formal client (for example, IP1 in the figure) is sent to the formal database as it is, and the trial client (for example, IP2 in the figure) sends it as it is. Sends the incoming request to the static routing module for processing and
The static routing module of the compute node routes and analyzes the SQL request based on the data partition setting information. By analyzing the SQL request, the SQL request is classified into three types: individual request for formal data, individual request for trial data, and simultaneous request for multi-part user data. The SQL request requesting formal data is sent to the formal database, and the SQL request requesting trial data and the SQL request requesting multi-part user data at the same time are rewritten by the data rewriting module and then sent to the trial database.

FIG. 9 is a schematic diagram of an overall architecture that realizes a dark clone based on the distributed database according to the present application, and as shown in FIG. 9, the tip of the distributed database is a plurality of applications, for example, application 1 of FIG. , Application 2 ... Communicating with application X, the plurality of applications are used to process application requests from official users and application requests from trial users, respectively.

A data node cluster in a distributed database includes a plurality of data nodes, such as data node 1, data node 2 ... data node M in FIG. 1, and is used to carry all user data. The data is distributed to multiple data nodes by the partition policy, and each data node carries some user data.

A compute node cluster in a distributed database includes a plurality of compute nodes, such as compute node 1, compute node 2 ... compute node N in FIG. 1, and is used to provide services to the outside, specifically any of FIG. Contains one compute node. Computational nodes adopt a non-shared architecture, provide a unified standard structured query language (SQL) interface for applications, and multiple compute nodes perform load balancing. The compute node can send and execute the corresponding SQL syntax to the corresponding data node by finding the corresponding dataset location based on the data partition information.

The present application further provides a computer-readable storage medium, which, when executed by a processor, stores a computer program that implements any one of the steps of the dark clone implementation method described above.

FIG. 10 is a schematic configuration diagram of a dark roach realization device according to the present application, and as shown in FIG. 10, a processor and an instruction for realizing any one of the above dark roach realization methods when executed by the processor are issued. Includes stored computer-readable storage media.

Computer-readable storage media include flash memory, hard disk, multimedia card, card-type memory (for example, SD memory card (SD card, Secure Digital Memory Card) or data register (DX, Data Register) memory, etc.), random access memory (for example). RAM, Random Access Memory), Static Random Access Memory (SRAM, Static Random Access Memory), Read Only Memory (ROM, Read Only Memory), Electrically Erasable Programmable Read Only Memory (EEPROM, Electrically Memory) Only one or more of (Only Memory), programmable read-only memory (ROM, Programmable Read-Only Memory), magnetic memory, magnetic disk, optical disk and the like.

The processor may be a central processing unit (CPU, Central Processing Unit), controller, microcontroller, microprocessor or other data processing chip.

Hereinafter, the dark clone realization method according to the present application will be described in detail with reference to specific application scenes.

The first embodiment is an example in which the distributed database supports the multi-stage dark clone of the service, and FIG. 11 is a schematic diagram of the first embodiment for realizing the dark clone according to the present application, which is shown in FIG. As such, the distributed database of the first embodiment has a plurality of data nodes, and usually all the data are set in the formal library. When the user wants to compare a plurality of trial modifications at the same time, the data of the trial user is assigned to a plurality of other data nodes, for example, trial library 1, trial library 2, ... trial library n, and executed. Specifically, it includes the following.

Use the distributed database redistribution feature to set the partition information of the table according to the user distribution setting information rule of the trial test, and adopt the appropriate pre-configured partition rule that satisfies the service to perform the redistribution operation on the data. The data of the test user applied to the first-class trial modification is set in the trial library 1, the data of the test user applied to the second-class trial modification is set in the trial library 2, and ... It is applied to the n-class trial modification. Set the test user data in the trial library n and set it.
Perform a trial modification operation of the table structure for each of partitions 1 to n, and perform an alter operation on the table on the new partition, for example, alter T set C = C + 1, alter T set C = C + 2, ... alter. Set T set C = C + n, and in different partitions, add the corresponding SQL rewrite settings for the trial modification operation of the table structure, for example, add C = Cn to the data node n.

In this way, when the system executes, the static routing module of the compute node analyzes the original SQL syntax and divides the routing, and for all the divided SQLs, the data rewrite module corresponds to the SQL. Find the SQL rewrite rule of the execution node, rewrite it, and then send it to the corresponding trial library to execute it. Using the syntax select C from T as an example, the syntax on the corresponding partition 2 is rewritten to select C-1 from T, the syntax on the corresponding partition 3 is rewritten to select c-2 from T, and so on. Rewrite the syntax of to select c-n from T.

All online single-node transactions (online services involving only one user data) will return as is, and if the compute node determines that it will send the syntax select c from Node in set n only to node n, then the syntax will be changed. Rewrite to select cn from T from id in set n, and return the batch node cross transaction (service involving multiple users of multiple partitions) after being summarized by the compute node, and the compute node has the syntax select c from T. If it is determined that the if id in set {n-1, n} is to be transmitted to the node (n-1) and the node n, the syntax is divided into two syntaxes and one syntax is set c- (n-1) from. T whoe id in set n-1 is sent to the node (n-1), and another syntax set c-n from T where id in set n is sent to the node n.

After the test of each trial node is completed, if a trial modification type meets the user's demand, an update script is directly generated by the SQL rewrite setting information corresponding to the modification type of the trial node, and the update script is used. You can make a trial update to the entire database.

The second embodiment is that the distributed database supports the unified construction of the global service, and FIG. 12 is a schematic diagram of the second embodiment for realizing the dark clone according to the present application, which is shown in FIG. As such, in the second embodiment, the service design department is in charge of building a unified application platform, and by requesting a unified data structure, the statistics of the data report are facilitated, and the service construction department at the bottom is each. Suppose you have some special demands and different processing methods for the same service. In this case, the distributed database system according to the present application can be used to meet the unified construction demand for global services. Specifically, it includes the following.

Different service construction departments use the same table name T for unified services and also meet the requirements for platform data unification. Using the distributed database redistribution function, set the partition information of the table according to the user distribution setting rule of each department, repartition the data, for example, set the data of department 1 to the service library 1, and set the data of department 2 to the service library 1. The data is set in the service library 2, ... The data of the department n is set in the service library n, and each construction department modifies each table structure and executes each alter syntax according to the respective service logic. In different partitions, each department adds a corresponding SQL rewrite configuration rule for the table structure modification operation.

Thus, when the system runs, the compute node's static routing module parses and splits the SQL syntax, and all post-split SQL requests have a corresponding split service library of the corresponding split service library. The data rewrite module finds the corresponding SQL rewrite rule, rewrites it, and then sends it to the corresponding split service library for execution.

Only the data related to each service library of each service construction department is returned as it is, and the unified batch transaction of the platform department, for example, the service library cross query such as global statistics is returned after being summarized by the compute node.

Different service libraries always maintain different table structures and data rewrite rules. In this way, when each service conflicts with the unified requirements of the platform, it is adjusted by modifying each table structure and rewriting the setting rule.

The third embodiment rapidly updates and rolls back the service using the distributed database, and FIG. 13 is a schematic diagram of the third embodiment that realizes the dark clone according to the present application, and is shown in FIG. As described above, in the third embodiment, the corresponding update and rollback method according to the present application is used to quickly realize the update and rollback of the service version, and basically do not occupy the service time, and the service is interrupted. Does not cause. Specifically, it includes the following.

If the service needs to be updated, the original table T (A, B, C) can be converted to table T by modifying the table structure and executing the syntax alter set C = B + C without having to modify the service code. You may rank up to (A, B, B + C). For the modification of the table structure from table T to table T', add C = B + C, which is the corresponding SQL rewrite setting information, and after adding, set all the SQL syntax related to table T'in the relevant SQL rewrite setting. It is rewritten according to the information, that is, C in the original SQL code is replaced with CB, and after the replacement is completed, it is transmitted to the corresponding data node and executed.

If the service needs to be rolled back, there is no need to modify the service code, just execute the operation of the syntax alter set C = CB on the table T, return the table T to the data structure of the previous table T, and at the same time. , SQL rewrite setting information C = B + C may be deleted. In this way, it is easy to roll back the service to the previous version.

In the fourth embodiment, the service is hotfixed by using the SQL automatic rewriting function and the routing mask function, and FIG. 14 is a schematic diagram of the fourth embodiment for realizing the dark clone according to the present application, which is shown in FIG. As described above, in the fourth embodiment, the number of data nodes in the distributed database does not matter. Service execution is usually normal. When a service logic bug (bug) related to SQL appears, the service logic can be modified by the SQL rewriting function or the SQL mask function according to the present application to reduce the influence of the bug on the user. Specifically, it includes the following.

In the process of operating normally in the production environment, especially in the case of mobile phone APP, if the logic bug appears, the application has already been distributed to the user and there is no need to directly modify the user's client APP, such a problem. On the other hand, the present application has two processing methods.
For a bug whose logic is simple, add or modify the SQL rewrite setting information of the related table based on the logic that modifies the bug, and modify the SQL logic of the original bug to the normal SQL logic. To achieve the purpose of bug modification. For example, assuming that the original SQL syntax is select A from table T, the unit calculation is wrong in the service, and it is 1000 * A that needs to be returned, the SQL rewrite setting information A = 1000. * You can modify this logic by adding A,
If the logic is complex, for example, if the SQL modification still cannot modify the service logic, the static routing module of the compute node can mask the corresponding SQL syntax, for example, the service has the SQL syntax. When executing, the error is returned as it is, and the range of influence of the bug is also limited in this way.

The dark clone implementation method and device according to the present application solve the problem that the application dark clone must constantly modify the data structure and service logic without modifying the service code, and all the modification operations are put together in a database. And the distributed data redistribution feature makes it easier for service departments to build dark clone data faster, saves system resource input, and improves the efficiency of service trial iterations.

The above is merely a preferred embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the invention should be included in the scope of protection of the invention.

Claims (12)

To process the received application request and determine that the received application request is a trial request,
It analyzes the trial request based on the preset partition setting information and determines whether the trial request is the first trial request requesting the trial data or the second trial request requesting the trial data and the official data. To decide and
If the trial request is determined to be the first trial request, the first trial request may be transferred to the trial data node where the trial user corresponding to the trial request is located and executed, or the trial request may be the second. If it is determined that the trial request is made, the part of the second trial request that requests the data of the trial user is transferred to the trial data node where the trial user corresponding to the trial request is located and executed. The part of the second trial request that requests the official user's data is transferred to the official data node where the official user is located and executed .
After analyzing the trial request based on the preset partition setting information and before performing the execution .
Further including rewriting the part of the first trial request or the second trial request that requests the data of the trial user based on the preset trial modification setting information.
Dark , including rewriting the table structure of the trial user database based on the trial modification settings information so that the data table structure used by the trial user and the official user are different, but the names used are the same. How to realize the launch. Further including distinguishing the data of the trial user from the data of the official user in the database layer by utilizing the data partitioning function of the distributed database based on the preset partition setting information.
The method of claim 1, wherein the same user uses the same partitioning information. If it is determined that the received application request is a formal request for formal data,
The method of claim 1 or 2 , further comprising transferring and executing a formal request to a formal data node in which a formal user corresponding to the formal request is located. By filtering the received application request according to a preset IP whitelist, whether the received application request is a trial request requesting trial data or a formal request requesting formal data. The method of claim 1 for distinguishing. The IP whitelist includes a formal request IP whitelist and a trial request IP whitelist.
Filtering received application requests by the IP whitelist
Formal Request To determine that an application request from a formal application database client in the IP whitelist is a formal request,
Trial Request To determine that the application request from the trial application database client in the IP whitelist is the trial request.
The method of claim 4 , comprising discarding the application request from the non-whitelist IP. Analyzing the trial request based on the preset partitioning information is
Determining the table required by the query based on the trial request
The method of claim 1 or 2 , comprising querying the partitioning rules for the table based on the partitioning information and determining the data node corresponding to the trial request according to the partitioning rules. After the execution of the data of the second trial request is completed,
Claim 1 or 2 , further comprising summarizing a part of the execution result transmitted to the trial data node where the trial user is located and a part of the execution result transmitted to the formal data node where the formal user is located. The method described. Claim 1 or further comprises redistributing the data and moving some hotspot data to additional partitions or transitioning some hotspot data from a heavily loaded data node to a lightly loaded data node. The method according to 2 . The method of claim or 2 , further comprising modifying the table structure of the corresponding trial table according to the application characteristics of the trial user. If it is determined that the trial application has been officially published, the data structure of all official databases should be modified to the data structure of the trial database corresponding to the trial application.
The method of claim 1 or 2 , further comprising rolling back the data structure of the trial database corresponding to the trial application to the data structure of the formal database if it is determined that the trial application is not applicable. A filtering module configured to process incoming application requests and determine that the received application request is a trial request,
Analyzes the trial request based on the loaded partition configuration information and determines whether the trial request is the first trial request requesting trial data or the second trial request requesting trial data and formal data. With an analysis module configured to
Based on the preset trial modification setting information, the part of the first trial request or the second trial request that requests the data of the trial user is rewritten and the rewritten first trial request or the second trial request is made. A processing module configured to output to the transfer module, and
When the trial request is determined to be the first trial request, the analysis module transfers the first trial request to the trial data node where the trial user corresponding to the trial request is located, or executes the analysis module. If it is determined that the trial request is the second trial request, transfers the part of the second trial request requesting the data of the trial user to the trial data node where the trial user corresponding to the trial request is located. Includes a transfer module configured to transfer and execute the portion of the second trial request that requests the formal user's data to the formal data node where the formal user is located.
The processing module further rewrites the table structure of the trial user database based on the trial modification setting information so that the data table structure used by the trial user and the official user are different, but the names used are the same. Dark clone realization device configured as . A data node cluster in a distributed database that contains two or more data nodes and is configured to carry user data.
Includes a compute node cluster in a distributed database that contains two or more compute nodes and is configured to provide services externally, data is distributed across multiple data nodes by partitioning information, and the compute nodes claim. 11. A darkness realization system including the darkness realization device according to 1 .

Images