CN106936899B - Configuration method of distributed statistical analysis system and distributed statistical analysis system - Google Patents
Configuration method of distributed statistical analysis system and distributed statistical analysis system Download PDFInfo
- Publication number
- CN106936899B CN106936899B CN201710105031.7A CN201710105031A CN106936899B CN 106936899 B CN106936899 B CN 106936899B CN 201710105031 A CN201710105031 A CN 201710105031A CN 106936899 B CN106936899 B CN 106936899B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- fragment
- leader
- statistical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000007619 statistical method Methods 0.000 title claims abstract description 86
- 238000000034 method Methods 0.000 title claims abstract description 50
- 239000012634 fragment Substances 0.000 claims abstract description 92
- 238000013467 fragmentation Methods 0.000 claims abstract description 20
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 20
- 230000005012 migration Effects 0.000 claims description 23
- 238000013508 migration Methods 0.000 claims description 23
- 238000013500 data storage Methods 0.000 claims description 18
- 238000007726 management method Methods 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 11
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 238000005096 rolling process Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 7
- 238000013507 mapping Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 6
- 239000012792 core layer Substances 0.000 description 4
- 230000006872 improvement Effects 0.000 description 4
- 239000010410 layer Substances 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 3
- 230000010076 replication Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012938 design process Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1034—Reaction to server failures by a load balancer
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a configuration method of a distributed statistical analysis system, wherein the distributed statistical analysis system comprises a ZooKeeper cluster, a service node and a computing node cluster, and the method comprises the following steps: electing a leader node in the computing node cluster, fragmenting data in each computing node according to a data fragmentation principle, and electing a leader fragment from copies of the data fragments; after receiving the statistical analysis request, the service node applies for a computing node from the leader node, and the leader node feeds back the computing node with the minimum task amount to the service node; after obtaining the feedback computing node, the service node sends a statistical request to the computing node; and the computing node searches the leader fragment, applies for the leader fragment to obtain an idle data fragment copy, and distributes the statistical task to the data fragment copy to execute the statistical task. The invention also provides a distributed statistical analysis system based on the configuration method.
Description
Technical Field
The invention relates to a configuration method of a distributed statistical analysis system, in particular to a configuration method of functions of configuration cluster management, data storage and migration, statistical analysis query and statistical task load sharing, and a distributed statistical analysis system obtained by the configuration method.
Background
A distributed system is a computer system interconnected by a plurality of processing resources. These processing resources, which may also be referred to as node devices, execute the same persona under unified control. For example, chinese patent CN102497280 discloses a distributed system, which can realize mutual awareness among multiple device nodes. The management efficiency is improved. It does not disclose the management and configuration of the tasks specifically performed by the device nodes.
Distributed systems are often required to have the capability of statistical analysis. However, in the current distributed statistical system, the statistical analysis task is not distributed among the device nodes according to the actual operation condition of the processing resource. Situations may occur where processing resources are not fully utilized.
Disclosure of Invention
The invention provides a configuration method of a distributed statistical analysis system and the distributed statistical analysis system obtained by the configuration method. High availability and high expandability of the system are ensured, and the load sharing of the statistical analysis task is realized.
In order to solve the above problem, in a first aspect, the present invention provides a configuration method for a distributed statistical analysis system, where the distributed statistical analysis system includes a ZooKeeper cluster, a service node, and a compute node cluster, and the method includes:
electing a leader node in the computing node cluster, fragmenting data in each computing node according to a data fragmentation principle, and electing a leader fragment from copies of the data fragments;
after receiving the statistical analysis request, the service node applies for a computing node from the leader node, and the leader node feeds back the computing node with the minimum task amount to the service node;
after obtaining the feedback computing node, the service node sends a statistical request to the computing node;
and the computing node searches the leader fragment, applies for the leader fragment to obtain an idle data fragment copy, and allocates the statistical task to the data fragment copy to execute the statistical task.
Optionally or preferably, the method further comprises:
and the service node sends the updated data to the leader node, and the leader node finds all the copies of the corresponding data fragments and synchronizes the data to all the copies of the data fragments.
Optionally or preferably, the data synchronization adopts a two-phase commit protocol, and if all the two-phase commit protocol succeeds, the two-phase commit protocol returns a success, and if any one copy fails, the two-phase commit protocol backs down and returns a failure.
Optionally or preferably, the method further comprises: and under the condition of adjusting the number of the data fragments or the fragmentation rule, remapping the relationship between the data and the data fragments.
Optionally or preferably, the method further comprises the step of adjusting the distributed statistical system when the device status changes, wherein the adjusting comprises at least one of:
when the equipment failure is detected, checking the number and the distribution condition of the shard copies, and if the number difference of the shard copies is large, performing distribution adjustment and migration on the shards to ensure that the number of the shard copies is basically the same; and
and when the new equipment is detected to be added, checking the number and the distribution condition of the copies of each board, copying the boards with the small number of copies to the new equipment if the number of the copies of the boards is different, and re-slicing if the number of the copies of the boards is the same.
Optionally or preferably, the step of searching for the leader fragment and applying for the leader fragment to obtain an idle data fragment copy by the computing node, and the step of allocating the statistical task to the data fragment copy to execute the statistical task includes:
the computing nodes executing the computing tasks divide the statistical operation into a plurality of tasks, and divide ID lists needing to be counted into groups according to the fragment rules, the computing nodes send the tasks and the ID groups to corresponding leader fragments, the leader fragments are responsible for selecting and handing over data fragment copies with less load to execute the tasks, and after all the tasks are executed, the executing nodes collect all the computing results and combine the results, and then return the results to the service nodes.
Optionally or preferably, if the statistical job requires multiple data accesses, the execution node performs task splitting and result merging again according to the requirement of the statistical job after collecting the merged result until the statistical job is completely run.
In a second aspect, the present invention provides a distributed statistical analysis system, which includes a ZooKeeper cluster, a service node, and a compute node cluster. Further comprising:
the cluster management module is used for electing a leader node in the computing node cluster, fragmenting data in each computing node according to a data fragmentation principle, and electing a leader fragment in a copy of the data fragments;
the statistical analysis module is used for applying for a computing node from the leader node after the service node receives the statistical analysis request, and the leader node feeds back the computing node with the minimum task quantity to the service node; after obtaining the feedback computing node, the service node sends a statistical request to the computing node; and the computing node searches the leader fragment, applies for the leader fragment to obtain an idle data fragment copy, and allocates the statistical task to the data fragment copy to execute the statistical task.
Optionally or preferably, the method further comprises:
and the data storage module is used for sending the updated data to the leader node by the service node, and the leader node finds all the copies of the corresponding data fragments and synchronizes the data to all the copies of the data fragments.
Optionally or preferably, the data synchronization adopts a two-phase commit protocol, and if all the two-phase commit protocol succeeds, the two-phase commit protocol returns a success, and if any one copy fails, the two-phase commit protocol backs down and returns a failure.
The configuration method of the distributed statistical system and the distributed statistical analysis system provided by the invention can share tasks and preferentially distribute the tasks to the computing nodes with smaller load and the data fragmentation operation, so that relatively idle processing resources can be automatically and fully utilized, the statistical efficiency is improved, and the system is ensured to have high expansibility and stability.
Drawings
Fig. 1 is an architecture diagram of a distributed statistical analysis system according to an embodiment of the present invention;
FIG. 2 is a flowchart of a configuration method of a distributed statistical analysis system according to an embodiment of the present invention;
FIG. 3 is another flowchart of a configuration method of a distributed statistical analysis system according to an embodiment of the present invention;
FIG. 4 is another flowchart of a method for configuring a distributed statistical analysis system according to an embodiment of the present invention;
FIG. 5 is another flowchart of a method for configuring a distributed statistical analysis system according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a distributed statistical analysis system according to a second embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Example one
The embodiment provides a configuration method of a distributed statistical analysis system, wherein, as shown in fig. 1, the distributed statistical analysis system mainly comprises three parts: a ZooKeeper cluster, a service node and a compute node cluster. The ZooKeeper cluster is used for state management of the compute node cluster. The service node is responsible for the work of decomposing and integrating statistical services, updating and controlling data and the like. The service node receives the statistical analysis request of the front end and analyzes the statistical analysis request into a content searching part and a statistical analysis part, the task of the content searching part is handed to the content searching engine to run, and the task of the statistical analysis part is handed to the computing node cluster to run. The computing node cluster is used for the work of data fragmentation and backup, the splitting and result merging of computing tasks, the load of the computing tasks and the like.
Interpretation of terms: the terms referred to in the present invention have the following meanings, unless otherwise specified:
interpretation of terms:
ZooKeeper: ZooKeeper is a distributed, open-source distributed application coordination service, is an open-source implementation of Chubby of Google, and is an important component of Hadoop and Hbase. It is a software that provides a consistent service for distributed applications, and the functions provided include: configuration maintenance, domain name service, distributed synchronization, group service, etc. ZooKeeper contains a simple set of primitives that provide interfaces for Java and C. In the invention, the ZooKeeper cluster is used for state management of the computing node cluster.
Solr: solr is a stand-alone full-text search server written in Java running in a Servlet container (e.g., Apache Tomcat or Jetty). Its main functions include full-text retrieval, hit marking, facet search, dynamic clustering, database integration, and rich text (such as Word and PDF) processing. Solr is highly scalable and provides distributed search and index replication.
Shard: data slicing;
the Shard copy: a fragmented copy of the data (Shard-1 Replica, Shard-2 Replica, Shard-3 Replica … …);
core: a set of functions (Core 1, Core2 … … in fig. 1) to manage the shard replicas and perform statistical analysis work on the replica data; core includes two parts: statistical analysis and data storage. For data storage, core participates in data update operation as a participant of the two-phase commit protocol. And storing each copy of the shrard in the Core in an instance mode, directly calling the instance locally, and binding remote services at a remote end.
over reader: a leader node among the compute nodes.
Based on the above distributed statistical analysis system, as shown in fig. 2-5, the configuration method includes the following steps:
s1: and electing a leader node in the computing node cluster, fragmenting data in each computing node according to a data fragmentation principle, and electing the leader fragment from the copies of the data fragments.
In step S1, the main purpose is to manage each compute node cluster according to a predetermined policy. This part of the functionality can be done by the ZooKeeper cluster.
The initial and termination states of the cluster are as follows:
1) initial state
ZooKeeper cluster;
each independent computing node, the core is initialized and the share belonging to the core is designated;
2) end state
The computing nodes elect overtree, and each node knows overtree and other computing nodes;
each core is distributed with a board, and copies of the same board cannot appear on the same computing node;
a board leader is chosen among multiple copies of each board, and the leader and other members of the board are known to each copy of the same board.
The following describes in detail the sub-steps of the cluster management step between the initial state and the end state:
s101: overteer election
Based on the ZooKeeper cluster, an overkeeper can be easily selected by a plurality of computing nodes. The method comprises the following steps:
1) starting the computer, and registering the IP of each computing node to the ZooKeeper by each computing node;
path: statsalysis/live _ nodes
2) Each computing node monitors StatsAnalysis/live _ nodes/, and acquires and stores IP information of all computing nodes;
3) after obtaining the IP of each computing node, each computing node registers and contends for overteerer to the zookeeper;
path: statsalysis/oversweer
4) Each computing node monitors StatsAnalysis/overkeeper/, and acquires and stores IP information of overkeeper;
s102: data slicing
In the data slicing process, the configuration file specifies the board to which the core belongs, and it is required to ensure that the copies of the same board cannot be located on the same computing node.
The data slicing method specifically comprises the following steps:
1) starting a computer, initializing the core by each computing node according to the number of the configured cores, and registering each core to the zookeeper; the path is as follows: statsanlysis/cores/;
2) the overviewer determines a data slicing rule according to the board to which each core belongs, and synchronizes to other computing nodes, specifically, see a data storage part (described below); path: Statsanlysis/shrard/Barrel 2 Shard;
s103: shard leader election
And after the boards are distributed, carrying out leader election on the copies in each board. The method comprises the following steps:
1) the Core monitors StasAnalysis/cores/, and acquires the shared board;
2) shard copies register for contending for the board leader with the ZooKeeper cluster;
the path is as follows: StatsAnalysis/shardleader/1, StatsAnalysis/shardleader/2, …;
3) monitoring the path of the board leader to which each copy belongs in the zookeeper by each copy, and acquiring and storing the information of each board leader;
in summary, the overall configuration process of the cluster management module can be summarized as the following steps:
1) configuring a ZooKeeper cluster;
2) starting a computer by a computing node, establishing a relation with a control page, configuring the number of the boards and the number of the single board copies, and triggering a cluster adding action;
3) each computing node creates own Cores (the number is determined by configuration parameters), and registers to the ZooKeeper, wherein the registration path is Statsalysis/Cores/;
4) each computing node registers to the ZooKeeper, and the registration path is StatsAnnalysis/live _ nodes/;
5) each computing node contends for overtweeler from ZooKeeper, and the path is Statsanalysis/overtweeer/;
6) monitoring StatsAnnalysis/live _ nodes/acquiring all computing nodes by overviewer;
7) the overviewer creates a path StatsAnnalysis/shardleader/{ board number } to the zookeeper, and is used for competing for the board leader among the board copies;
8) monitoring StatsAnnalysis/cores/acquiring each core by overviewer, distributing the board copies according to the cores, the number of the copies and the number of the boards, and writing the board numbers into the cores;
9) each core monitors a corresponding node, and reads a node value to obtain a distributed guard number;
10) each core registers for a board leader in the ZooKeeper, and the path is Statsanlysis/shardeal/{ board number };
11) the board leader monitors StatsAnalysis/cores/obtains each copy of the board.
S2, the service node applies for a computing node from the leader node after receiving the request of statistical analysis, and the leader node feeds back the computing node with the minimum task quantity to the service node; after obtaining the feedback computing node, the service node sends a statistical request to the computing node; and the computing node searches the leader fragment, applies for the leader fragment to obtain an idle data fragment copy, and distributes the statistical task to the data fragment copy to execute the statistical task.
In step S2, after receiving the statistical analysis request from the client, the service node decomposes the request into an inquiry statement and a statistical statement, first, holds the inquiry statement to obtain a matched enterprise ID list from the solr, then obtains an idle computing node from the overkeeper, and sends the enterprise ID list and the statistical statement to the computing node for statistical analysis.
The statistical analysis can be roughly divided into three steps: and (4) statistical statement analysis, task splitting and result merging. The statistical statement analysis is responsible for parsing the statistical statement. When the statistical analysis operation is transferred to the execution node through the overteer, the execution node splits the operation into a plurality of tasks, and divides the enterprise ID list needing to be counted into groups according to the fragment rule, the execution node is responsible for sending the tasks and the enterprise ID group to the leader of the corresponding board, and the board leader is responsible for selecting the copy with less load to execute the tasks. And after all tasks are executed, the execution node collects all calculation results, combines the results and returns the results to the service node. The single subtask of the task split only occupies smaller processing resources and can be completed quickly, thereby being beneficial to the redistribution of the subtask and avoiding the condition of idle resource waste caused by the task execution time process. In some embodiments, if the job needs to access data multiple times, as a further improvement, after collecting the merged result, the execution node performs task splitting and result merging again according to the job requirement until the job is completely run. This maintains consistency of the statistics even if the data is accessed multiple times.
In the three steps of the statistical analysis:
the format of the statistical analysis statement can be set according to the icon analysis rule of the existing DP data platform. The analytic code is taken from the single version statistical analysis code.
The task splitting specifically includes grouping enterprise ID lists related to the statistical analysis operation according to a splitting rule, wherein the number of the groups is the number of the shards. Each task is responsible for the statistical work of the data of one group. The specific statistical analysis code is modified and realized by a single version of statistical analysis code.
And the result merging step comprises the steps of collecting the statistical results of all tasks and carrying out post-processing according to the statistical analysis requirements. The specific merged code is modified and realized by single-edition statistical analysis code.
In summary, the specific process executed by the statistical analysis query module can be summarized as follows:
1) the service node receives the statistical analysis request and splits the request into two parts: a search request and a statistics request;
2) the service node calls a solr service to inquire a related enterprise ID list;
3) the service node monitors the directory StasAnalysis/oversweer/acquires the current oversweer, applies for a computing node to the oversweer, and feeds back the computing node with the minimum task amount to the service node (completes load sharing among the computing nodes);
4) the service node acquires a computing node for task operation and sends a statistical request and a related enterprise ID list to the computing node;
5) the computing node groups the enterprise ID lists according to the fragmentation rule;
6) the computing node finds the position of each board leader and applies for idle board copies to the position (load sharing among the board copies is completed);
7) the computing node sends the statistical statement and the related enterprise ID to the board copies, the statistical statement and the related enterprise ID are completed by the board copies, and if the result cannot be obtained by one-time interaction, the computing node is responsible for multiple interactions;
during the statistical work of the sub-tasks of the statistical analysis, corresponding enterprise information can be obtained from the local Shard copy according to the input enterprise ID list, and analysis is performed according to the statistical rules.
8) And the computing node collects the statistical results of the shard copies and returns the statistical results to the service node for result caching.
S3: and the service node sends the updated data to the leader node, and the leader node finds all the copies of the corresponding data fragments and synchronizes the data to all the copies of the data fragments.
Step S3 is mainly for completing the data storage work. And the data synchronization adopts a two-stage submission protocol, if all the copies are successful, the success is returned, and if any copy fails, the rollback is carried out and the failure is returned.
Determining the data fragmentation rule comprises:
1) acquiring the lower 10 bits of the enterprise ID, and grouping by value to obtain 1024 buckets (namely Barrel);
2) the list is used to store Barrel to shrard mappings.
The storage position of the mapping relation from Barrel ID to Shard ID in the ZooKeeper cluster is StatsAnalysis/Shard/, the name of the znode is Barrel2Shard, and the value of the mapping relation is json character string.
In summary, the overall flow of data storage can be summarized as follows:
1) the service node monitors StatsAnnalysis/oversweer/, acquires the current oversweer and sends the update data to the oversweer;
2) the over obtains the board id by calculation according to the fragment rule, searches for the lead of the board, and sends the updated data to the board lead;
3) the Shard leader updates the data consistently to all copies of the Shard.
S4: and under the condition of adjusting the number of the data fragments or the fragmentation rule, remapping the relationship between the data and the data fragments.
Step S4 works for migration of data. Data migration is divided into two cases: copying a single copy of the guard and re-slicing the number of modified guards. During the data migration, the update operation of the data is not allowed.
When copying the copy, if a device failure is encountered or a new device is added, the data may need to be synchronized from the shard leader, and at this time, a strategy of copying the entire copy may be adopted.
In the re-slicing, when the number of the shards is changed, the data is re-sliced, the mapping relation from Barrel to the shard is modified and updated to ZooKeeper, and the Barrel data with the mapping relation changed is copied from the original shard to the new shard and deleted from the original shard.
S5: when the state of the equipment changes, adjusting the distributed statistical system, wherein the adjustment comprises at least one of the following steps:
when the equipment failure is detected, checking the number and the distribution condition of the shard copies, and if the number difference of the shard copies is large, performing distribution adjustment and migration on the shards to ensure that the number of the shard copies is basically the same; and
and when the new equipment is detected to be added, checking the number and the distribution condition of the copies of each board, copying the boards with the small number of copies to the new equipment if the number of the copies of the boards is different, and re-slicing if the number of the copies of the boards is the same.
Therefore, when equipment fails or new equipment is added, the system can automatically and timely adjust the allocation of resources, and the expansibility of the system is ensured.
Example two
The embodiment provides a distributed statistical analysis system, which is configured by implementing a configuration method in the first embodiment, and obtains a corresponding module for executing the configuration method. The method specifically comprises the following steps: the system comprises a cluster management module, a data storage and migration module, a statistical analysis query module and a statistical task load sharing module (shown in fig. 6).
The cluster management module is used for establishing and maintaining the connection between each computing node and the ZooKeeper cluster, and is specifically used for electing an overkeeper node (i.e., a leader node) in the computing node cluster, fragmenting data in each computing node according to a data fragmentation principle, and electing a leader fragment (i.e., a sharder) in a copy of the data fragmentation.
The distribution of the shard copies needs to be adjusted as the device state changes. The device state change mainly comprises two conditions of device failure and new device adding.
When the equipment failure is detected, the cluster management module checks the number and the distribution condition of the shard copies, and if the number difference of the shard copies is large, the shards are subjected to distribution adjustment and migration so as to ensure that the number of the shard copies is basically the same and enhance the robustness of the cluster.
And when the adding of a new device is detected, checking the number and the distribution condition of the copies of each board, copying the boards with the small number of copies to the new device if the number of the copies of the boards is different, and re-slicing if the number of the copies of the boards is the same.
Here, the shard number may be specified in the configuration file, and the fragmentation status is not automatically specified by the cluster.
Adjusting the distribution of the shard copies may be accomplished by a cluster management module in cooperation with a data migration module (described below).
The initial and termination states of the cluster are as follows:
1) initial state
ZooKeeper cluster;
each independent computing node, the core is initialized and the share belonging to the core is designated;
2) end state
The computing nodes elect overtree, and each node knows overtree and other computing nodes;
each core is distributed with a board, and copies of the same board cannot appear on the same computing node;
a board leader is chosen among multiple copies of each board, and the leader and other members of the board are known to each copy of the same board.
Between the initial state and the end state, the configuration process of the cluster management module corresponds to the contents of steps S101-S103 in the first embodiment. And will not be described in detail herein.
And the data storage and migration module is used for sending the updated data to the overviewer by the service node, and the overviewer searches all the copies of the corresponding board and synchronizes the data to all the copies of the board. And the data synchronization adopts a two-stage submission protocol, if all the copies are successful, the success is returned, and if any copy fails, the rollback is carried out and the failure is returned. When equipment fails or new equipment is added, the state of the cluster changes, and after the cluster state is stable, the overserver supports the data migration work according to the board condition in the cluster.
According to the above configuration method, the data storage and migration module specifically includes a data storage module and a data migration module (corresponding to the contents of steps S3 and S4 in the implementation of one embodiment, respectively).
And the data storage module is used for updating the data to each copy of each shelf and ensuring the consistency of the data in the shelf. Further, the data storage module is further configured to determine a fragmentation rule and a coherence protocol, and determining the fragmentation rule includes:
1) acquiring lower 10 bits of enterprise ID, and grouping according to values to obtain 1024 buckets;
2) the list is used to store Barrel to shrard mappings.
The coherence protocol includes: and a two-phase submission protocol is adopted among the copies in Shard to ensure the atomicity of data updating.
The two-phase submission protocol is composed of two-phase submission protocol modules and specifically comprises two parts: coordinators (coordinators) and participants (participants). The coordinator plays a role in management and coordination, and the participants are actual operators for data access.
The data migration module is used for adjusting the distribution of the boards, only for copying and deleting the content of the board copies under the condition that the number of the boards and the slicing rule are not changed, and for remapping the data relationship under the condition that the number of the boards or the slicing rule are adjusted.
Data migration is divided into two cases: copying a single copy of the guard and re-slicing the number of modified guards. During the data migration, the update operation of the data is not allowed. The specific process may refer to the implementation manner of step S4 in the implementation example.
And the statistical analysis query module is used for acquiring a matched enterprise ID list from the solr by holding the query statement after the service node receives the statistical analysis request of the client and decomposes the request into the query statement and the statistical statement, acquiring an idle computing node from the overkeeper and sending the enterprise ID list and the statistical statement to the computing node for statistical analysis. And rewriting the statistical statement by the computing node, then accessing each board reader to obtain a copy with a smaller load in each board, and sending the running task to the node where the copy is located for execution. And after receiving the calculation results returned by the boards, the calculation nodes collect the calculation results and return the statistical results.
And when the task is not executed for a long time, the computing node kills the overtime task and reselects one of the shard copies to run the task. When the rmi synchronous calling is adopted in the code, the function of killing the overtime task is not realized.
The statistical analysis query module is specifically configured to perform the following procedures:
the statistical analysis can be roughly divided into three steps: and (4) statistical statement analysis, task splitting and result merging. The statistical statement analysis is responsible for parsing the statistical statement. When the statistical analysis operation is transferred to the execution node through the overteer, the execution node splits the operation into a plurality of tasks, and divides the enterprise ID list needing to be counted into groups according to the fragment rule, the execution node is responsible for sending the tasks and the enterprise ID group to the leader of the corresponding board, and the board leader is responsible for selecting the copy with less load to execute the tasks. And after all tasks are executed, the execution node collects all calculation results, combines the results and returns the results to the service node. And if the operation needs to access the data for multiple times, the execution node performs task splitting and result merging again according to the operation requirement after collecting the merging result until the operation is finished. The task of finding less loaded compute nodes or shard copies is performed by a load sharing module (described below).
The specific implementation manner of the three steps of the statistical analysis can be implemented by referring to the corresponding content of S2 in the first embodiment.
And the statistical task load sharing module is used for being matched with the cluster management module and the statistical analysis query module for use. In some embodiments, the task related to load sharing may also be performed during the statistical analysis as part of the statistical analysis query module. Load sharing is divided into two cases: and based on the load sharing of the operation and the load sharing of the task, selecting a computing node for running the statistical analysis operation based on the load sharing of the operation, and selecting a copy of the shard for running the computing task based on the load sharing of the task.
The load sharing plays an important role in improving the utilization rate of equipment, the main work content is to monitor effective computing resources and task allocation conditions, and to select idle computing resources for tasks applying for the computing resources.
In the execution process of the statistical analysis query module, as described above, the following two load sharing sub-steps are mainly included:
the first is to select execution nodes and run copies of tasks. The part can be independently integrated into a selected submodule and independently used, so that code multiplexing is improved.
The second method is that the overviewer extracts the task quantity of the computing nodes from each computing node, and the board leader extracts the task quantity of each copy of the board from each copy. And finally, selecting the operation resources according to the selection strategy for selecting the operation resources with the minimum task quantity.
As a further improvement, the system may further include the following modules:
and an equipment fault adjusting module. The impact of equipment failure can be divided into three aspects: the impact on cluster state, the impact on statistical analysis, and the impact on data storage. The equipment fault adjustment module can therefore be used to perform the following three steps:
1. effect on Cluster status
When the computing node and the cores are lost, the overserver and the board leader are reselected. The election mode is automatic election, and codes do not need to be written;
2. influence on statistical analysis
If the computing node where the statistical analysis operation is located is in fault, the service node triggers the statistical analysis process again;
if the computing node where the subtask is located fails, the computing node where the job is located triggers the task again;
3. impact on data storage
No response is made to the data store.
The device adds an adjustment module. There are two cases of adding devices: flash and new device joining. For the flash situation, the device addition adjusting module is configured to add a timeout limit, only after the cluster state change is overtime, the cluster state change (excluding overserver and board leader) triggers the change of data storage (the reconnected device is added as a new device, and the original data is deleted), and if the cluster state is recovered within the timeout period (excluding overserver and board leader), the change of data storage is not performed. During the time-out calculation, the update operation of the data is not allowed.
The addition of new equipment may have two consequences: and copying and migrating the shard copy, and re-slicing. In both cases, the migration is performed in a manner that the target pulls from the source by taking the bucket as a unit and adopting a json string format. During data change, the data updating operation is not allowed, but the normal statistical analysis function cannot be influenced. Migration work of data (including replication and re-fragmentation) is hosted by overviewer.
Replication of Shard copy: the copy is carried out by taking the bucket as a unit, and the copy is finished only when the whole copy is completely migrated, so as to prevent the source from failing in the process of copy or transfer.
And (4) re-slicing: and (4) transferring by taking the barrel as a unit, and modifying mapping information from the barrel to the board after transferring one barrel, so as to mark that the information of one barrel is successfully re-sliced. Re-fragmentation occurs only when the number of Shard copies all satisfy the specified number of copies and a sufficient number of cores remain, and cannot occur simultaneously with the copying and migration of the Shard copies.
In the interface design process of each module, the embodiment of the invention considers the hierarchy of a distributed system. The cluster of compute nodes contains three distinct levels: a compute node layer, a Core layer, statics, and a DataStore layer. The computing node layer is responsible for maintaining cluster topology, maintaining fragmentation rules, fragmenting data, copying and migrating data, fragmenting again, decomposing and summarizing results of statistical analysis operation and the like. The Core layer is responsible for consistency of data updates for each Shard. Statics is responsible for statistical analysis of individual shrd copies, while DataStore is responsible for storage of the data of the Shard copy.
Load sharing is divided into two levels: and calculating the load sharing of a node layer and the load sharing of a Core layer, wherein the two load sharing only have different processing contents. The consistency among a plurality of copies of the same shredd on the Core layer adopts a two-stage submission protocol to ensure the atomicity of data operation, the Core of the Shard reader serves as a coordinator, and each Core of the shredd copies serves as a participating unit. State management, leader election and the like of the whole cluster are realized based on the zookeeper cluster.
Therefore, when designing the interface, the interface can be designed according to the above-mentioned hierarchy and referring to the data exchange relationship among the modules.
The configuration method of the distributed statistical analysis system and the distributed statistical analysis system provided by the invention are described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the core concepts of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (6)
1. A configuration method of a distributed statistical analysis system, wherein the distributed statistical analysis system comprises a ZooKeeper cluster, a service node and a computing node cluster, and is characterized in that the method comprises the following steps:
electing a leader node in the computing node cluster, fragmenting data in each computing node according to a data fragmentation principle, and electing a leader fragment from copies of the data fragments;
after receiving the statistical analysis request, the service node applies for a computing node from the leader node, and the leader node feeds back the computing node with the minimum task amount to the service node; after obtaining the feedback computing node, the service node sends a statistical request to the computing node; the computing node searches the leader fragment and applies for the leader fragment to obtain an idle data fragment copy, and distributes the statistical task to the data fragment copy to execute the statistical task;
under the condition of adjusting the number of data fragments or the fragment rule, remapping the relationship between the data and the data fragments;
data migration is divided into two cases: copying a single copy of the fragment and modifying the number of the fragment to re-fragment; in the process of data migration, the updating operation of the data is not allowed;
when a copy is copied, if equipment failure occurs or new equipment is added, if data needs to be synchronized from the leader fragment, a strategy of copying the whole copy is adopted;
the method for the computing node to search the leader fragment and apply for the leader fragment to obtain an idle data fragment copy, and the step of distributing the statistical task to the data fragment copy to execute the statistical task comprises the following steps:
the computing nodes executing the computing tasks divide the statistical operation into a plurality of tasks, and group ID lists needing to be counted according to a fragment rule, the computing nodes send the tasks and each group of IDs to corresponding leader fragments, the leader fragments are responsible for selecting and handing over idle-loaded data fragment copies to execute the tasks, and after all the tasks are executed, the executing nodes collect all the computing results and combine the results, and then return the results to the service nodes; and if the statistical operation needs to access data for multiple times, the execution node performs task splitting and result merging again according to the requirement of the statistical operation after collecting the merging result until the statistical operation is finished.
2. The method of configuring a distributed statistical analysis system according to claim 1, further comprising:
and the service node sends the updated data to the leader node, and the leader node finds all the copies of the corresponding data fragments and synchronizes the data to all the copies of the data fragments.
3. The method of claim 2, wherein the data synchronization employs a two-phase commit protocol, wherein if all are successful, a success is returned, and if any one copy fails, a rollback and a failure are returned.
4. A distributed statistical analysis system comprises a ZooKeeper cluster, a service node and a computing node cluster, and is characterized by further comprising:
the cluster management module is used for electing a leader node in the computing node cluster, fragmenting data in each computing node according to a data fragmentation principle, and electing a leader fragment in a copy of the data fragments;
the statistical analysis module is used for applying for a computing node from the leader node after the service node receives the statistical analysis request, and the leader node feeds back the computing node with the minimum task quantity to the service node; after obtaining the feedback computing node, the service node sends a statistical request to the computing node; the computing node searches the leader fragment and applies for the leader fragment to obtain an idle data fragment copy, and distributes the statistical task to the data fragment copy to execute the statistical task;
under the condition of adjusting the number of data fragments or the fragment rule, remapping the relationship between the data and the data fragments;
data migration is divided into two cases: copying a single copy of the fragment and modifying the number of the fragment to re-fragment; in the process of data migration, the updating operation of the data is not allowed;
when a copy is copied, if equipment failure occurs or new equipment is added, if data needs to be synchronized from the leader fragment, a strategy of copying the whole copy is adopted;
the method for the computing node to search the leader fragment and apply for the leader fragment to obtain an idle data fragment copy, and the step of distributing the statistical task to the data fragment copy to execute the statistical task comprises the following steps:
the computing nodes executing the computing tasks divide the statistical operation into a plurality of tasks, and group ID lists needing to be counted according to a fragment rule, the computing nodes send the tasks and each group of IDs to corresponding leader fragments, the leader fragments are responsible for selecting and handing over idle-loaded data fragment copies to execute the tasks, and after all the tasks are executed, the executing nodes collect all the computing results and combine the results, and then return the results to the service nodes; and if the statistical operation needs to access data for multiple times, the execution node performs task splitting and result merging again according to the requirement of the statistical operation after collecting the merging result until the statistical operation is finished.
5. The distributed statistical analysis system of claim 4, further comprising:
and the data storage module is used for sending the updated data to the leader node by the service node, and the leader node finds all the copies of the corresponding data fragments and synchronizes the data to all the copies of the data fragments.
6. The distributed statistical analysis system of claim 5, wherein the data synchronization employs a two-phase commit protocol, returning success if all succeeds, and rolling back and returning failure if any copy fails.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710105031.7A CN106936899B (en) | 2017-02-25 | 2017-02-25 | Configuration method of distributed statistical analysis system and distributed statistical analysis system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710105031.7A CN106936899B (en) | 2017-02-25 | 2017-02-25 | Configuration method of distributed statistical analysis system and distributed statistical analysis system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106936899A CN106936899A (en) | 2017-07-07 |
| CN106936899B true CN106936899B (en) | 2021-02-05 |
Family
ID=59424728
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710105031.7A Expired - Fee Related CN106936899B (en) | 2017-02-25 | 2017-02-25 | Configuration method of distributed statistical analysis system and distributed statistical analysis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106936899B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107231437A (en) * | 2017-07-18 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of task backup management method and device |
| CN109951508B (en) * | 2017-12-21 | 2022-02-25 | 中国移动通信集团重庆有限公司 | Zookeeper-based request coordination method and device, computer equipment and storage medium |
| CN108572845B (en) * | 2018-03-15 | 2022-05-31 | 华为技术有限公司 | Upgrading method of distributed micro-service cluster and related system |
| CN108829709A (en) * | 2018-05-03 | 2018-11-16 | 广州金山安全管理系统技术有限公司 | Distributed database management method, apparatus, storage medium and processor |
| CN108737543B (en) * | 2018-05-21 | 2021-09-24 | 高新兴智联科技有限公司 | Distributed Internet of things middleware and working method |
| CN109214132B (en) * | 2018-10-30 | 2023-06-30 | 中国运载火箭技术研究院 | LVC simulation-oriented uncoupled streaming type large-flux asynchronous task processing system |
| CN110209910B (en) * | 2019-05-20 | 2021-06-04 | 无线生活（杭州）信息科技有限公司 | Index switching scheduling method and scheduling device |
| CN110597809B (en) * | 2019-08-21 | 2023-05-23 | 中山大学 | Consistency algorithm system supporting tree-like data structure and implementation method thereof |
| CN112765262B (en) * | 2019-11-05 | 2023-02-28 | 金篆信科有限责任公司 | Data redistribution method, electronic equipment and storage medium |
| CN110933107A (en) * | 2019-12-16 | 2020-03-27 | 许昌学院 | Configuration method of distributed statistical analysis system and distributed statistical analysis system |
| CN111737018B (en) * | 2020-08-26 | 2020-12-22 | 腾讯科技（深圳）有限公司 | ZooKeeper configuration file storage processing method, device, equipment and medium thereof |
| CN113268472B (en) * | 2021-07-15 | 2021-10-12 | 北京华品博睿网络技术有限公司 | Distributed data storage system and method |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101504663A (en) * | 2009-03-17 | 2009-08-12 | 北京大学 | Swarm intelligence based spatial data copy self-adapting distribution method |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101697526A (en) * | 2009-10-10 | 2010-04-21 | 中国科学技术大学 | Method and system for load balancing of metadata management in distributed file system |
| US9805108B2 (en) * | 2010-12-23 | 2017-10-31 | Mongodb, Inc. | Large distributed database clustering systems and methods |
| CN103023970B (en) * | 2012-11-15 | 2015-07-22 | 中国科学院计算机网络信息中心 | Method and system for storing mass data of Internet of Things (IoT) |
| CN103258036A (en) * | 2013-05-15 | 2013-08-21 | 广州一呼百应网络技术有限公司 | Distributed real-time search engine based on p2p |
| CN103414761B (en) * | 2013-07-23 | 2017-02-08 | 北京工业大学 | Mobile terminal cloud resource scheduling method based on Hadoop framework |
| CN103929500A (en) * | 2014-05-06 | 2014-07-16 | 刘跃 | Method for data fragmentation of distributed storage system |
| CN104008152B (en) * | 2014-05-21 | 2017-12-01 | 华南理工大学 | Support the framework method of the distributed file system of mass data access |
| US9767130B2 (en) * | 2014-12-31 | 2017-09-19 | Nexenta Systems, Inc. | Methods and systems for key sharding of objects stored in distributed storage system |
| US11030171B2 (en) * | 2015-01-09 | 2021-06-08 | Ariba, Inc. | Elastic sharding of data in a multi-tenant cloud |
| CN105550229B (en) * | 2015-12-07 | 2019-05-03 | 北京奇虎科技有限公司 | The method and apparatus of distributed memory system data reparation |
-
2017
- 2017-02-25 CN CN201710105031.7A patent/CN106936899B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101504663A (en) * | 2009-03-17 | 2009-08-12 | 北京大学 | Swarm intelligence based spatial data copy self-adapting distribution method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106936899A (en) | 2017-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106936899B (en) | Configuration method of distributed statistical analysis system and distributed statistical analysis system | |
| US10795905B2 (en) | Data stream ingestion and persistence techniques | |
| US10691716B2 (en) | Dynamic partitioning techniques for data streams | |
| CA2929777C (en) | Managed service for acquisition, storage and consumption of large-scale data streams | |
| EP3069228B1 (en) | Partition-based data stream processing framework | |
| EP3069495B1 (en) | Client-configurable security options for data streams | |
| US9460185B2 (en) | Storage device selection for database partition replicas | |
| CA2930026C (en) | Data stream ingestion and persistence techniques | |
| CN102253869A (en) | Scaleable fault-tolerant metadata service | |
| TW201229795A (en) | Web service patterns for globally distributed service fabric | |
| US10712964B2 (en) | Pre-forking replicas for efficient scaling of a distributed data storage system | |
| US10747739B1 (en) | Implicit checkpoint for generating a secondary index of a table | |
| CN114448983A (en) | ZooKeeper-based distributed data exchange method | |
| US20070276945A1 (en) | Fault-Tolerant Resource Committal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210205 |