CN105243125B - Operation method, device, cluster and the inquiry data method and device of PrestoDB clusters - Google Patents

Operation method, device, cluster and the inquiry data method and device of PrestoDB clusters Download PDF

Info

Publication number
CN105243125B
CN105243125B CN201510633927.3A CN201510633927A CN105243125B CN 105243125 B CN105243125 B CN 105243125B CN 201510633927 A CN201510633927 A CN 201510633927A CN 105243125 B CN105243125 B CN 105243125B
Authority
CN
China
Prior art keywords
survival
coordinator node
node
clusters
coordinator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201510633927.3A
Other languages
Chinese (zh)
Other versions
CN105243125A (en
Inventor
吕信
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN201510633927.3A priority Critical patent/CN105243125B/en
Publication of CN105243125A publication Critical patent/CN105243125A/en
Application granted granted Critical
Publication of CN105243125B publication Critical patent/CN105243125B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33Querying
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network

Abstract

The invention discloses a kind of operation method of PrestoDB clusters, device, cluster and inquiry data method and devices.The operation method of PrestoDB clusters includes:ZooKeeper clusters receive respective IP address and the port that at least two coordinator nodes are sent;The coordinator node that the IP address that first is received and port represent is determined as coordinator node of surviving, and the coordinator node other than coordinator node of surviving notifies survival coordinator node to calculate node as spare coordinator node;During inquiry is performed, the current viability of detection survival coordinator node;If detecting that survival coordinator node breaks down, a coordinator node is elected in spare coordinator node, as new survival coordinator node, which is notified to calculate node.The present invention improves the usable efficiency of PrestoDB clusters, realizes the High Availabitity of PrestoDB clusters.

Description

Operation method, device, cluster and the inquiry data method and device of PrestoDB clusters
Technical field
The present embodiments relate to computer technology more particularly to a kind of operation method of PrestoDB clusters, device, The method and apparatus of PrestoDB clusters and PrestoDB clusters inquiry data.
Background technology
Along with the rise of big data, the business datum amount of Internet company rises year by year, therefore major Internet company All carry out big data technology in inside, and data warehouse is built for core business system, data warehouse is divided into two kinds at present Type:Off-line data warehouse and real-time data warehouse.Wherein, the representative products in off-line data warehouse are exactly hive, the product by In bottom Computational frame be MapReduce, therefore be suitable for off-line analysis and the calculating of super large data set, for requirement of real-time Compare high data analysis and calculating is not appropriate for;The representative products of real-time data warehouse are PrestoDB, the product by FaceBook is developed, and employs distributed data calculating and the transmission mode of PipeLine, analysis and calculating for big data It disclosure satisfy that and real-time data analysis and the requirement calculated within -20 minutes, are met at 100 milliseconds.
Since PrestoDB is a distributed computing framework based on memory, carry out data analysis and calculate when It waits, all calculating (Worker) node in PrestoDB clusters carries out practical data processing and calculating, and coordinates (Coordinator) node is substantially carried out carrying out heartbeat detection and each between the scheduling of query task and each node The state and information of the calculating task run on Worker nodes summarize and count.Therefore Coordinator nodes are the equal of The management node of entire PrestoDB clusters, for all Worker nodes under the overall leadership and inquiry and calculating task.
In the prior art, PrestoDB clusters can only specify a Coordinator node, may result in this way There are Single Point of Faliures for Coordinator nodes:Once hardware fault occurs for the server where Coordinator nodes, then need Using PrestoDB clusters stop service, modification PrestoDB cluster configuration files reassign a new server as Coordinator nodes, restart cluster.PrestoDB clusters just can be subnormal again after above-mentioned a series of operation Service is provided, but operation can cause PrestoDB clusters to be off providing service within a period of time in this way, reduce The usable efficiency of PrestoDB clusters.
Invention content
In view of this, the embodiment of the present invention provides a kind of operation method of PrestoDB clusters, device, PrestoDB clusters And the method and apparatus of PrestoDB clusters inquiry data, to improve the usable efficiency of PrestoDB clusters.
In a first aspect, an embodiment of the present invention provides a kind of operation method of PrestoDB clusters, the method includes:
ZooKeeper clusters receive respective IP address and the port that at least two coordinator nodes are sent;
The coordinator node that the IP address and port that ZooKeeper clusters receive first represent is determined as survival and coordinates Node, the coordinator node other than the survival coordinator node as spare coordinator node, by the survival coordinator node notify to Calculate node accepts querying command by the survival coordinator node and calculating task is handed down to calculate node;
During inquiry is performed, the current viability of the ZooKeeper clusters detection survival coordinator node;
If ZooKeeper clusters detect that the survival coordinator node breaks down, in the spare coordinator node In elect a coordinator node, as new survival coordinator node, which is notified to calculate node.
Second aspect, the embodiment of the present invention additionally provide a kind of running gear of PrestoDB clusters, and described device includes:
Address receiving module, for receiving respective IP address and the port that at least two coordinator nodes are sent;
Node determining module, the coordinator node represented for the IP address for receiving first and port are determined as surviving Coordinator node, the coordinator node other than the survival coordinator node lead to the survival coordinator node as spare coordinator node Know to calculate node, querying command is accepted by the survival coordinator node and calculating task is handed down to calculate node;
Survival detection module, for during inquiry is performed, detecting the current viability of the survival coordinator node;
Module is elected, if for detecting that the survival coordinator node breaks down, in the spare coordinator node In elect a coordinator node, as new survival coordinator node, which is notified to calculate node.
The third aspect, the embodiment of the present invention additionally provide a kind of PrestoDB clusters, including at least two coordinator nodes, meter Operator node and ZooKeeper clusters;
The ZooKeeper clusters include the running gear of the PrestoDB clusters described in any embodiment of the present invention.
Fourth aspect, the embodiment of the present invention additionally provides a kind of method of PrestoDB clusters inquiry data, using this hair PrestoDB clusters described in bright any embodiment perform, the method includes:
Client specifies IP address and the port of ZooKeeper clusters;
Client receives querying command, and current survival coordinator node is obtained from the ZooKeeper clusters;
The querying command is submitted to the survival coordinator node by client, is looked by the survival coordinator node described Order is ask to be handled to obtain calculating task and calculating task is handed down to calculate node to carry out inquiry calculating;
Client obtains query result from the survival coordinator node.
5th aspect, the embodiment of the present invention additionally provide a kind of device of PrestoDB clusters inquiry data, described device Including:
Module is specified in address, for specifying the IP address of ZooKeeper clusters and port;
Inquire-receive module for receiving querying command, and obtains current survival association from the ZooKeeper clusters Point of adjustment;
Module is submitted in inquiry, for the querying command to be submitted to the survival coordinator node, is coordinated by the survival Node is handled to obtain calculating task and calculating task is handed down to calculate node to carry out inquiry calculating to the querying command;
As a result acquisition module, for obtaining query result from the survival coordinator node.
Operation method, device, PrestoDB clusters and the PrestoDB of PrestoDB clusters provided in an embodiment of the present invention Cluster inquire data method and apparatus, by be configured in PrestoDB clusters at least two coordinator nodes, calculate node and ZooKeeper clusters elect a coordinator node as survival by ZooKeeper clusters at least two coordinator node Coordinator node, other coordinator nodes detect the survival coordinator node as spare coordinator node, and during inquiry is performed Current viability once detecting that the survival coordinator node breaks down, is then selected again in the spare coordinator node A coordinator node is lifted as new survival coordinator node, PrestoDB clusters is avoided and stops after coordinator node failure Service improves the usable efficiency of PrestoDB clusters, realizes the High Availabitity of PrestoDB clusters.
Description of the drawings
Fig. 1 is a kind of flow chart of the operation method for PrestoDB clusters that the embodiment of the present invention one provides;
Fig. 2 is a kind of structure diagram of the running gear of PrestoDB clusters provided by Embodiment 2 of the present invention;
Fig. 3 is a kind of deployment schematic diagram for PrestoDB clusters that the embodiment of the present invention three provides;
Fig. 4 is the flow chart of the method for a kind of PrestoDB clusters inquiry data that the embodiment of the present invention four provides;
Fig. 5 is the structure diagram of the device of a kind of PrestoDB clusters inquiry data that the embodiment of the present invention five provides.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched The specific embodiment stated is used only for explaining the present invention rather than limitation of the invention.It also should be noted that in order to just Part related to the present invention rather than full content are illustrated only in description, attached drawing.
In embodiments of the present invention, the address of at least two coordinator nodes is specified in the configuration file of PrestoDB clusters With the address of ZooKeeper clusters.When including three servers including two coordinator nodes, ZooKeeper clusters, configuration rule Fan Ru:Coordinator node address=IP address 1:Port 1;IP address 2:Port 2, that is, it is address 1 and address 2 to specify IP address For two nodes as candidate coordinator node, port is respectively port 1 and port 2.ZooKeeper addresses=IP address 1:Port 1;IP address 2:Port 2;IP address 3:Port 3 specifies 3 three IP address 1, IP address 2 and IP address server compositions ZooKeeper clusters, and port 1, port 2 and port 3 are all equal under normal circumstances.Coordinator node specified in this way and Behind the address of ZooKeeper clusters, convenient for the communication between follow-up each node.At least two are specified to coordinate in configuration file The address of node and ZooKeeper clusters can be realized by adding configuration item, that is, add two configuration items, a configuration item For specifying the IP address of at least two coordinator nodes and port, another configuration item is used to specify the IP of ZooKeeper clusters Address and port.
Embodiment one
Fig. 1 be the embodiment of the present invention one provide a kind of PrestoDB clusters operation method flow chart, the present embodiment The situation of election new survival coordinator node when the survival coordinator node in PrestoDB clusters breaks down is applicable to, it should Method can be performed by ZooKeeper clusters, be specifically comprised the following steps:
Step 110, ZooKeeper clusters receive respective IP address and the port that at least two coordinator nodes are sent.
In the present embodiment, PrestoDB clusters include at least two coordinator nodes, multiple calculate nodes and ZooKeeper Cluster.Wherein, at least two coordinator nodes are configured in PrestoDB clusters, the failures such as the machine of delaying occur convenient for survival coordinator node Afterwards, new survival coordinator node is elected from other coordinator node, it is not necessary to stop the service of PrestoDB clusters, so as to protect Demonstrate,prove the high availability of PrestoDB clusters.Above-mentioned work(can be realized in two coordinator nodes of preferred disposition in PrestoDB clusters Energy.
After coordinator node starts, IP address and port to the pocket transmission itself of ZooKeeper collection, ZooKeeper clusters connect After receiving the IP address and port that coordinator node sends, the IP address and port are saved in coordinator node list, are convenient for Subsequently to the election for coordinator node of surviving.
Wherein, ZooKeeper is one distributed, the distributed application program coordination service of open source code, target It is the error-prone key service of packaged complexity, the system of interface and performance efficiency easy to use, function-stable is supplied to User.When disposing ZooKeeper clusters, preferred deployment odd number node, because ZooKeeper clusters are with machine number mistake of delaying Entire cluster partly can just be allowed to be delayed machine.
Step 120, the coordinator node that the IP address and port that ZooKeeper clusters receive first represent is determined as It survives coordinator node, the survival is coordinated section as spare coordinator node by the coordinator node other than the survival coordinator node Point notice accepts querying command by the survival coordinator node and calculating task is handed down to calculate node to calculate node.
When ZooKeeper clusters start election survival coordinator node, the IP address that first is received and port represent Coordinator node be determined as coordinator node of surviving, the coordinator node that subsequently received IP address and port represent is determined as standby With coordinator node, the survival coordinator node is notified to calculate node, querying command is accepted simultaneously by the survival coordinator node Querying command is handled to obtain calculating task, calculating task is handed down to calculate node, is performed by calculate node specific Calculating task.
Step 130, during inquiry is performed, the ZooKeeper clusters detection survival coordinator node is current to be deposited Activity.
During PrestoDB clusters perform inquiry, ZooKeeper clusters pass through the communication with coordinator node of surviving Interaction detects the current viability of the survival coordinator node in real time.
Wherein, the current viability of the ZooKeeper clusters detection survival coordinator node preferably includes:
ZooKeeper clusters receive the status information that the survival coordinator node is sent every setting time;
If reach the state letter that the setting time ZooKeeper clusters are not received by the survival coordinator node Breath, then extend to the second setting time, if reaching second setting time, is not received by the status information, then really The fixed survival coordinator node breaks down.
Survival coordinator node, to the primary status information of oneself of ZooKeeper collection pocket transmission, is reached every setting time During setting time, ZooKeeper clusters receive the status information that survival coordinator node is sent, it is believed that survival coordinator node is Normal work, without failure;When reaching setting time, ZooKeeper clusters are not received by the survival coordinator node hair The status information sent then extends the time of receiving status information to the second setting time, if reaching second setting time, Still without the status information for receiving survival coordinator node transmission, it is determined that the survival coordinator node breaks down.
Step 140, if ZooKeeper clusters detect that the survival coordinator node breaks down, described spare A coordinator node is elected in coordinator node, as new survival coordinator node, by the new survival coordinator node notify to Calculate node.
When ZooKeeper clusters detect that the survival coordinator node machine of delaying occurs when failures, in the spare coordination A coordinator node is elected in node, using the coordinator node as new survival coordinator node, which is coordinated into section Point notice subsequently accepts querying command by the new survival coordinator node and is responsible for meter to the calculate node in PrestoDB clusters Calculation task issues.
The present embodiment receives respective IP address and the end that at least two coordinator nodes are sent by ZooKeeper clusters Mouthful, the coordinator node of the IP address that first is received and port expression is determined as coordinator node of surviving, and the survival is coordinated The survival coordinator node is notified to calculate node, is deposited by described as spare coordinator node by the coordinator node other than node Coordinator node living accepts querying command and calculating task is handed down to calculate node, during inquiry is performed, ZooKeeper The current viability of the cluster detection survival coordinator node, if the survival coordinator node breaks down, in spare association A coordinator node is elected in point of adjustment, as new survival coordinator node, which is notified to meter Operator node once detecting that survival coordinator node breaks down, elects a new survival from spare coordinator node immediately Coordinator node avoids PrestoDB clusters and stops service after coordinator node failure, improves PrestoDB clusters Efficiency can be used, realize the High Availabitity of PrestoDB clusters.
Based on the above technical solution, it after the survival coordinator node that this is new notifies to calculate node, also wraps It includes:
ZooKeeper clusters instruction calculate node forces the calculating unsuccessfully submitted by the survival coordinator node of failure Task.
ZooKeeper clusters notify the new survival coordinator node to meter after new survival coordinator node is elected Operator node, and indicate that calculate node forces what is unsuccessfully submitted by the i.e. preceding survival coordinator node of the survival coordinator node of failure Calculating task continues to execute the calculating task to avoid calculate node, so as to save the memory headroom of calculate node, further carries The usable efficiency of high PrestoDB clusters.
Embodiment two
Fig. 2 is a kind of structure diagram of the running gear of PrestoDB clusters provided by Embodiment 2 of the present invention, such as Fig. 2 Shown, the running gear of the PrestoDB clusters described in the present embodiment is configured in ZooKeeper clusters, including:Address receives Module 210, node determining module 220, survival detection module 230 and election module 240.
Wherein, address receiving module 210 is used to receive respective IP address and the end that at least two coordinator nodes are sent Mouthful;
Node determining module 220 is used for the IP address for receiving first and the coordinator node of port expression is determined as depositing Coordinator node living, the coordinator node other than the survival coordinator node is as spare coordinator node, by the survival coordinator node It notifies to calculate node, querying command is accepted by the survival coordinator node and calculating task is handed down to calculate node;
Detection module 230 of surviving is used for during inquiry is performed, and detects the current survival of the survival coordinator node Property;
If election module 240 is for detecting that the survival coordinator node breaks down, in the spare coordination section A coordinator node is elected in point, as new survival coordinator node, which is notified to save to calculating Point.
Preferably, it further includes:
Indicating module, after notifying to calculate node in the survival coordinator node that this is new, instruction calculate node is strong The calculating task that system is unsuccessfully submitted by the survival coordinator node of failure.
Preferably, the detection module includes:
Receiving unit, for receiving the status information that the survival coordinator node is sent every setting time;
Determination unit, if for reaching the state letter that the setting time is not received by the survival coordinator node Breath, then extend to the second setting time, if reaching second setting time, is not received by the status information, then really The fixed survival coordinator node breaks down.
The said goods can perform the operation method of PrestoDB clusters that any embodiment of the present invention is provided, and have execution The corresponding function module of method and advantageous effect.
Embodiment three
Fig. 3 is a kind of deployment schematic diagram for PrestoDB clusters that the embodiment of the present invention three provides, as shown in figure 3, this reality The PrestoDB clusters described in example are applied to include:At least two coordinator nodes 310, calculate node 320 and ZooKeeper clusters 330.
Wherein, ZooKeeper clusters 330 include the PrestoDB clusters described in any embodiment of the present invention, ZooKeeper Cluster is used for the election survival coordinator node at least two coordinator node, and detects the survival of the survival coordinator node Property, detecting the survival coordinator node failure, then elect a coordinator node as new survival coordinator node, and The new survival coordinator node is notified to calculate node.The ZooKeeper clusters preferably include at least three servers.
When performing inquiry, survival coordinator node carries out heartbeat for dispatching between query task and each calculate node Detect and summarize and count the state and information of the calculating task run in each calculate node;Calculate node is used to carry out data Processing and calculating.
The present embodiment in PrestoDB clusters by being configured at least two coordinator nodes, calculate node and ZooKeeper Cluster can realize the High Availabitity of PrestoDB clusters, improve the usable efficiency of PrestoDB clusters.
Example IV
Fig. 4 be the embodiment of the present invention four provide a kind of PrestoDB clusters inquiry data method flow chart, this reality It applies example and is applicable to situation according to the PrestoDB clusters inquiry data described in any embodiment of the present invention, this method can be by Client performs, and specifically comprises the following steps:
Step 410, client specifies IP address and the port of ZooKeeper clusters.
Client submit inquire when, need IP address and the port of specified ZooKeeper clusters, convenient for subsequently from ZooKeeper clusters get survival coordinator node.
Step 420, client receives querying command, and obtains current survival from the ZooKeeper clusters and coordinate Node.
Client receives querying command input by user, according to the IP address of ZooKeeper clusters and port, from described Current survival coordinator node is obtained in ZooKeeper clusters.
Step 430, the querying command is submitted to the survival coordinator node by client, by the survival coordinator node The querying command is handled to obtain calculating task and calculating task is handed down to calculate node to carry out inquiry calculating.
The querying command is submitted to the survival coordinator node by client, is looked into described in the survival coordinator node parsing It askes order and obtains query execution plan, the query execution stage is produced according to the query execution plan, by the query execution rank Section is separated into multiple calculating tasks, and multiple calculating tasks are handed down to calculate node, are carried out by calculate node according to calculating task Inquiry calculates, and the survival coordinator node is summarized and counted to the inquiry result of calculation of calculate node, obtains query result.
Step 440, client obtains query result from the survival coordinator node.
Client obtains final query result from the survival coordinator node.
The present embodiment specifies IP address and the port of ZooKeeper clusters by client, after querying command is received, Current survival coordinator node is obtained from ZooKeeper clusters, querying command is submitted into the survival coordinator node, by depositing Coordinator node living is handled to obtain calculating task and calculating task is handed down to calculate node to look into the querying command It askes and calculates, obtain query result from the survival coordinator node, realize and data are inquired according to PrestoDB clusters, ensure that The high availability of PrestoDB clusters improves the usable efficiency of PrestoDB clusters.
Embodiment five
Fig. 5 is the structure diagram of the device of a kind of PrestoDB clusters inquiry data that the embodiment of the present invention five provides, As shown in figure 5, the device of the PrestoDB clusters inquiry data described in the present embodiment is configured in client, including:Address refers to Module 530 and result acquisition module 540 are submitted in cover half block 510, inquire-receive module 520, inquiry.
Wherein, module 510 is specified for specifying IP address and the port of ZooKeeper clusters in address;
Inquire-receive module 520 is used to receive querying command, and obtain current survival from the ZooKeeper clusters Coordinator node;
Inquiry submits module 530 to be used to the querying command submitting to the survival coordinator node, is assisted by the survival Point of adjustment is handled to obtain calculating task and calculating task is handed down to calculate node to carry out inquiry meter to the querying command It calculates;
As a result acquisition module 540 is used to obtain query result from the survival coordinator node.
The said goods can perform the method that the embodiment of the present invention four is provided, have the corresponding function module of execution method and Advantageous effect.
Note that it above are only presently preferred embodiments of the present invention and institute's application technology principle.It will be appreciated by those skilled in the art that The present invention is not limited to specific embodiment described here, can carry out for a person skilled in the art various apparent variations, It readjusts and substitutes without departing from protection scope of the present invention.Therefore, although being carried out by above example to the present invention It is described in further detail, but the present invention is not limited only to above example, without departing from the inventive concept, also It can include other more equivalent embodiments, and the scope of the present invention is determined by scope of the appended claims.

Claims (10)

1. a kind of operation method of PrestoDB clusters, which is characterized in that the PrestoDB clusters, which include at least two, to be coordinated Node, multiple calculate nodes and ZooKeeper clusters, the method includes:
ZooKeeper clusters receive respective IP address and the port that at least two coordinator nodes are sent;
The coordinator node that the IP address and port that ZooKeeper clusters receive first represent is determined as survival and coordinates section Point, the coordinator node other than the survival coordinator node notify the survival coordinator node to meter as spare coordinator node Operator node accepts querying command by the survival coordinator node and calculating task is handed down to calculate node;
During inquiry is performed, the current viability of the ZooKeeper clusters detection survival coordinator node;
If ZooKeeper clusters detect that the survival coordinator node breaks down, selected in the spare coordinator node A coordinator node is enumerated, as new survival coordinator node, which is notified to calculate node.
2. it according to the method described in claim 1, it is characterized in that, is notified by the new survival coordinator node to calculate node Later, it further includes:
ZooKeeper clusters instruction calculate node forces the calculating unsuccessfully submitted by the survival coordinator node of failure to be appointed Business.
3. method according to claim 1 or 2, which is characterized in that ZooKeeper clusters detect the survival coordinator node Current viability includes:
ZooKeeper clusters receive the status information that the survival coordinator node is sent every setting time;
If reaching the status information that the setting time ZooKeeper clusters are not received by the survival coordinator node, The second setting time is extended to, if reaching second setting time, is not received by the status information, it is determined that described Coordinator node of surviving breaks down.
4. a kind of running gear of PrestoDB clusters, which is characterized in that the PrestoDB clusters, which include at least two, to be coordinated Node, multiple calculate nodes and ZooKeeper clusters, described device include:
Address receiving module, for receiving respective IP address and the port that at least two coordinator nodes are sent;
Node determining module, the coordinator node represented for the IP address for receiving first and port are determined as survival coordination Node, the coordinator node other than the survival coordinator node as spare coordinator node, by the survival coordinator node notify to Calculate node accepts querying command by the survival coordinator node and calculating task is handed down to calculate node;
Survival detection module, for during inquiry is performed, detecting the current viability of the survival coordinator node;
Module is elected, if for detecting that the survival coordinator node breaks down, is selected in the spare coordinator node A coordinator node is enumerated, as new survival coordinator node, which is notified to calculate node.
5. device according to claim 4, which is characterized in that further include:
Indicating module, after notifying to calculate node in the survival coordinator node that this is new, instruction calculate node forces to lose Lose the calculating task submitted by the survival coordinator node of failure.
6. device according to claim 4 or 5, which is characterized in that the detection module includes:
Receiving unit, for receiving the status information that the survival coordinator node is sent every setting time;
Determination unit, if for reaching the status information that the setting time is not received by the survival coordinator node, The second setting time is extended to, if reaching second setting time, is not received by the status information, it is determined that described Coordinator node of surviving breaks down.
7. a kind of PrestoDB clusters, which is characterized in that including at least two coordinator nodes, calculate node and ZooKeeper collection Group;
The ZooKeeper clusters include the running gear of any PrestoDB clusters of claim 4-6.
8. PrestoDB clusters according to claim 7, which is characterized in that the ZooKeeper clusters include at least three Platform server.
9. a kind of method of PrestoDB clusters inquiry data, is held using the PrestoDB clusters described in claim 7 or 8 Row, which is characterized in that the method includes:
Client specifies IP address and the port of ZooKeeper clusters;
Client receives querying command, and current survival coordinator node is obtained from the ZooKeeper clusters;
The querying command is submitted to the survival coordinator node by client, and the inquiry is ordered by the survival coordinator node Order is handled to obtain calculating task and calculating task is handed down to calculate node to carry out inquiry calculating;
Client obtains query result from the survival coordinator node.
10. a kind of device of PrestoDB clusters inquiry data, which is characterized in that including:
Module is specified in address, for specifying the IP address of ZooKeeper clusters and port;
Inquire-receive module for receiving querying command, and obtains current survival from the ZooKeeper clusters and coordinates section Point;
Module is submitted in inquiry, for the querying command to be submitted to the survival coordinator node, by the survival coordinator node The querying command is handled to obtain calculating task and calculating task is handed down to calculate node to carry out inquiry calculating;
As a result acquisition module, for obtaining query result from the survival coordinator node.
CN201510633927.3A 2015-09-29 2015-09-29 Operation method, device, cluster and the inquiry data method and device of PrestoDB clusters Active CN105243125B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510633927.3A CN105243125B (en) 2015-09-29 2015-09-29 Operation method, device, cluster and the inquiry data method and device of PrestoDB clusters

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510633927.3A CN105243125B (en) 2015-09-29 2015-09-29 Operation method, device, cluster and the inquiry data method and device of PrestoDB clusters

Publications (2)

Publication Number Publication Date
CN105243125A CN105243125A (en) 2016-01-13
CN105243125B true CN105243125B (en) 2018-07-06

Family

ID=55040773

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510633927.3A Active CN105243125B (en) 2015-09-29 2015-09-29 Operation method, device, cluster and the inquiry data method and device of PrestoDB clusters

Country Status (1)

Country Link
CN (1) CN105243125B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107193960B (en) * 2017-05-24 2020-11-10 南京大学 Distributed crawler system and periodic incremental grabbing method
CN107807608A (en) * 2017-11-02 2018-03-16 腾讯科技(深圳)有限公司 Data processing method, data handling system and storage medium
CN111131361B (en) * 2018-10-31 2023-03-24 北京国双科技有限公司 Processing method and device for connection nodes in cluster query system
CN111522649B (en) * 2019-02-01 2022-04-12 北京京东尚科信息技术有限公司 Distributed task allocation method, device and system
CN109977161A (en) * 2019-03-28 2019-07-05 上海中通吉网络技术有限公司 The monitoring system of presto cluster

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977124A (en) * 2010-11-05 2011-02-16 山东中创软件工程股份有限公司 Service clustering method and system based on ZooKeeper technology
CN103188110A (en) * 2013-03-29 2013-07-03 汉柏科技有限公司 Zookeeper-based cloud platform service hot backup method
CN103312809A (en) * 2013-06-24 2013-09-18 北京汉柏科技有限公司 Distributed management method for service in cloud platform
CN103401704A (en) * 2013-07-24 2013-11-20 佳都新太科技股份有限公司 Implementation scheme of distributed log collecting server
CN103729436A (en) * 2013-12-27 2014-04-16 中国科学院信息工程研究所 Distributed metadata management method and system
CN104486108A (en) * 2014-12-08 2015-04-01 畅捷通信息技术股份有限公司 Node configuration method base on Zookeeper and node configuration system based on Zookeeper

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9690671B2 (en) * 2013-11-01 2017-06-27 Cloudera, Inc. Manifest-based snapshots in distributed computing environments

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977124A (en) * 2010-11-05 2011-02-16 山东中创软件工程股份有限公司 Service clustering method and system based on ZooKeeper technology
CN103188110A (en) * 2013-03-29 2013-07-03 汉柏科技有限公司 Zookeeper-based cloud platform service hot backup method
CN103312809A (en) * 2013-06-24 2013-09-18 北京汉柏科技有限公司 Distributed management method for service in cloud platform
CN103401704A (en) * 2013-07-24 2013-11-20 佳都新太科技股份有限公司 Implementation scheme of distributed log collecting server
CN103729436A (en) * 2013-12-27 2014-04-16 中国科学院信息工程研究所 Distributed metadata management method and system
CN104486108A (en) * 2014-12-08 2015-04-01 畅捷通信息技术股份有限公司 Node configuration method base on Zookeeper and node configuration system based on Zookeeper

Also Published As

Publication number Publication date
CN105243125A (en) 2016-01-13

Similar Documents

Publication Publication Date Title
CN105243125B (en) Operation method, device, cluster and the inquiry data method and device of PrestoDB clusters
CN104281496B (en) The distribution method and distributor of virtual resource
CN107547595B (en) Cloud resource scheduling system, method and device
CN105677462A (en) Distributed task system based on internet of things and business processing method
CN108243012B (en) Charging application processing system, method and device in OCS (online charging System)
CN106919445A (en) A kind of method and apparatus of the container of Parallel Scheduling in the cluster
CN105812423B (en) A kind of cloud system configuration method, server and device
CN105162632A (en) Automatic processing system for server cluster failures
CN109614227A (en) Task resource concocting method, device, electronic equipment and computer-readable medium
US20150365311A1 (en) Scalable software monitoring infrastructure, using parallel task queuing, to operate in elastic cloud environments
US20180239761A1 (en) Collision detection using state management of configuration items
CN104484228B (en) Distributed parallel task processing system based on Intelli DSC
CN109359034A (en) A kind of operation system test method, computer readable storage medium and terminal device
CN103326880B (en) Genesys calling system high availability cloud computing monitoring system and method
CN108038126A (en) Method, apparatus, terminal device and storage medium derived from a kind of data
CN114598586B (en) Multi-cloud scene computing power gridding method and system
Mahato et al. Reliability modeling and analysis for deadline-constrained grid service
US8751640B2 (en) System and method for enhancing efficiency and/or efficacy of switchover and/or failover in providing network based services with high availability
CN110247818A (en) A kind of data monitoring method, device, storage medium and server
CN113722141B (en) Method and device for determining delay reason of data task, electronic equipment and medium
Zhang et al. A relationship-based VM placement framework of cloud environment
Wang et al. Research on key technology of edge-node resource scheduling based on linear programming
CN115130911A (en) Method and device for processing delivery tasks failed to execute by robot
CN105245375B (en) A kind of network-based high-efficiency storage method
CN114706675A (en) Task deployment method and device based on cloud edge cooperative system

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant