CN104717294A - Data extracting method, main server and cluster - Google Patents
Data extracting method, main server and cluster Download PDFInfo
- Publication number
- CN104717294A CN104717294A CN201510127956.2A CN201510127956A CN104717294A CN 104717294 A CN104717294 A CN 104717294A CN 201510127956 A CN201510127956 A CN 201510127956A CN 104717294 A CN104717294 A CN 104717294A
- Authority
- CN
- China
- Prior art keywords
- data
- child servers
- master server
- treatment step
- cloud platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data extracting method, a main server and a data conversion cluster. The method includes the steps that the data conversion cluster including the main server and multiple secondary servers is built, the main server in the data conversion cluster divides primary processing of the data to be extracted into two or more processing steps and sends the information of the divided processing steps to the multiple secondary servers in the data conversion cluster in a configuration file mode to trigger the secondary servers to execute respective corresponding processing steps, and the data subjected to primary processing are triggered to be extracted to a cloud platform. By means of the scheme, the data extraction efficiency can be improved.
Description
Technical field
The present invention relates to network communication technology field, particularly a kind of data pick-up method, master server and data transaction cluster.
Background technology
Along with the development of cloud computing technology, cloud computing technology constantly lands the mainstay becoming and support every profession and trade Information Technology Development.Usually, user is needed to be preserved in the database in cloud platform by the data pick-up in local system, such as, traditional operation system is structured on relevant database mostly, can by the data batchmove in the relevant database of user this locality in cloud database.After this, then can realize the various services based on cloud platform, such as, user shares the data etc. in the database of cloud platform.
Being drawn into the process of cloud platform by data from user's local data base, how to realize data extraction process more efficiently, then become an important problem.
Summary of the invention
The invention provides a kind of data pick-up method, master server and data transaction cluster, more efficiently can realize data pick-up.
A kind of data pick-up method, sets up the data transaction cluster comprising master server and multiple child servers; Comprise:
The first process treating extracted data is decomposed into plural treatment step by the master server in described data transaction cluster;
The information of each treatment step decomposed is sent to each child servers in described data transaction cluster by described master server with the form of configuration file, perform each self-corresponding treatment step respectively to trigger this each child servers, and trigger the data pick-up after having processed through first in cloud platform.
Described first is treated to and can walks abreast and without the need to carrying out the process merged;
Data pick-up after described triggering has processed through first comprises to cloud platform: trigger each child servers described and be directly directly sent in cloud platform by the data after having processed separately;
Or,
Described first is treated to and can walks abreast and need the process carrying out merging; Described by treat extracted data first process be decomposed into plural treatment step after, comprise further: determine the fractionation relation between each treatment step;
Data pick-up after described triggering has processed through first comprises to cloud platform: trigger each child servers described and respective result is beamed back described master server, described master server is according to the fractionation relation between each treatment step described, each result received is integrated, the data after integrating are sent in cloud platform.
Describedly to walk abreast and process without the need to carrying out merging comprises: warehouse-in process.
The method comprises further:
Master server in described data transaction cluster receives the dynamic registration request of child servers, at interval of the polling cycle of presetting, master server monitors whether child servers is in effective status, and according to Query Result, upgrade the child servers be in described data transaction cluster.
Master server in described data transaction cluster comprises before the process treating extracted data is decomposed into plural treatment step further:
Described master server judges current the need of carrying out large data sets group process according to preset strategy, if, then continue processing treat extracted data first described in performing and be decomposed into plural treatment step, otherwise, directly by master server, data to be extracted are processed, and be sent in cloud platform.
A kind of master server, is arranged in data transaction cluster, comprises:
Resolving cell, for being decomposed into plural treatment step by the first process treating extracted data;
Parallel processing element, for the information of each treatment step decomposed to be sent to each child servers in described data transaction cluster with the form of configuration file, perform each self-corresponding treatment step respectively to trigger this each child servers, and trigger the data pick-up after having processed through first in cloud platform.
Described first is treated to and can walks abreast and without the need to carrying out the process merged;
Described parallel processing element comprises the first triggers unit, is directly directly sent in cloud platform by the data after having processed separately for triggering each child servers described;
Or,
Described first is treated to and can walks abreast and need the process carrying out merging;
Described resolving cell, is further used for the fractionation relation determined between each treatment step;
Described parallel processing element comprises the second triggers unit and merging treatment subelement, wherein,
Second triggers unit, beams back respective result for triggering each child servers described;
Described merging treatment subelement, according to the fractionation relation between each treatment step described, integrates each result received, and the data after integrating is sent in cloud platform.
Comprise further: updating block, for receiving the dynamic registration request of described child servers, monitor whether child servers is in effective status at interval of the polling cycle of presetting, and according to Query Result, upgrade the child servers be in described data transaction cluster.
A kind of data transaction cluster, comprising: multiple child servers and above-mentioned arbitrary master server, wherein,
Each child servers, for performing each self-corresponding treatment step respectively.
Each child servers, is further used for directly being sent in cloud platform by the data after having processed; Or, be further used for result to beam back described master server;
Described master server, is further used for, after receiving each result that child servers returns, according to the fractionation relation between each treatment step described, integrating each result received, the data after integrating is sent in cloud platform.
Embodiments provide a kind of data pick-up method, master server and data transaction cluster, by setting up the data transaction cluster that deal with data extracts, and utilize the cooperation of master server in this cluster and child servers, namely a process is decomposed into multiple treatment step by master server, by the plurality for the treatment of step of each child servers parallel processing in cluster, thus more efficiently can realize data pick-up.
Accompanying drawing explanation
Fig. 1 is the flow chart of data pick-up method in one embodiment of the invention.
Fig. 2 is the flow chart of data pick-up method in another embodiment of the present invention.
Fig. 3 is the structural representation of master server in one embodiment of the invention.
Fig. 4 is the composition schematic diagram of data transaction cluster in one embodiment of the invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described.Obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
One embodiment of the invention proposes a kind of data pick-up method, see Fig. 1, comprising:
Step 101: set up the data transaction cluster comprising master server and multiple child servers.
Step 102: the first process treating extracted data is decomposed into plural treatment step by the master server in data transaction cluster.
Step 103: the information of each treatment step decomposed is sent to each child servers in described data transaction cluster by master server with the form of configuration file, perform each self-corresponding treatment step respectively to trigger this each child servers, and trigger the data pick-up after having processed through first in cloud platform.
Embodiments provide a kind of data pick-up method, master server and data transaction cluster, by setting up the data transaction cluster that deal with data extracts, and utilize the cooperation of master server in this cluster and child servers, namely master server is decomposed into multiple treatment step by processing arbitrarily, by the plurality for the treatment of step of each child servers parallel processing in cluster, thus can more efficiently realize data from database, be drawn into cloud platform.
In an embodiment of the invention, master server can will be able to split in parallel processing and process without the need to carrying out follow-up union operation, such as, using warehouse-in process as the above-mentioned first process, the data of cloud platform to be sent to are split according to predetermined ratio, distribute to different child servers, the data after having processed separately are then directly put in storage and are sent in cloud platform by each child servers respectively.
In an embodiment of the invention, master server can will can walk abreast and need the process carrying out follow-up merging to split; Like this, step 102, after the first process treating extracted data is decomposed into plural treatment step by master server, may further include: master server determines the fractionation relation between each treatment step; Correspondingly, in step 103, data pick-up after triggering has processed through first comprises to cloud platform: trigger each child servers described and respective result is beamed back described master server, described master server is according to the fractionation relation between each treatment step described, each result received is integrated, the data after integrating are sent in cloud platform.
In an embodiment of the invention, dynamically updating of data transaction cluster can also be realized, specifically comprise: the master server in data transaction cluster receives the dynamic registration request of child servers, at interval of the polling cycle of presetting, master server monitors whether child servers is in effective status, and according to Query Result, upgrade the child servers be in described data transaction cluster.
In an embodiment of the invention, before step 102, can further include: master server judges current the need of carrying out large data sets group process according to preset strategy, if, then continue processing treat extracted data first described in performing and be decomposed into plural treatment step, otherwise, directly by master server, data to be extracted are processed, and are sent in cloud platform.
Another embodiment of the present invention it is also proposed a kind of data pick-up method, and see Fig. 2, the method comprises:
Step 201: set up the data transaction cluster comprising master server and multiple child servers in advance.
This step is in order to data pick-up sets up special cluster.
Step 202: be the master server in data transaction cluster and multiple child servers allocate communications port.
Step 203: the master server in data transaction cluster obtains data to be extracted.
Step 204: master server judges current the need of carrying out large data sets group process according to preset strategy, if so, then performs step 206, otherwise, perform step 205.
Step 205: directly processed data to be extracted by master server, and be sent in cloud platform, terminates current process.
Step 206: master server treats extracted data analysis, wherein will be decomposed into plural treatment step to one or more process of these data to be extracted respectively.
Such as, by the warehouse-in process company of being decomposed into an above treatment step, thus a part for the treatment of step of this warehouse-in process can be performed separately by the multiple child servers in data transaction cluster, thus improve the efficiency of warehouse-in process.
Step 207: master server utilizes the communication port of each child servers in the data transaction cluster distributed, sends to each child servers in described data transaction cluster with the form of configuration file by the information of each treatment step decomposed.
Step 208: each child servers, according to the configuration file received, performs each self-corresponding treatment step respectively.
Step 209: the data after having processed separately are directly sent in cloud platform by each child servers respectively.
In this step 209, if this treatment step needs follow-up union operation, then child servers is according to the back information carried in configuration file, respective result is beamed back described master server, described master server is according to the fractionation relation between each treatment step predetermined, each result received is integrated, the data after integrating are sent in cloud platform.
One embodiment of the invention proposes a kind of master server, and see Fig. 3, this master server is arranged in data transaction cluster, comprising:
Resolving cell 301, for being decomposed into plural treatment step by the first process treating extracted data;
Parallel processing element 302, to send to each child servers in described data transaction cluster with the form of configuration file for the information of each treatment step of being decomposed by resolving cell 301, perform each self-corresponding treatment step respectively to trigger this each child servers, and trigger the data pick-up after having processed through first in cloud platform.
In an embodiment of the invention, described first be treated to and can walk abreast and without the need to carrying out the process merged;
Described parallel processing element 302 comprises the first triggers unit, directly the data after having processed separately directly is sent in cloud platform for triggering each child servers described.
In an embodiment of the invention, described first be treated to and can walk abreast and need the process carrying out merging;
Described resolving cell 301, is further used for the fractionation relation determined between each treatment step;
Described parallel processing element 302 comprises the second triggers unit and merging treatment subelement, wherein,
Second triggers unit, beams back respective result for triggering each child servers described;
Described merging treatment subelement, according to the fractionation relation between each treatment step described, integrates each result received, and the data after integrating is sent in cloud platform.
In an embodiment of the invention, master server may further include: updating block (not shown), for receiving the dynamic registration request of described child servers, monitor whether child servers is in effective status at interval of the polling cycle of presetting, and according to Query Result, upgrade the child servers be in described data transaction cluster.
One embodiment of the invention proposes a kind of data transaction cluster, see Fig. 4, comprising: the master server 402 that multiple child servers 401 and above-mentioned any embodiment propose, wherein,
Each child servers 401, for performing each self-corresponding treatment step respectively.
In an embodiment of the invention, each child servers 401, is further used for directly being sent in cloud platform by the data after having processed; Or, be further used for result being beamed back described master server 402;
Described master server 402, be further used for after receiving each result that child servers 401 returns, according to the fractionation relation between each treatment step described, each result received is integrated, the data after integrating are sent in cloud platform.
Each embodiment of the present invention at least has following beneficial effect:
1, by setting up the data transaction cluster that deal with data extracts, and utilize the cooperation of master server in this cluster and child servers, namely a process is decomposed into multiple treatment step by master server, by the plurality for the treatment of step of each child servers parallel processing in cluster, thus more efficiently can realize data pick-up.
2, can dynamically update each server in data transaction cluster, such as, master server receives child servers can dynamic registration, and master server goes to monitor whether child servers is in effective status, achieves the dynamic expansion of data transaction cluster at interval of 30s.
It should be noted that, in this article, such as term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element " being comprised " limited by statement, and be not precluded within process, method, article or the equipment comprising described key element and also there is other identical factor.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within the scope of protection of the invention.
Claims (10)
1. a data pick-up method, is characterized in that, sets up the data transaction cluster comprising master server and multiple child servers; Comprise:
The first process treating extracted data is decomposed into plural treatment step by the master server in described data transaction cluster;
The information of each treatment step decomposed is sent to each child servers in described data transaction cluster by described master server with the form of configuration file, perform each self-corresponding treatment step respectively to trigger this each child servers, and trigger the data pick-up after having processed through first in cloud platform.
2. method according to claim 1, is characterized in that, described first is treated to and can walks abreast and without the need to carrying out the process merged;
Data pick-up after described triggering has processed through first comprises to cloud platform: trigger each child servers described and be directly directly sent in cloud platform by the data after having processed separately;
Or,
Described first is treated to and can walks abreast and need the process carrying out merging; Described by treat extracted data first process be decomposed into plural treatment step after, comprise further: determine the fractionation relation between each treatment step;
The information of each treatment step decomposed is sent to each child servers in described data transaction cluster by described master server with the form of configuration file, comprise further: result back information is carried in described configuration file by described master server, then this configuration file is sent to each child servers in described data transaction cluster
Data pick-up after described triggering has processed through first comprises to cloud platform: trigger each child servers described according to configuration file, respective result is beamed back described master server, described master server is according to the fractionation relation between each treatment step described, each result received is integrated, the data after integrating are sent in cloud platform.
3. method according to claim 1, is characterized in that, describedly to walk abreast and process without the need to carrying out merging comprises: warehouse-in process.
4. method according to claim 1, is characterized in that, the method comprises further:
Master server in described data transaction cluster receives the dynamic registration request of child servers, at interval of the polling cycle of presetting, master server monitors whether child servers is in effective status, and according to Query Result, upgrade the child servers be in described data transaction cluster.
5. method according to claim 1, is characterized in that, the master server in described data transaction cluster comprises before the process treating extracted data is decomposed into plural treatment step further:
Described master server judges current the need of carrying out large data sets group process according to preset strategy, if, then continue processing treat extracted data first described in performing and be decomposed into plural treatment step, otherwise, directly by master server, data to be extracted are processed, and be sent in cloud platform.
6. a master server, is characterized in that, is arranged in data transaction cluster, comprises:
Resolving cell, for being decomposed into plural treatment step by the first process treating extracted data;
Parallel processing element, for the information of each treatment step decomposed to be sent to each child servers in described data transaction cluster with the form of configuration file, perform each self-corresponding treatment step respectively to trigger this each child servers, and trigger the data pick-up after having processed through first in cloud platform.
7. master server according to claim 6, is characterized in that,
Described first is treated to and can walks abreast and without the need to carrying out the process merged;
Described parallel processing element comprises the first triggers unit, is directly directly sent in cloud platform by the data after having processed separately for triggering each child servers described;
Or,
Described first is treated to and can walks abreast and need the process carrying out merging;
Described resolving cell, is further used for the fractionation relation determined between each treatment step;
Described parallel processing element comprises the second triggers unit and merging treatment subelement, wherein,
Second triggers unit, beams back respective result for triggering each child servers described;
Described merging treatment subelement, according to the fractionation relation between each treatment step described, integrates each result received, and the data after integrating is sent in cloud platform.
8. master server according to claim 6, it is characterized in that, comprise further: updating block, for receiving the dynamic registration request of described child servers, monitor whether child servers is in effective status at interval of the polling cycle of presetting, and according to Query Result, upgrade the child servers be in described data transaction cluster.
9. a data transaction cluster, is characterized in that, comprising: arbitrary described master server in multiple child servers and claim 6 to 8, wherein,
Each child servers, for performing each self-corresponding treatment step respectively.
10. cluster according to claim 9, is characterized in that, each child servers, is further used for directly being sent in cloud platform by the data after having processed; Or, be further used for result to beam back described master server;
Described master server, is further used for, after receiving each result that child servers returns, according to the fractionation relation between each treatment step described, integrating each result received, the data after integrating is sent in cloud platform.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510127956.2A CN104717294A (en) | 2015-03-23 | 2015-03-23 | Data extracting method, main server and cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510127956.2A CN104717294A (en) | 2015-03-23 | 2015-03-23 | Data extracting method, main server and cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104717294A true CN104717294A (en) | 2015-06-17 |
Family
ID=53416242
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510127956.2A Pending CN104717294A (en) | 2015-03-23 | 2015-03-23 | Data extracting method, main server and cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104717294A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105184182A (en) * | 2015-07-22 | 2015-12-23 | 中国科学技术大学苏州研究院 | Private information extraction-based query method for cloud computing private range |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6175876B1 (en) * | 1998-07-09 | 2001-01-16 | International Business Machines Corporation | Mechanism for routing asynchronous state changes in a 3-tier application |
| CN102014169A (en) * | 2010-12-22 | 2011-04-13 | 北京中电普华信息技术有限公司 | Distributed service system as well as distributed service system task execution method and device |
| CN102158533A (en) * | 2011-01-28 | 2011-08-17 | 浙江大学 | Distributed web service selection method based on QoS (Quality of Service) |
| CN102724290A (en) * | 2012-05-23 | 2012-10-10 | 华为技术有限公司 | Method, device and system for getting target customer group |
| CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL all-in-one machine system |
-
2015
- 2015-03-23 CN CN201510127956.2A patent/CN104717294A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6175876B1 (en) * | 1998-07-09 | 2001-01-16 | International Business Machines Corporation | Mechanism for routing asynchronous state changes in a 3-tier application |
| CN102014169A (en) * | 2010-12-22 | 2011-04-13 | 北京中电普华信息技术有限公司 | Distributed service system as well as distributed service system task execution method and device |
| CN102158533A (en) * | 2011-01-28 | 2011-08-17 | 浙江大学 | Distributed web service selection method based on QoS (Quality of Service) |
| CN102724290A (en) * | 2012-05-23 | 2012-10-10 | 华为技术有限公司 | Method, device and system for getting target customer group |
| CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL all-in-one machine system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105184182A (en) * | 2015-07-22 | 2015-12-23 | 中国科学技术大学苏州研究院 | Private information extraction-based query method for cloud computing private range |
| CN105184182B (en) * | 2015-07-22 | 2017-11-24 | 中国科学技术大学苏州研究院 | A kind of querying method of the privately owned scope of cloud computing based on private information extraction |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105450705B (en) | Business data processing method and equipment | |
| CN104580284A (en) | Service assignment device and service assignment method | |
| CN103064745B (en) | A kind of method and system of task matching process | |
| CN105468720A (en) | Method for integrating distributed data processing systems, corresponding systems and data processing method | |
| CN105071994B (en) | A kind of mass data monitoring system | |
| CN107861811B (en) | Task information transmission method and device in workflow system and computer equipment | |
| CN106161485A (en) | Resource regulating method, device and the system of a kind of infrastructure service cluster | |
| CN108111337B (en) | Method and equipment for arbitrating main nodes in distributed system | |
| CN112307105A (en) | Timing task running method, device, equipment and storage medium based on multithreading | |
| CN105376347A (en) | IP address allocation method and system | |
| CN111090519B (en) | Task execution method and device, storage medium and electronic equipment | |
| CN101499022A (en) | Internal memory space releasing system and method | |
| DE60221156D1 (en) | METHOD AND SYSTEM FOR DISTRIBUTING THE WORKLOAD IN A NETWORK OF COMPUTER SYSTEMS | |
| CN106790489B (en) | Parallel data loading method and system | |
| CN107229628A (en) | The method and device of distributed data base pretreatment | |
| CN107315756B (en) | Log processing method and device | |
| CN107277188B (en) | Method, client, server and service system for determining IP address attribution information | |
| CN104717294A (en) | Data extracting method, main server and cluster | |
| CN110909060A (en) | Data transmission method and system | |
| CN104615778A (en) | Method, device and system for avoiding re-extracting data | |
| CN111309397B (en) | Data distribution method, device, server and storage medium | |
| CN105740054A (en) | Virtual machine management method and device | |
| CN102025534A (en) | Single-plate resource allocation method and device thereof | |
| CN104793924A (en) | Calculation mission processing method and device | |
| CN104852858B (en) | A kind of flow forwarding method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150617 |
|
| WD01 | Invention patent application deemed withdrawn after publication |