CN103347053A - Remote coprocessing procedure call method based on pipeline technology - Google Patents
Remote coprocessing procedure call method based on pipeline technology Download PDFInfo
- Publication number
- CN103347053A CN103347053A CN2013102418775A CN201310241877A CN103347053A CN 103347053 A CN103347053 A CN 103347053A CN 2013102418775 A CN2013102418775 A CN 2013102418775A CN 201310241877 A CN201310241877 A CN 201310241877A CN 103347053 A CN103347053 A CN 103347053A
- Authority
- CN
- China
- Prior art keywords
- pipeline
- rpc
- data
- module
- call
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a remote coprocessing procedure call method based on the pipeline technology. The remote coprocessing procedure call method is used for solving the problems that data transmission quantity among nodes in a database cluster is large, and cross-node task execution is interacted and complex. In a database cluster system, remote procedure call types are divided according to processing functions, and each function comprises a corresponding function number and a processing module. The unique pipeline technology is used for transmitting data to solve the problem that cross-node data transmission quantity is large. The unique pipeline technology comprises a pipeline building module, a pipe data transmitting module, a pipeline data receiving module and a pipeline using flow window management module. A pipeline is built and generated according to an invoking function number, a two-end node number and a distributed session number, and partition of the pipeline is determined by union of the session number and the opposite-end node number. According to task characteristics, a plurality of pipelines can be selected simultaneously to call and execute gathering processing results. Data access can be conducted conveniently due to pipeline characteristics, network congestion caused by large data transmission quantity is avoided to flow window management.
Description
Technical field
The present invention relates to a kind of remote collaborative processing procedure call method based on pipe technology, belong to field of computer technology.
Background technology
The inter-node communication of data-base cluster, on physical layer and data link layer is transmission protocol layer, and the mutual level of the communication of application layer and transport layer is the comparison ambiguity always, and method flow is various.The effect function is uneven.In the running of cloud database, many operations are not finished in this locality by a node, need a plurality of nodes to work in coordination with and finish the work.And some are long and complicated operations just need be passed through remote procedure call in the process that task is carried out, executable operations on other nodes, and it not is to settle at one go that task requests is carried out, and the request that has need be carried out feedback according to long-range substep and dynamically adjust follow-up manner of execution.The task of having is carried out and is carried out by different level, and each XM of bottom obtains and the analyzing and processing bottom data obtains the intermediate object program collection, is aggregated into task upper strata analyzing and processing again and realizes that multimachine is concurrent and finish the work.Based on this, we have invented a kind of remote collaborative processing procedure call method based on pipe technology.Realized that data volume is big and the task of complicated operation achieves the goal by remote procedure call.
Summary of the invention
The objective of the invention is to invent a kind of remote collaborative processing procedure call method (hereinafter to be referred as RPC) based on pipe technology, it is big to be used for solving the data-base cluster volume of transmitted data, the problem that mutual complicated task cross-node is carried out.In data base cluster system, the RPC call type is divided according to processing capacity, and each function has corresponding function number and processing module.The RPC caller is according to handled functional module, interpolation function number in initiating to call, pass a parameter to XM by the RPC pipeline, the executor number call according to function that corresponding processing module is executed the task and by pipeline with result transmission to the described promoter of calling.Above-mentioned RPC pipeline is big for solving the cross-node volume of transmitted data, the pipe technology transmission data of the present invention's original creation, and it comprises the pipeline creation module, pipeline data sending module, pipeline data receiver module and pipeline use traffic window administration module.The structure of pipeline is formed and is comprised: the caller node number, and peer node number, session session number, create flag bit, the flow window, data send buffering area, and data receive buffering area, and data are read and are sent skew, wait the transmission thread, wait and read thread etc.
The establishment of pipeline is according to calling function number, and the two ends node number creates to generate for the session of distribution number, and the division of pipeline is by session number, and it is definite that peer node number is united.The present invention carries out the convergence processing result according to the optional majority pipeline of task characteristic concurrent invocation.Pipe characteristic is taken conveniently data, and the reportedly big generation network congestion of throughput rate of factor is avoided in the management of flow window.
The main functional module of the present invention has:
1, RPC calls initiation module
2, RPC calls the pipeline transmission module
3, RPC calling task Executive Module
4, RPC calls Back Administration Module
Wherein RPC calls the pipeline transmission module and comprises: RPC calls the pipeline creation module, and RPC calls the pipeline data sending module, and RPC calls the pipeline data receiver module, and RPC calls pipeline use traffic window administration module.
RPC calling task Executive Module is responsible for task and is carried out.
RPC call that feedback after Back Administration Module mainly be responsible for to be carried out and RPC finish to wither away and abnormal ending after resource discharge cleaning
The objective of the invention is to be achieved through the following technical solutions:
1. initiate to call from the RPC called side and finish to call, mainly experience following steps
S1: the far call promoter calls rpc_begin (peer_nid.func_id) establishment rpc pipeline according to the RPC type of service of self;
S2: streaming parameter is mail to RPC pipeline data buffering area;
S3: refresh RPC pipeline data buffering area and make the data of buffering area flow to the pipeline opposite end;
S4: hang and self wait for that execute the task in the opposite end and feedback result;
S5: read long-range return results from the RPC pipeline, judge to continue to carry out from the S2 step order according to the result and down carry out or carry out the end of S6 step;
S6: call rpc_stop and discharge the RPC conduit resource, finish to call.
2. the task executor mainly experiences following steps
S1: the RPC call request of receiving from the RPC pipeline number is called processing unit according to function and is carried out the RPC request;
S2: execution result is fed back to called side from the RPC pipeline.
In above-mentioned 2 S1 steps, it is that RPC calling task Executive Module is responsible for the task execution that request is handled: read data by the executor from pipeline, processing unit according to the funcall that reads number selection correspondence is executed the task according to parameter, because of the wide range that the RPC remote procedure call is used general, respectively calling will carrying out of task has nothing in common with each other, so with number differentiation of different functions, one of each function correspondence is processing unit independently, called side is called according to function number initiation, execution side carries out in this locality according to the processing unit of function correspondence, and is complete afterwards by pipeline feedback result and error message etc.
Above-mentioned RPC calls the flow window administration module in the pipeline transmission module, and the reportedly big generation network congestion of throughput rate of factor is avoided in management, may further comprise the steps:
Each node of S1 arranges the maximum useful discharge window number according to self-ability and network condition;
S2 uses pipeline transmission data, and detecting available window number is 0 commentaries on classics S3 as the available window number, the commentaries on classics S5 otherwise the utilizable flow window number successively decreases;
The new pipeline transmission data of S3 find that the available window number is 0, block pipeline, enter waiting list;
S4 finds the utilizable flow window, and the first pipeline transmission data in the wake-up waiting formation change 2;
S5 pipeline transmission ED discharges pipeline, and the utilizable flow window number increases progressively one, if former available window is 0 pipeline that changes in the formation of S4 wake-up waiting.
Beneficial effect
The present invention has following beneficial effect:
1, in resource not under the situation in this locality, the cross-node far call is carried out the complex interaction process;
2, by the pipeline streaming data, improve efficiency of transmission greatly, take conveniently: get time spent side, need not then be blocked in the pipeline;
3, the dissimilar RPC on the same working node calls and can concurrently carry out, pipeline utilance height, and result one end convergence processing in multitube road is convenient;
4, the RPC flow is regulated in the design of flow window, increases network utilization, and avoids network congestion.
Embodiment
Further specify technical scheme of the present invention below in conjunction with accompanying drawing, but the content that the present invention protects is not limited to the following stated.
As shown in Figure 1, based on the remote collaborative processing procedure call method (hereinafter to be referred as RPC) of pipe technology, comprise that RPC calls initiator's caller, call the executor of the side of being called with RPC, they are in different nodes and are connected with network.For realizing the execution of task cross-node, comprise following functional module: RPC calls initiation module, and RPC calls the pipeline transmission module, RPC calling task Executive Module, and RPC calls Back Administration Module.
The enforcement of RPC remote procedure calling (PRC) method comprises two aspects of caller and executor:
1.RPC call the promoter, select the RPC remote procedure calling (PRC) method to carry out as required and call, from calling initiation module, initiate to call with pipe method transmission parameter, call release conduit resource when finishing.
The RPC called side finishes to call from initiating to call, and mainly experiences following steps:
S1: the funcall that the far call promoter determine to use according to self RPC type of service number, call rpc_begin (peer_nid.func_id) and create the rpc pipeline;
S2: streaming parameter is mail to RPC pipeline data buffering area;
S3: refresh RPC pipeline data buffering area and make the data of buffering area flow to the pipeline opposite end;
S4: hang and self wait for that execute the task in the opposite end and feedback result;
S5: read long-range return results from the RPC pipeline, judge to continue to carry out from the S2 step order according to the result and down carry out or carry out the end of S6 step;
S6: call rpc_stop and discharge the RPC conduit resource, finish to call.
2. the executor receives RPC teleprocessing request from pipeline, reads parameter information, and in this locality by executing the task request of processing unit, feedback execution result, and carry out subsequent request or execution end process flow process as required.
The executor mainly experiences following steps:
S1: the RPC call request of receiving from the RPC pipeline number is called processing unit according to function and is carried out the RPC request;
S2: execution result is fed back to called side from the RPC pipeline.
In above-mentioned 2 S1 steps, it is that RPC calling task Executive Module is responsible for the task execution that request is handled: read data by the executor from pipeline, processing unit according to the funcall that reads number selection correspondence is executed the task according to parameter, complete error message by occurring in pipeline feedback result and the implementation afterwards.
In conjunction with shown in Figure 1, the RPC remote procedure calling (PRC) method, caller is in two different nodes with the executor, caller is that realization task cross-node is carried out, may further comprise the steps: caller is created the RPC pipeline earlier, pipeline is united definite by session number with other end node number, after creating pipeline, select number initiation of corresponding funcall to call the stream transmission parameter according to the function that realizes, flush data to XM, and waiting for that the executor feeds back execution result, the executor is from pipeline read functions call number and parameter, by the function number corresponding processing unit of selection, analytic parameter, and execute the task in this locality, the error message that runs in execution result and the implementation etc. is fed back to caller by pipeline, caller receives feedback result and judges whether that continuing to transmit parameter continues to carry out to the executor, if continue to carry out then transmit after the parameter etc. to be feedback, otherwise finish the call operation flow process, discharge conduit resource, finish RPC and call.
Claims (7)
1. based on the remote collaborative processing procedure call method (hereinafter to be referred as RPC) of pipe technology, it is the method that a kind of data-base cluster task cross-node far call is carried out, belong to computer software fields, it is characterized in that, it comprises that RPC calls initiation module, data transmission module, RPC task execution module, the RPC call type is divided according to processing capacity, and each function has corresponding function number and processing unit.
Initiate to call according to task function appointed function number 2.RPC call initiation module, the RPC task execution module number is selected corresponding processing unit processes task requests according to function.
3. data transmission module uses unique pipe technology transmission data.
4.RPC remote procedure calling (PRC) method is characterized in that, the described pipe technology that right 1 described data transmission module uses comprises the pipeline creation module, pipeline data sending module, pipeline data receiver module and pipeline use traffic window administration module.
5.RPC remote procedure calling (PRC) method is characterized in that, right 1 described pipe technology transmission data may further comprise the steps:
S1:RPC far call person is according to function number establishment RPC pipeline;
S2:RPC far call person streaming refreshes to the opposite end to RPC pipeline input parameter;
The long-range executor of S3:RPC reads the data based function number corresponding processing unit processes data of selection from pipeline;
The long-range executor of S4:RPC feeds back to RPC far call person with the result by pipeline transmission;
S5:RPC far call person receives feedback, selects continuation alternately or finishes to discharge conduit resource according to feedback.
6.RPC remote procedure calling (PRC) method is characterized in that, right 2 described flow window administration modules avoid factor reportedly the big generation network congestion of throughput rate may further comprise the steps:
Each node of SS1 arranges the maximum useful discharge window number according to self-ability and network condition;
SS2 uses pipeline transmission data, and detecting available window number be 0 commentaries on classics SS3 as the available window number, otherwise the utilizable flow window number successively decreases one, changes SS5;
The new pipeline transmission data of SS3 find that the available window number is 0, block pipeline, enter waiting list;
SS4 finds the utilizable flow window, and the first pipeline transmission data in the wake-up waiting formation are changeed SS2;
SS5 pipeline transmission ED discharges pipeline, and the utilizable flow window number increases progressively one, if former available window is 0 pipeline that changes in the formation of SS4 wake-up waiting.
7.RPC remote procedure calling (PRC) method is characterized in that, can a plurality of pipelines transmits data to same node simultaneously and carries out the data aggregation process, realizes the task execution concurrence.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013102418775A CN103347053A (en) | 2013-06-19 | 2013-06-19 | Remote coprocessing procedure call method based on pipeline technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013102418775A CN103347053A (en) | 2013-06-19 | 2013-06-19 | Remote coprocessing procedure call method based on pipeline technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103347053A true CN103347053A (en) | 2013-10-09 |
Family
ID=49281827
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013102418775A Pending CN103347053A (en) | 2013-06-19 | 2013-06-19 | Remote coprocessing procedure call method based on pipeline technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103347053A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016169374A1 (en) * | 2015-04-23 | 2016-10-27 | 珀金埃尔默企业管理（上海）有限公司 | Proxy framework service system and method |
| CN111221662A (en) * | 2019-10-16 | 2020-06-02 | 贝壳技术有限公司 | Task scheduling method, system and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050071578A1 (en) * | 2003-09-25 | 2005-03-31 | International Business Machines Corporation | System and method for manipulating data with a plurality of processors |
| CN1852209A (en) * | 2006-05-31 | 2006-10-25 | 北京和利时系统工程股份有限公司 | Remote process transfering method and system in distribution type control system |
| CN101901207A (en) * | 2010-07-23 | 2010-12-01 | 中国科学院计算技术研究所 | Operating system of heterogeneous shared storage multiprocessor system and working method thereof |
| CN102681881A (en) * | 2011-03-07 | 2012-09-19 | 中国银联股份有限公司 | Cross-computer scheduling method and system thereof |
| CN103116520A (en) * | 2012-11-02 | 2013-05-22 | 深圳键桥通讯技术股份有限公司 | Remote procedure call remote position control (RPC) method based on transmission control protocol (TCP)/user datagram protocol (UDP) |
-
2013
- 2013-06-19 CN CN2013102418775A patent/CN103347053A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050071578A1 (en) * | 2003-09-25 | 2005-03-31 | International Business Machines Corporation | System and method for manipulating data with a plurality of processors |
| CN1852209A (en) * | 2006-05-31 | 2006-10-25 | 北京和利时系统工程股份有限公司 | Remote process transfering method and system in distribution type control system |
| CN101901207A (en) * | 2010-07-23 | 2010-12-01 | 中国科学院计算技术研究所 | Operating system of heterogeneous shared storage multiprocessor system and working method thereof |
| CN102681881A (en) * | 2011-03-07 | 2012-09-19 | 中国银联股份有限公司 | Cross-computer scheduling method and system thereof |
| CN103116520A (en) * | 2012-11-02 | 2013-05-22 | 深圳键桥通讯技术股份有限公司 | Remote procedure call remote position control (RPC) method based on transmission control protocol (TCP)/user datagram protocol (UDP) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016169374A1 (en) * | 2015-04-23 | 2016-10-27 | 珀金埃尔默企业管理（上海）有限公司 | Proxy framework service system and method |
| CN111221662A (en) * | 2019-10-16 | 2020-06-02 | 贝壳技术有限公司 | Task scheduling method, system and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103294531B (en) | A kind of method for allocating tasks and system | |
| US9703610B2 (en) | Extensible centralized dynamic resource distribution in a clustered data grid | |
| CN106708917B (en) | A kind of data processing method, device and OLAP system | |
| CN103870340A (en) | Data processing method and control node in stream computation system and stream computation system | |
| WO2019001092A1 (en) | Load balancing engine, client, distributed computing system, and load balancing method | |
| CN106027596A (en) | Task distributing method and device | |
| CN109416646B (en) | Optimization method for container allocation and processing equipment | |
| CN103516744A (en) | A data processing method, an application server and an application server cluster | |
| CN103944769A (en) | RPC (Remote Procedure Call) protocol based cluster resource unified management system | |
| CN103023980B (en) | A kind of method and system of cloud platform processes user service request | |
| CN105847891A (en) | Resource preemption method and device | |
| CN110109756A (en) | A kind of network target range construction method, system and storage medium | |
| CN107291536B (en) | Application task flow scheduling method in cloud computing environment | |
| CN103544285A (en) | Data loading method and device | |
| CN102436399A (en) | Load-balanced acquisition method | |
| CN104239555A (en) | MPP (massively parallel processing)-based parallel data mining framework and MPP-based parallel data mining method | |
| CN110187960A (en) | A kind of distributed resource scheduling method and device | |
| CN106657099B (en) | Spark data analysis service publishing system | |
| CN104102533A (en) | Bandwidth aware based Hadoop scheduling method and system | |
| CN105786447A (en) | Method and apparatus for processing data by server and server | |
| CN108280018B (en) | Method and system for analyzing and optimizing communication overhead efficiency of node workflow | |
| CN104112049A (en) | P2P (peer-to-peer) architecture based cross-data-center MapReduce task scheduling system and P2P architecture based cross-data-center MapReduce task scheduling method | |
| CN105610992A (en) | Task allocation load balancing method of distributive flow computing system | |
| CN106161520A (en) | Big market demand platform and exchange method based on it | |
| CN108282526A (en) | Server dynamic allocation method and system between double clusters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20131009 |