CN104243610A - Distributed file transmission service method - Google Patents
Distributed file transmission service method Download PDFInfo
- Publication number
- CN104243610A CN104243610A CN201410516628.7A CN201410516628A CN104243610A CN 104243610 A CN104243610 A CN 104243610A CN 201410516628 A CN201410516628 A CN 201410516628A CN 104243610 A CN104243610 A CN 104243610A
- Authority
- CN
- China
- Prior art keywords
- service
- transfer
- client
- server
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Computer And Data Communications (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a distributed file transmission service method which is used for single or batch uploading and downloading services and aims to solve the bottlenecks of file transmission services in terms of concurrent transmission and transmission efficiency. The method includes: (1) a client application interface; (2), a server transfer service; (3) a server distributed file processing service. The method has the advantages that the concurrent quantity and service efficiency bottlenecks during simultaneous uploading and downloading of large-batch files can be effectively solved; the subscribing service interface and a unified transfer service interface are used, the defect of file transmission service failure due to the fact that communication port number cannot be displayed and acquired under some special application environments is overcome, and services can be performed safely and effectively.
Description
Technical field
In file transfer services, files passe and the concurrent situation of carrying out of download service of batch happen occasionally, and meanwhile, the batch documents sent out in wink is uploaded and download service has higher requirement to file transfer services component in quantity and efficiency of service.In addition, under some application circumstances, client end slogan resource can not explicitly obtain owing to relating to the problems such as information security, this invention is intended to solve the bottleneck problem in quantity and efficiency run into when batch documents upload and download carries out simultaneously, and under solving some special applications, client communication port numbers can not explicit acquisition and cause the problem of serv-fail.
Background technology
In file transfer services, what file transfer services component provided uploads files in batch, download treatable quantity and efficiency have higher requirement, the task exceeding file transfer services component ability to bear even collapses causing the decline of file transfer services component performance, and large batch of file is uploaded and download service scene occurs in the application often simultaneously, therefore, file transfer services component must process that large batch of file is uploaded simultaneously, download service efficiently.
Current Miscellaneous Documents transmission service mostly adopts the storage mode of centralized file management, uploading and downloading of file all needs to same station server application service, due to the finiteness of service end task disposal ability and service provision capacity, file in enormous quantities is uploaded simultaneously, download the blocking that must cause service end task, thus cause the decline of service processing efficiency or service to provide failure, even cause the collapse of service end.
The network communication protocol of current file transfer services mostly depends on ICP/IP protocol, this need for environment has clear and definite communication IP address and port numbers, but under some application circumstances, the communication port numbers of client cannot explicit acquisition, or often change, current file transmission service directly provides the mode of service by explicit IP address and port numbers, can cause the decline of efficiency of service, even cannot provide service.
In sum, centralized file management mode will make file transfer services efficiency reduce, along with the bottleneck of file transfer services arrives fast, for meeting the demand of current large data age batch documents transmission service, file transfer services must improve efficiency of service, support that large batch of file is uploaded, download service simultaneously; In addition, for tackling the application of some particular surroundings, must solve client communication port numbers cannot the problem that often changes of explicit acquisition or port numbers, and non-explicit port obtains and the transmission service of strain is conducive to improving the fail safe of service.Under other applied environment, adopt self-defining communication protocol to provide service, support that multi-communication protocol will be conducive to improving the ambient adaptability of service.
Summary of the invention
The object of this invention is to provide a kind of distributed file transfer services component supporting multi-communication protocol, adopt the processing mode of cloud service, distributedly provide files passe and download service, and solve under some applied environment cannot explicit acquisition client communication port numbers and cause the problem that cannot provide service.
The object of the invention is to realize in the following manner, a transfer server is adopted to dispatch, multiple servers is adopted to provide distributed file server, task amount is assigned in multiple stage Resource Server, thus solve file in enormous quantities and upload simultaneously, bottleneck problem during download in concurrency and efficiency of service, because transfer server receives only very little command information, the concurrency that then transfer server can process is about 20M/0.1K=200000, actual file service will be distributed in the Resource Server of cluster, customer satisfaction transmission rate is improved in actual file transmission service, the task concurrency that 400k/s can not affect transfer server was brought up to by former 10K/s, increase the quantity of Resource Server in cluster, file transfer services speed will be improved, in like manner, when a transfer server cannot reach the concurrency of practical application needs, mode by cluster replaces single transfer server to complete scheduler task and process super large batch concurrent tasks, reach the concurrency of certain specific satisfaction like this, its computing formula is as follows:
Total concurrency=transfer server number * customer satisfaction speed places an order the concurrency of transfer server
In addition, ICP/IP protocol send the instruction message of 100 the byte left and right sizes time used (
tcmd)for Microsecond grade, size be 4M file transmission time (
tfile)for several seconds to tens seconds,
tcmd << Tfile, the time that the transmission of instruction message consumes can not produce significantly impact to the efficiency of file transfer services;
Distributed document transmission performing step is as follows:
1) client subscription interface
Client subscription interface provides the subscribe request interface of file transfer services for client, client sends file server requests by subscribing interface to service end, and wait for the message that service end is return, parse from the return message of service end and communicate and request service for this request service provides the service side address of service, client again directly to set up with this service side;
Under ICP/IP protocol, client is set up socket by subscribing interface to service end application and is connected, after connection establishment, file transfer services request is sent to service end by this connection, after request sends, this connection will keep connection status, until client initiatively disconnects service;
2) service end transfer service
Service end transfer service operation, on service end transfer server, comprising: one is, receives the subscribe request of client and is transmitted to the service of service end distributed treatment; Two are, receive service end distributed treatment and serve the result returned subscribe request service, and this result is returned to the client of this subscribe request;
After service end transfer service reception to the subscribe request of client, solicited message is resolved, obtain and register subscription service code and the mailing address of this subscriber, for meeting the application under application circumstances, if adopt ICP/IP protocol to receive client-requested, then the socket recording the employing of this subscribe request connects by service end transfer service, and adopts this connection to the return service request result of client;
If service end transfer service receives the return message of subscription service, carry out message resolving the subscription service code obtaining this message, and in registration form, search the subscribe request person's address information corresponding to this service codes according to this service codes, by this address information, will subscribe to result return give client subscription requestor;
3) service end distributed treatment service
Service end distributed treatment service operation is on each Resource Server of service end, the subscribe request that this service reception transfer service forwards, after message parse, by Storm Distributed Services processing method, subscribe request is processed in real time, obtain the service provider information of this subscribe request, and give service end transfer service by return for result.
Three features of distributed document transmission service component:
1) the file transfer services subscribing interface arranged, form sends subscribe request according to the rules, and subscribe request has certain format to specify.Below a format sample:
Heading+service codes+service request parameter+message marking
Heading is protocol massages head, and length specifies according to the agreement adopted; Service codes is generated automatically by subscription service, for identification service request; Required parameter is for stating required service content; Message marking is used for identification message type;
2) service classified, dissimilar service request obtains service from different service sides;
3) transmission of messages supports various protocols.
The present invention produces useful gain from efficiency, easily extensible and safety three aspects:
1) distributed document transmission service, by distributed file service, solve file in enormous quantities and upload simultaneously, download run in efficiency and quantitative bottleneck problem, raising service provides efficiency;
2) by supporting the loading of multi-protocols and self-defining communication protocol, extensibility and the applicability of service is improved;
3) under ICP/IP protocol and mutation agreement thereof, client takes the long mode connected to communicate with service end, and the means Deterministic service such as non-explicit port acquisition is safe and reliable in addition.
Accompanying drawing explanation
Fig. 1 is system configuration schematic diagram;
Fig. 2 is client flow chart;
Fig. 3 is service end transfer service procedure figure.
Embodiment
With reference to Figure of description, a kind of distributed document transmission service method of the present invention is described in detail below.
For theoretical foundation of the present invention is described, now make following hypothesis:
1) a station server available network bandwidth is
bnet;
2) a station server disk read-write speed is
rhd;
3) the spendable Thread Count of a station server is
nthread;
4) the TCP/IP linking number that a station server can access simultaneously is
nlink;
5) transmission rate of customer satisfaction is Rclient;
Then the transmission rate of a station server is:
Rtrans?=
?min{?Bnet?,
Rhd?}
The receivable concurrent tasks amount of current server is:
Nconcurrent?=?
min{?Rtrans/Rclient?,
?Nthread?,
?Nlink?}
In general,
bnet >> Rhd, namely the transmission rate of server will depend on the read-write speed of disk, and through reality measuring and calculating, the read-write speed of ordinary magnetic disc is approximately 20M/s, as
rclient=10K/s(in practical application will much larger than this value), then
rtrans/Rclient=2K; When disk utilization reaches 100%, the stagnation of whole method of operation will be caused; The theoretical boundary of the TCP/IP linking number that method can access simultaneously is that 65535(is relevant to concrete operation method and version), but do not reach this value far away in practical application, through reality measuring and calculating,
nthreadwhen value about 5000, good performance can be obtained.By analyzing above and can finding out: accessible concurrent tasks amount is:
Nconcurrent?=?
Rtrans/Rclient?=?2K
Under Single-Server file transfer services pattern, when customer satisfaction improves, must cause server the reduction of treatable concurrent tasks amount.
A transfer server is adopted to dispatch in the present invention, multiple stage Resource Server is adopted to provide distributed file server, task amount is assigned in multiple stage Resource Server, thus solves that file in enormous quantities is uploaded, bottleneck problem when downloading in concurrency and efficiency of service simultaneously.Because transfer server receives only very little command information (being approximately 100 bytes), the concurrency that then transfer server can process is about 20M/0.1K=200000, actual file service will be distributed in the Resource Server of cluster, the task concurrency that customer satisfaction transmission rate (bringing up to 400k/s by former 10K/s) can not affect transfer server is improved in actual file transmission service, increase the quantity of Resource Server in cluster, file transfer services speed will be improved.In like manner, when a transfer server cannot reach the concurrency of practical application needs, the mode by cluster replaces single transfer server to complete scheduler task and process super large batch concurrent tasks.The number of concurrent computing formula reaching certain specific satisfaction is like this as follows:
Total concurrency=transfer server number * customer satisfaction speed places an order the concurrency of transfer server
In addition, ICP/IP protocol sends the instruction message of 100 the byte left and right sizes time used
tcmdfor Microsecond grade, size is the file transmission time of 4M
tfilefor several seconds to tens seconds,
tcmd << Tfile, the time that the transmission of instruction message consumes can not produce significantly impact to the efficiency of file transfer services.
With reference to shown in Fig. 1, its specific implementation is as follows:
1) client subscription interface
Client subscription interface provides the subscribe request interface of file transfer services for client, client sends file server requests by subscribing interface to service end, and wait for the message that service end is return, parse from the return message of service end and communicate and request service for this request service provides the service side address of service, client again directly to set up with this service side;
Under ICP/IP protocol, client is set up socket by subscribing interface to service end application and is connected, after connection establishment, file transfer services request is sent to service end by this connection, after request sends, this connection will keep connection status, until client initiatively disconnects service;
2) service end transfer service
Service end transfer service operation, on service end transfer server, comprising: one is, receives the subscribe request of client and is transmitted to the service of service end distributed treatment; Two are, receive service end distributed treatment and serve the result returned subscribe request service, and this result is returned to the client of this subscribe request;
After service end transfer service reception to the subscribe request of client, solicited message is resolved, obtain and register subscription service code and the mailing address of this subscriber, for meeting the application under application circumstances, if adopt ICP/IP protocol to receive client-requested, then the socket recording the employing of this subscribe request connects by service end transfer service, and adopts this connection to the return service request result of client;
If service end transfer service receives the return message of subscription service, carry out message resolving the subscription service code obtaining this message, and in registration form, search the subscribe request person's address information corresponding to this service codes according to this service codes, by this address information, will subscribe to result return give client subscription requestor;
3) service end distributed treatment service
Service end distributed treatment service operation is on each Resource Server of service end, the subscribe request that this service reception transfer service forwards, after message parse, by Storm Distributed Services processing method, subscribe request is processed in real time, obtain the service provider information of this subscribe request, and give service end transfer service by return for result.
Except the technical characteristic described in specification, be the known technology of those skilled in the art.
Claims (2)
1. a distributed document transmission service method, it is characterized in that, a transfer server is adopted to dispatch, multiple stage Resource Server is adopted to provide distributed file server, task amount is assigned in multiple stage Resource Server, thus solve file in enormous quantities and upload simultaneously, bottleneck problem during download in concurrency and efficiency of service, because transfer server receives only very little command information, the concurrency that then transfer server can process is about 20M/0.1K=200000, actual file service will be distributed in the Resource Server of cluster, customer satisfaction transmission rate is improved in actual file transmission service, the task concurrency that 400k/s can not affect transfer server was brought up to by former 10K/s, increase the quantity of Resource Server in cluster, file transfer services speed will be improved, in like manner, when a transfer server cannot reach the concurrency of practical application needs, mode by cluster replaces single transfer server to complete scheduler task and process super large batch concurrent tasks, reach the concurrency of certain specific satisfaction like this, its computing formula is as follows:
Total concurrency=transfer server number * customer satisfaction speed places an order the concurrency of transfer server
In addition, ICP/IP protocol send the instruction message of 100 the byte left and right sizes time used (
tcmd)for Microsecond grade, size be 4M file transmission time (
tfile)for several seconds to tens seconds,
tcmd << Tfile, the time that the transmission of instruction message consumes can not produce significantly impact to the efficiency of file transfer services;
Distributed document transmission performing step is as follows:
1) client subscription interface
Client subscription interface provides the subscribe request interface of file transfer services for client, client sends file server requests by subscribing interface to service end, and wait for the message that service end is return, parse from the return message of service end and communicate and request service for this request service provides the service side address of service, client again directly to set up with this service side;
Under ICP/IP protocol, client is set up socket by subscribing interface to service end application and is connected, after connection establishment, file transfer services request is sent to service end by this connection, after request sends, this connection will keep connection status, until client initiatively disconnects service;
2) service end transfer service
Service end transfer service operation, on service end transfer server, comprising: one is, receives the subscribe request of client and is transmitted to the service of service end distributed treatment; Two are, receive service end distributed treatment and serve the result returned subscribe request service, and this result is returned to the client of this subscribe request;
After service end transfer service reception to the subscribe request of client, solicited message is resolved, obtain and register subscription service code and the mailing address of this subscriber, for meeting the application under application circumstances, if adopt ICP/IP protocol to receive client-requested, then the socket recording the employing of this subscribe request connects by service end transfer service, and adopts this connection to the return service request result of client;
If service end transfer service receives the return message of subscription service, carry out message resolving the subscription service code obtaining this message, and in registration form, search the subscribe request person's address information corresponding to this service codes according to this service codes, by this address information, will subscribe to result return give client subscription requestor;
3) service end distributed treatment service
Service end distributed treatment service operation is on each Resource Server of service end, the subscribe request that this service reception transfer service forwards, after message parse, by Storm Distributed Services processing method, subscribe request is processed in real time, obtain the service provider information of this subscribe request, and give service end transfer service by return for result.
2., according to said method, three features of distributed document transmission service component can be inferred:
(1) the file transfer services subscribing interface arranged, form sends subscribe request according to the rules, and subscribe request has certain format to specify;
Below a format sample:
Heading+service codes+service request parameter+message marking
Heading is protocol massages head, and length specifies according to the agreement adopted; Service codes is generated automatically by subscription service, for identification service request; Required parameter is for stating required service content; Message marking is used for identification message type;
(2) service classified, dissimilar service request obtains service from different service sides;
(3) transmission of messages supports various protocols.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410516628.7A CN104243610B (en) | 2014-09-30 | 2014-09-30 | A kind of distributed document transmits method of servicing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410516628.7A CN104243610B (en) | 2014-09-30 | 2014-09-30 | A kind of distributed document transmits method of servicing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104243610A true CN104243610A (en) | 2014-12-24 |
| CN104243610B CN104243610B (en) | 2017-07-07 |
Family
ID=52230938
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410516628.7A Active CN104243610B (en) | 2014-09-30 | 2014-09-30 | A kind of distributed document transmits method of servicing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104243610B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104539506A (en) * | 2015-01-27 | 2015-04-22 | 浪潮集团有限公司 | Network transmission service component and network transmission system |
| CN106230817A (en) * | 2016-07-29 | 2016-12-14 | 中国电子科技集团公司第二十八研究所 | Distributed mass data transmission method and system |
| CN106375481A (en) * | 2016-11-09 | 2017-02-01 | 江麓机电集团有限公司 | Distributed ERP and MES data synchronous connection system and data communication method |
| CN106534296A (en) * | 2016-11-09 | 2017-03-22 | 江麓机电集团有限公司 | Distributed file transfer method and transfer platform |
| CN106657402A (en) * | 2017-02-21 | 2017-05-10 | 上海微喂网络科技有限公司 | Transmission method and transmission architecture for real-time multimedia content delivery network |
| CN106997570A (en) * | 2017-03-24 | 2017-08-01 | 武汉天喻教育科技有限公司 | One kind education cloud data exchange system and its method for interchanging data |
| CN112261146A (en) * | 2020-10-23 | 2021-01-22 | 杭州卷积云科技有限公司 | Edge cloud cooperative communication system and method based on message communication and file transmission |
| CN113923211A (en) * | 2021-11-18 | 2022-01-11 | 许昌许继软件技术有限公司 | Message mechanism-based file transmission system and method for power grid control |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1455347A (en) * | 2002-04-30 | 2003-11-12 | 电子科技大学 | Distributed parallel scheduling wide band network server system |
| CN101207550A (en) * | 2007-03-16 | 2008-06-25 | 中国科学技术大学 | Load balancing system and method for multi business to implement load balancing |
| WO2011072979A1 (en) * | 2009-12-15 | 2011-06-23 | International Business Machines Corporation | Concurrent execution of request processing and analytics of requests |
| CN103873321A (en) * | 2014-03-05 | 2014-06-18 | 国家电网公司 | Distributed file system-based simulation distributed parallel computing platform and method |
-
2014
- 2014-09-30 CN CN201410516628.7A patent/CN104243610B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1455347A (en) * | 2002-04-30 | 2003-11-12 | 电子科技大学 | Distributed parallel scheduling wide band network server system |
| CN101207550A (en) * | 2007-03-16 | 2008-06-25 | 中国科学技术大学 | Load balancing system and method for multi business to implement load balancing |
| WO2011072979A1 (en) * | 2009-12-15 | 2011-06-23 | International Business Machines Corporation | Concurrent execution of request processing and analytics of requests |
| CN103873321A (en) * | 2014-03-05 | 2014-06-18 | 国家电网公司 | Distributed file system-based simulation distributed parallel computing platform and method |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104539506A (en) * | 2015-01-27 | 2015-04-22 | 浪潮集团有限公司 | Network transmission service component and network transmission system |
| CN106230817A (en) * | 2016-07-29 | 2016-12-14 | 中国电子科技集团公司第二十八研究所 | Distributed mass data transmission method and system |
| CN106230817B (en) * | 2016-07-29 | 2019-04-02 | 中国电子科技集团公司第二十八研究所 | Distributed mass data transmission method and system |
| CN106375481A (en) * | 2016-11-09 | 2017-02-01 | 江麓机电集团有限公司 | Distributed ERP and MES data synchronous connection system and data communication method |
| CN106534296A (en) * | 2016-11-09 | 2017-03-22 | 江麓机电集团有限公司 | Distributed file transfer method and transfer platform |
| CN106657402A (en) * | 2017-02-21 | 2017-05-10 | 上海微喂网络科技有限公司 | Transmission method and transmission architecture for real-time multimedia content delivery network |
| CN106657402B (en) * | 2017-02-21 | 2019-10-22 | 上海微喂网络科技有限公司 | The transmission method and transmission architecture of real-time multimedia contents distribution network |
| CN106997570A (en) * | 2017-03-24 | 2017-08-01 | 武汉天喻教育科技有限公司 | One kind education cloud data exchange system and its method for interchanging data |
| CN112261146A (en) * | 2020-10-23 | 2021-01-22 | 杭州卷积云科技有限公司 | Edge cloud cooperative communication system and method based on message communication and file transmission |
| CN112261146B (en) * | 2020-10-23 | 2022-05-24 | 杭州卷积云科技有限公司 | Edge cloud cooperative communication system and method based on message communication and file transmission |
| CN113923211A (en) * | 2021-11-18 | 2022-01-11 | 许昌许继软件技术有限公司 | Message mechanism-based file transmission system and method for power grid control |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104243610B (en) | 2017-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104243610A (en) | Distributed file transmission service method | |
| CN110300050A (en) | Information push method, device, computer equipment and storage medium | |
| US20140317071A1 (en) | Method and device for transferring file | |
| CN111460129B (en) | Method, device, electronic equipment and storage medium for generating identification | |
| CN110109766B (en) | Data interaction method and device based on cross-department and cross-platform data sharing exchange | |
| CN104539510A (en) | Multi-protocol-based information pushing system and method | |
| CN102594886B (en) | Method and device for direct communication between browsers, and communication system | |
| CN105471700B (en) | A kind of methods, devices and systems of Message Processing | |
| CN113031874A (en) | Cache processing method, device, equipment and storage medium based on Kubernetes cluster | |
| CN115686875A (en) | Method, apparatus and program product for transferring data between multiple processes | |
| CN114500633B (en) | Data forwarding method, related device, program product and data transmission system | |
| CN105792247B (en) | data pushing method and device | |
| CN112104679B (en) | Method, apparatus, device and medium for processing hypertext transfer protocol request | |
| JP2006235831A (en) | Condition management device, method, and program | |
| CN103068052B (en) | Dynamic resource allocation method, system and Portal server | |
| US20190370293A1 (en) | Method and apparatus for processing information | |
| CN109150927B (en) | File issuing method and device for file storage system | |
| CN110417875B (en) | Method, system, equipment and storage medium for docking between cloud servers | |
| CN110798495A (en) | Method and server for end-to-end message push in cluster architecture mode | |
| CN112073488A (en) | Method and device for processing request | |
| CN104270444A (en) | Method capable of being used for configuring mobile phone application menu | |
| CN115914319A (en) | Remote management method and system for virtual reality equipment | |
| CN110661850A (en) | Edge calculation method, system, computer equipment and storage medium | |
| CN108289165B (en) | Method and device for realizing camera control based on mobile phone and terminal equipment | |
| CN106936643B (en) | Equipment linkage method and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |