CN108075989A - A kind of load balancing network middleware implementation method based on extensible protocol - Google Patents
A kind of load balancing network middleware implementation method based on extensible protocol Download PDFInfo
- Publication number
- CN108075989A CN108075989A CN201611043144.0A CN201611043144A CN108075989A CN 108075989 A CN108075989 A CN 108075989A CN 201611043144 A CN201611043144 A CN 201611043144A CN 108075989 A CN108075989 A CN 108075989A
- Authority
- CN
- China
- Prior art keywords
- message
- thread
- size
- thread pool
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
- H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of load balancing network middleware implementation methods based on extensible protocol：1st, server-side receives the multiframe message of client；Multiframe packet parsing is acquisition type of message after single frames message by step 2；Step 3 will read message and be put into reading thread pool, will write message and is put into and writes thread pool, wherein, small size message is using the method for salary distribution on stack, and large-size message is using the method for salary distribution on heap；Step 4 reads thread pool and writes thread pool load-balancing algorithm is called to select specific processing thread respectively, and message content is put into corresponding thread message queue；Step 5, for writing message, write thread and write data into disk file, and return to confirmation message；For reading message, thread is read then according to the corresponding disk file of search index, data are read in and are cached, and assembles good news and returns to corresponding client.There is high-performance, extensible protocol and load balancing, can effectively saving network flow and improve the speed of response.
Description
Technical field
The present invention relates to a kind of load balancing network middleware implementation methods based on extensible protocol.
Background technology
Time series data treatment technology is widely used in Wide Area Measurement System（WAMS）, supervisory control of substation, scheduling, direct current,
In multiple power automatic systems such as steady control, to meet the needs of it stores magnanimity, high-frequency data.Time series database is mainly used for
Second grade, Millisecond high-frequency data are handled, each storage cycle (millisecond or second) can generate batch of data, therefore data volume pole
Greatly, it is high to the requirement of the readwrite performances of network traffics and network middleware, the demand be different from traditional network middleware, it is necessary to
Customized development is suitable for the network middleware of time series database, is optimized for message, network and storage characteristics.
Traditional network middleware implementation method is as follows：
(1) network middleware based on far call (RPC, Remote Procedure Call)
The basic communication model of far call middleware is that a kind of synchronization based on Client/Server interprocess communication models is led to
Letter form is extension of the process type Programming in Distributed Application.The middleware provides long-range clothes for Client
The procedural abstraction of business, bottom message passing operations are fully transparent to Client.In RPC, Client is request clothes
The caller (Caller) of business, and Server be then perform Client request and called program.
(2) network middleware based on distributed objects (ORB, Object Request Broker)
The network middleware of this form is with DCOM（Distributed Component Object Model, Distributed Component Object pattern）And CORBA
（Common Object Request Broker Architecture）For represent, but no matter which kind of standard with realize, by with
Lower part composition.
A, the remote object of service and function is actually accomplished, is responsible for actually accomplishing system service, receives remote request.
B, access client is acted on behalf of, and is responsible for receiving request from client, and is converted the request into far call and be sent to clothes
Business end.
C, Object Request Broker ORB provides a communications framework, the transparent transmission pair in heterogeneous distributed computing environment
As request, it is responsible for positioning object implementatio8 and is returned the result after request is transferred to object implementatio8, in distributed network middleware
In core status.
(3) network middleware based on message queue (Message-oriented Middleware)
Message based network middleware is mainly used for delivering message between different application, and the message for being absorbed in isomerous environment is handed over
It changes, current most popular message-oriented network middleware is the ActiveMQ of Apache.Message may not include the meaning of one's words and state,
Therefore message-oriented network middleware is most transparent selection for application.Using source, the connection that need not be concerned about message
Mode, and middleware need not be concerned about the meaning of one's words of message, so can be very good to realize and application decoupling.
Although more than network middleware is all more mature frame, big data quantity, high performance requirements are not particularly suited for
Time series data processing.First, these frames are all general frameworks, can not optimize for the key-value pair characteristic of time series data, make
Communications protocol is excessively huge and cause the wasting of resources；Secondly, general framework needs to consider excessive scene, therefore middleware mistake
In complexity, such as CORBA is exactly a heavyweight frame, it will ties down entire time series database deployment；Again, traditional middleware
Underlying operating system correlation is shielded well, accomplishes versatility and cross-platform, but this is to lose certain system performance
On the basis of, therefore the big data quantity of time series data and the requirement of low-response time delay can not be met.
The content of the invention
In view of the above-mentioned problems, the present invention provides a kind of load balancing network middleware realization side based on extensible protocol
Method has many advantages, such as high-performance, extensible protocol and load balancing, can effectively saving network flow and improve the speed of response.
To realize above-mentioned technical purpose and the technique effect, the invention is realized by the following technical scheme：
A kind of load balancing network middleware implementation method based on extensible protocol, which is characterized in that include the following steps：
Step 1, server-side receive the multiframe message of client, wherein each frame message is 258 bytes, each frame message
Form be：
Message size+the message content of+1 byte of type of message of 1 byte：
Type of message is used to read message and writes the judgement of message and the judgement of small size message and large-size message；
For small size message, message size is used to set the effective byte digit of message content, and message content is up to 256
Byte；For large-size message, message size empties, and message content is used to record the address of real data, and message content is maximum
For 256 bytes；
Multiframe packet parsing is acquisition type of message after single frames message by step 2；
Step 3 will read message and be put into reading thread pool, will write message and is put into and writes thread pool, wherein, small size message uses on stack points
With mode, large-size message is using the method for salary distribution on heap；
Step 4 reads thread pool and writes thread pool load-balancing algorithm is called to select specific processing thread respectively, and will be in message
It receives into corresponding thread message queue；
Step 5, for writing message, write thread and write data into disk file, and return to confirmation message；For reading message, line is read
Data are read in and cached, and assembled good news and return to corresponding client according to the corresponding disk file of search index by Cheng Ze.
Load-balancing algorithm include weight polling algorithm, random algorithm, weight random algorithm, response time priority algorithm,
Minimum connection priority algorithm.
It is preferred that reading thread pool and writing thread pool to include multiple reading threads respectively and multiple write thread.
It is preferred that minimum 8 bytes of message content.
The beneficial effects of the invention are as follows：
（1）Communications protocol can be extended by way of extending message frame number.
（2）It need not be that each client establishment handles thread, reduce system resource overhead, support client maximum simultaneously
Hair number significantly improves.Significantly improve message processing capability, 500 all things parts of processing per second.
（3）Network traffics are reduced, the CPU and memory that server-side occupies also are significantly reduced.
Description of the drawings
Fig. 1 is multi-frame message structure diagram of the present invention；
Fig. 2 is message form schematic diagram on stack of the present invention；
Fig. 3 is message form schematic diagram on heap of the present invention；
Fig. 4 is load balancing schematic diagram of the present invention；
Fig. 5 is a kind of flow diagram of the load balancing network middleware implementation method based on extensible protocol of the present invention.
Specific embodiment
Technical solution of the present invention is described in further detail with specific embodiment below in conjunction with the accompanying drawings, so that ability
The technical staff in domain can be better understood from the present invention and can be practiced, but illustrated embodiment is not as the limit to the present invention
It is fixed.
As shown in Figs. 1-5, a kind of load balancing network middleware implementation method based on extensible protocol, including walking as follows
Suddenly：
Step 1, as shown in Figure 1, server-side receives the multiframe message of client, wherein each frame message is 258 bytes,
It is specified by constant in compiling, the form of each frame message is：
Message size+the message content of+1 byte of type of message of 1 byte：
Type of message is used to read message and writes the judgement of message and the judgement of small size message and large-size message, small size
Message and large-size message are opposite, and critical value can be adjusted according to real system, are set by code macrodefinition, for example,
Message content is small size message within 256 bytes, is large-size message more than 256 bytes.
For small size message, message size is used to set the effective byte digit of message content, and message content is up to
256 bytes, minimum 8 bytes, when message content is 256 bytes, the afterbody of each frame message is efficient message,
Otherwise, the afterbody of each frame message there are invalid message, can set the effective byte digit of message content by message size,
As shown in Figure 2.
For large-size message, message size empties, and message size is put into specific data frame, and message content is used to record
The address of real data, the address space of direction include data pointer, size of data, the destructor function customized, reference count
With the information such as data content.Likewise, message content is up to 256 bytes, minimum 8 bytes, as shown in Figure 3.
It is fixed and parsed according to each frame message length, the information of needs is obtained from multiframe message, such as message header, message class
Type etc., the multi-frame message form can easily carry out message content extension, and communication association is extended by the way of message frame is extended
View, and existing structure and application message structure need not be destroyed.Message header is also regarded as an independent message frame, can so lead to
The mode for crossing extension message frame number extends communications protocol.
Multiframe packet parsing is acquisition type of message after single frames message by step 2：That is read-write type and size dimension disappears
Cease type.
Step 3 will read message and be put into reading thread pool, will write message and is put into and writes thread pool, is written and read after separation and is put into pair
The thread pool answered can reduce data sharing to the greatest extent, to reduce shared mutual exclusion expense, avoid mutex, semaphore, competition item
The synchronization means such as part.Read-write has multiple（Acquiescence creates 8 reading threads and 8 are write thread）Thread forms thread pool, common point
Director is engaged in.
Wherein, small size message because message-length it is limited, if using memory Heap Allocation, destructed, performance bottleneck can be caused,
RAM fragmentation is easily caused, so using method of salary distribution performance on stack more preferably.If single frames message is large-size message, it is necessary to anti-
Only stack overflow, while reference count is supported to avoid big memory copying expense, using the method for salary distribution on heap.
Step 4 reads thread pool and writes thread pool load-balancing algorithm is called to select specific processing thread respectively, and will disappear
Breath content is put into corresponding thread message queue.
As described in Figure 4, load-balancing algorithm includes weight polling algorithm, random algorithm, weight random algorithm, response time
Priority algorithm at least connects priority algorithm.Load-balancing algorithm of the thread pool according to selection is read and write, selectes specific processing line
Journey.The strategy of network middleware default choice is weight polling algorithm.Each read-write thread pool is pre-created one according to configuration
The reading thread pool of fixed number amount thread and thread pool is write, be managed by unified thread manager, IO is carried out by multithreading
Read-write can significantly improve resource utilization ratio, and can significantly reduce later stage frequent establishment destruction line using thread is pre-created
The expense of journey.Per thread has the message queue of oneself, and processing message is obtained from message queue.
Step 5, for writing message, write thread and write data into disk file, and return to confirmation message；For reading message,
Thread is read then according to the corresponding disk file of search index, and data are read in and are cached, and assembles good news and returns to corresponding visitor
Family end.
In conclusion the load balancing network middleware implementation method based on extensible protocol of the present invention, can realize
The load balancing of network middleware and expansible, solving traditional network middleware can not adaptation time sequence library big data quantity
With the requirement of low-response time delay so that service node resource is rationally utilized, and integrity service performance is significantly increased.Meanwhile
To procotol using multiframe extensible protocol, and small size message created on stack, large-size message creates on heap, subtract
The time of RAM fragmentation and memory copying is lacked, has reduced network entirety flow, there is preferable scalability.Read and write abruption
Technology alleviates the degree of coupling of each inter-thread data, and cross-thread is not present shared data, reduces substantial amounts of mutex, signal
The synchronization means such as amount, race condition.
It these are only the preferred embodiment of the present invention, be not intended to limit the scope of the invention, it is every to utilize this hair
The equivalent structure that bright specification and accompanying drawing content are made either equivalent process transformation or to be directly or indirectly used in other related
Technical field, be included within the scope of the present invention.
Claims (4)
1. a kind of load balancing network middleware implementation method based on extensible protocol, which is characterized in that include the following steps：
Step 1, server-side receive the multiframe message of client, wherein each frame message is 258 bytes, each frame message
Form be：
Message size+the message content of+1 byte of type of message of 1 byte：
Type of message is used to read message and writes the judgement of message and the judgement of small size message and large-size message；
For small size message, message size is used to set the effective byte digit of message content, and message content is up to 256
Byte；For large-size message, message size empties, and message content is used to record the address of real data, and message content is maximum
For 256 bytes；
Multiframe packet parsing is acquisition type of message after single frames message by step 2；
Step 3 will read message and be put into reading thread pool, will write message and is put into and writes thread pool, wherein, small size message uses on stack points
With mode, large-size message is using the method for salary distribution on heap；
Step 4 reads thread pool and writes thread pool load-balancing algorithm is called to select specific processing thread respectively, and will be in message
It receives into corresponding thread message queue；
Step 5, for writing message, write thread and write data into disk file, and return to confirmation message；For reading message, line is read
Data are read in and cached, and assembled good news and return to corresponding client according to the corresponding disk file of search index by Cheng Ze.
2. a kind of load balancing network middleware implementation method based on extensible protocol according to claim 1, special
Sign is, load-balancing algorithm includes weight polling algorithm, random algorithm, weight random algorithm, response time priority algorithm, most
Priority algorithm is connected less.
3. a kind of load balancing network middleware implementation method based on extensible protocol according to claim 1, special
Sign is, reads thread pool and write thread pool to include multiple reading threads respectively and multiple write thread.
4. a kind of load balancing network middleware implementation method based on extensible protocol according to claim 1, special
Sign is, minimum 8 bytes of message content.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611043144.0A CN108075989B (en) | 2016-11-11 | 2016-11-11 | Extensible protocol-based load balancing network middleware implementation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611043144.0A CN108075989B (en) | 2016-11-11 | 2016-11-11 | Extensible protocol-based load balancing network middleware implementation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108075989A true CN108075989A (en) | 2018-05-25 |
| CN108075989B CN108075989B (en) | 2021-05-28 |
Family
ID=62161515
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611043144.0A Active CN108075989B (en) | 2016-11-11 | 2016-11-11 | Extensible protocol-based load balancing network middleware implementation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108075989B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109286572A (en) * | 2018-09-30 | 2019-01-29 | 郑州冰川网络技术有限公司 | Dynamic domain name resolution method |
| CN116089414A (en) * | 2023-04-10 | 2023-05-09 | 之江实验室 | Time sequence database writing performance optimization method and device based on mass data scene |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801683A (en) * | 2004-12-31 | 2006-07-12 | 华为技术有限公司 | Message flow control method |
| US20070115939A1 (en) * | 2005-10-12 | 2007-05-24 | Samsung Electronics Co., Ltd. | Network on chip system employing an advanced extensible interface protocol |
| CN101194482A (en) * | 2003-08-13 | 2008-06-04 | 高通股份有限公司 | Signal interface for higher data rates |
| US8090873B1 (en) * | 2005-03-14 | 2012-01-03 | Oracle America, Inc. | Methods and systems for high throughput information refinement |
| CN103095591A (en) * | 2012-12-12 | 2013-05-08 | 南京南瑞继保电气有限公司 | Method increasing garp multicast registration protocol (GMRP) multicast registration reliability |
| CN103645862A (en) * | 2013-12-12 | 2014-03-19 | 北京同有飞骥科技股份有限公司 | Initialization performance improvement method of redundant arrays of inexpensive disks |
| CN105242882A (en) * | 2015-10-13 | 2016-01-13 | 东方网力科技股份有限公司 | Frame storage method and apparatus for timing data and query method and apparatus for timing data |
| CN105491150A (en) * | 2015-12-28 | 2016-04-13 | 中国民航信息网络股份有限公司 | Load balance processing method based on time sequence and system |
-
2016
- 2016-11-11 CN CN201611043144.0A patent/CN108075989B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101194482A (en) * | 2003-08-13 | 2008-06-04 | 高通股份有限公司 | Signal interface for higher data rates |
| CN1801683A (en) * | 2004-12-31 | 2006-07-12 | 华为技术有限公司 | Message flow control method |
| US8090873B1 (en) * | 2005-03-14 | 2012-01-03 | Oracle America, Inc. | Methods and systems for high throughput information refinement |
| US20070115939A1 (en) * | 2005-10-12 | 2007-05-24 | Samsung Electronics Co., Ltd. | Network on chip system employing an advanced extensible interface protocol |
| CN103095591A (en) * | 2012-12-12 | 2013-05-08 | 南京南瑞继保电气有限公司 | Method increasing garp multicast registration protocol (GMRP) multicast registration reliability |
| CN103645862A (en) * | 2013-12-12 | 2014-03-19 | 北京同有飞骥科技股份有限公司 | Initialization performance improvement method of redundant arrays of inexpensive disks |
| CN105242882A (en) * | 2015-10-13 | 2016-01-13 | 东方网力科技股份有限公司 | Frame storage method and apparatus for timing data and query method and apparatus for timing data |
| CN105491150A (en) * | 2015-12-28 | 2016-04-13 | 中国民航信息网络股份有限公司 | Load balance processing method based on time sequence and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109286572A (en) * | 2018-09-30 | 2019-01-29 | 郑州冰川网络技术有限公司 | Dynamic domain name resolution method |
| CN116089414A (en) * | 2023-04-10 | 2023-05-09 | 之江实验室 | Time sequence database writing performance optimization method and device based on mass data scene |
| CN116089414B (en) * | 2023-04-10 | 2023-09-08 | 之江实验室 | Time sequence database writing performance optimization method and device based on mass data scene |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108075989B (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459418B (en) | RDMA (remote direct memory Access) -based key value storage system transmission method | |
| CN105635298B (en) | A kind of data acquisition equipment uniform access system based on business separation principle | |
| CN104731569B (en) | A kind of data processing method and relevant device | |
| CN104754052A (en) | Message pushing method based on Websocket | |
| CN108055255A (en) | A kind of event base, expansible data management system and its management method | |
| CN110120919A (en) | A kind of Internet resources partition method and system for capacitor network | |
| CN111404818B (en) | Routing protocol optimization method for general multi-core network processor | |
| CN118227343B (en) | Data processing method, system, device, equipment, medium and product | |
| US8135851B2 (en) | Object request broker for accelerating object-oriented communications and method | |
| CN109828842A (en) | A kind of high-performance data acquisition engine method based on DPDK technological development | |
| CN108075989A (en) | A kind of load balancing network middleware implementation method based on extensible protocol | |
| Wu et al. | The Research and Implementation of parallel web crawler in cluster | |
| CN109829094A (en) | Distributed reptile system | |
| CN111797497B (en) | Communication method and system for electromagnetic transient parallel simulation | |
| CN118093219A (en) | Message processing method, device, equipment and storage medium | |
| CN111400034B (en) | Multi-core processor-oriented waveform resource allocation method | |
| Simon et al. | High-Performance Match-Action Table Updates from within Programmable Software Data Planes | |
| CN106997304B (en) | Input and output event processing method and device | |
| CN115499889B (en) | UPF forwarding plane realization method, device, system and related equipment based on VPP | |
| CN102902593A (en) | Protocol distribution processing system based on cache mechanism | |
| Li et al. | Improving spark performance with zero-copy buffer management and RDMA | |
| CN116841720A (en) | Resource allocation method, apparatus, computer device, storage medium and program product | |
| CN113835904A (en) | Remote procedure call control method, device, equipment and storage medium | |
| US8176117B2 (en) | Accelerator for object-oriented communications and method | |
| Martin et al. | High performance distributed objects using caching proxies for large scale applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |