CN114785695A - High-performance network communication library realized based on zeroC ICE - Google Patents

High-performance network communication library realized based on zeroC ICE Download PDF

Info

Publication number
CN114785695A
CN114785695A CN202210453577.2A CN202210453577A CN114785695A CN 114785695 A CN114785695 A CN 114785695A CN 202210453577 A CN202210453577 A CN 202210453577A CN 114785695 A CN114785695 A CN 114785695A
Authority
CN
China
Prior art keywords
data
zeroc
ice
communication
network communication
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202210453577.2A
Other languages
Chinese (zh)
Inventor
罗达辉
吕晓一
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Qianfang Technology Co ltd
Original Assignee
Shanghai Qianfang Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Qianfang Technology Co ltd filed Critical Shanghai Qianfang Technology Co ltd
Priority to CN202210453577.2A priority Critical patent/CN114785695A/en
Publication of CN114785695A publication Critical patent/CN114785695A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/12Discovery or management of network topologies
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/02Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/34Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters 

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention discloses a high-performance network communication library realized based on a zeroC ICE, which comprises: using a communication architecture of a zeroC ICE as a bottom layer framework of a high-performance network communication library; a network topology connection creation program for creating a network topology connection according to the configuration; a data retransmission program for ensuring the integrity of data in the communication process; and storing the data in the communication process, and taking out the data reading and storing program of the data when in use. The invention has high performance and supports proxy service, can quickly establish the network topology of the nodes participating in the calculation task, and the user can complete the calculation task in a coordinated way only by calling rpc interfaces like calling a local function without paying attention to internal realization and maintaining the network connection state.

Description

High-performance network communication library realized based on zeroC ICE
Technical Field
The invention relates to the field of computer software, in particular to a high-performance network communication library realized based on a zeroC ICE.
Background
In a distributed privacy computing network, each computing task relates to a plurality of node processes, the node processes of each node are dynamically started, the node processes are a server (receiving data sent by the client) and a client (sending data to the server), frequent communication is required among the node processes in the task computing process, if data is lost or incomplete in the task computing process, the task fails, and therefore the node processes need to be monitored in the task computing process to ensure normal and abnormal recovery of network connection so as to ensure the integrity of communication data.
The problems of how to ensure network connection, abnormal automatic recovery, data integrity and the like in the process of computing tasks are urgently needed to be solved by the prior art.
Moreover, the current rpc framework, such as the gRPC, is a "one-to-one response" method in the communication process, and data in the communication process are all acquired in real time without buffer data, and cannot be acquired according to requirements.
Accordingly, the prior art is deficient and needs improvement.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: the high-performance and supporting proxy service based on the zeroC ICE is provided, the network topology of nodes participating in the calculation task can be quickly established, a user does not need to be concerned about internal realization and maintenance of the network connection state, and the high-performance network communication library can cooperatively complete the calculation task only by calling rpc interfaces like calling a local function.
The technical scheme of the invention is as follows: a ZeroC ICE-based implementation of a high-performance network communication library, comprising: using a communication architecture of a zeroC ICE as a bottom layer framework of a high-performance network communication library; a network topology connection creation program for creating a network topology connection according to the configuration; a data retransmission program for ensuring the integrity of data in the communication process; and storing the data in the communication process, and taking out the data reading and storing program of the data when in use.
In the high-performance network communication library realized based on the zeroC ICE, a communication architecture of the zeroC ICE comprises an IceGrid service and a Glacier2 routing firewall, when a calculation task is started, a service of a node process automatically allocates a port and starts monitoring, an ip and the port of a server are bound on a specified AdapterId and a serverId and then are registered on the IceGrid service, and the Glacier2 finds a specified corresponding server from the IceGrid service through the specified AdapterId and the serverId and forwards messages.
In the high-performance network communication library implemented based on ZeroC ICE, the network topology connection creation program includes: and connecting the first high-performance network communication library of the first communication party with a second Glatier 2 routing firewall of the second communication party, and connecting the second network communication library of the second communication party with a first Glatier 2 routing firewall of the first communication party, so as to establish network topology connection between the first communication party and the second communication party, wherein the first communication party and the second communication party are any communication parties in the privacy computation network respectively.
Applied to the above technical solutions, in the high performance network communication library implemented based on ZeroC ICE, the data retransmission procedure includes the following steps: step A1: the client sends data and a service request to the server, and the step A2: after receiving the data and the service request, the server processes the service request, stores the data and returns a response state 0 to the client; step A3: the client waits for a response, calculates the residual overtime time T of the periodic task, then judges whether the T is 0 or not, and if not, continuously judges whether the response state is 0 or not, if so, the transmission is successful.
In the high-performance network communication library implemented based on ZeroC ICE, in step A3, when the response state is determined not to be 0, the high-performance network communication library sleeps for a set time period T, and then returns to continue to execute step a1, where T is less than T.
In the high-performance network communication library implemented based on ZeroC ICE, the data reading and storing program includes: after each server receives data from the client, storing the data in a cache according to a preset cache rule; and each server side acquires data from the cache according to the requirement, and deletes the acquired data from the cache.
In the high-performance network communication library implemented based on zeroC ICE, the preset caching rule is to store nodeid + msgid of the corresponding client as a key; and when the data is obtained from the cache, a Receive interface of the server side is called, and the nodeid + msgid is used as a key word key to obtain the data from the cache.
In the high-performance network communication library implemented based on ZeroC ICE, the data retransmission program is to store long connections between the client and the server in a manner of creating a ZeroC ICE stub object.
In the high-performance network communication library implemented based on ZeroC ICE, the data reading and storing program uses a combination of hash Map and Queue container to store data.
In the high-performance network communication library implemented based on ZeroC ICE, the high-performance network communication library is configured as a pluggable component.
The beneficial effects of the invention are as follows:
the invention uses zeroC ICE as a communication framework to realize rpc interface send of data transmission. The data retransmission program supports the functions of data retransmission, time-out and the like during abnormal conditions such as a network and the like, and ensures the integrity of data and the robustness of the program. The network topology connection establishing program supports the establishment of network topology connection according to configuration; the configuration can also comprise a certificate file, so that the communication safety is ensured. And moreover, the long connection between the client and the server is saved by using a mode of creating a zeroC ICE stub object, and the functions of efficient communication between the client and the server and automatic recovery after abnormal disconnection of a network are realized. The data are saved by using the combination of the Hash Map and the Queue container, the receiving sequence of the data can be ensured, the efficient storage and reading of the data are realized, the network topology of the nodes participating in the calculation task can be quickly established based on a zeroC ICE technology framework, a user can realize and maintain the network connection state without concerning the internal realization, and the calculation task can be completed cooperatively only by calling rpc interfaces like calling a local function, so that the communication efficiency is high.
Drawings
FIG. 1 is a schematic diagram of the implementation connection of the Internet zeroC ICE bottom layer communication framework program in the present invention;
FIG. 2 is a schematic diagram illustrating an executed connection of a network topology connection creation program according to the present invention;
FIG. 3 is a flowchart illustrating an implementation of a data retransmission procedure according to the present invention;
FIG. 4 is a schematic diagram illustrating the connection of the data reading and storing procedures of the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and the specific embodiments.
The embodiment provides a high-performance network communication library implemented based on ZeroC ICE, which includes: using a communication architecture of a zeroC ICE as a bottom layer framework of a high-performance network communication library; a network topology connection creation program for creating a network topology connection according to the configuration; a data retransmission program for ensuring the integrity of data in the communication process; and storing the data in the communication process, and taking out the data reading and storing program of the data when in use.
Wherein, zeroC ICE uses compact, high-efficiency binary protocol to minimize CPU and bandwidth consumption, IDL Slice of ICE is used to define the communication specification between client and server; while the gRPC uses proto protocol, which affects efficiency in message data serialization and deserialization. The high performance and proxy service support based on ZeroC ICE selects ZeroC ICE as the bottom layer communication framework of the high performance network communication library (Channel SDK for short) in this embodiment.
Ice is the most stable, powerful, high-performance, cross-platform and multi-language-supporting old brand open source middleware in the RPC communication field, and is particularly suitable for the programming of multiple development languages on one platform in the current Internet field; the Ice describes the interface of the service through a neutral Language slice (specification Language from Ice) which is irrelevant to a programming Language, so that the aim of separating an object interface from the object interface is fulfilled; the proto protocol is not used as IDL of a client and a server to describe request parameters, response structure and message data serialization (influence communication efficiency) of the service and the method like gRPC, and the Http2 protocol is used by gRPC, and the coding and transmission mechanism efficiency is far lower than that of a Socket mode of long connection, so that the communication efficiency of Channel SDK is greatly improved by selecting zeroC ICE as a communication framework.
Moreover, as shown in fig. 1, the communication architecture of the ZeroC ICE includes an IceGrid service and a Glacier2 routing firewall, when a computation task is started, a service of each node process automatically allocates a port and starts snooping, binds an ip and a port of a server thereof to a specified AdapterId and a servatid and then registers the server to the IceGrid service, and the Glacier2 routing firewall finds a specified corresponding server from the IceGrid service through the specified AdapterId and the servatid and forwards a message. Wherein adapterId and servant Id are the data Adaptation ID and the servo object ID of the server, respectively.
Thus, each participant (node) starts the node process when the calculation task is started, the service of the node process automatically allocates a port and starts monitoring, then the process binds the ip and the port of the server on the appointed adapterId and servant Id and then registers the ip and the port on the IceGrid service, and the Glacer 2 routing firewall finds the appointed corresponding server from the IceGrid through the appointed adapterId and servant Id to forward the message; the node process is used as a stub object which is created by the client and connected with the proxy service of other node processes, and the client sends data to the opposite terminal through the stub object. Wherein each participant can be a client and/or a server; each participant binds the ip and port of its server on the specified AdapterId and servertid and then registers on the IceGrid service of the own-side ZeroC ICE architecture.
As shown in fig. 2, the network topology connection creating program includes: connecting the first high-performance network communication library of the first communication party P0 with the second Glaser 2 routing firewall of the second communication party P1, and connecting the second network communication library of the second communication party with the first Glaser 2 routing firewall of the first communication party, thereby establishing the network topology connection of the first communication party and the second communication party; the first communication party and the second communication party are any communication parties in the privacy computing network, namely any nodes in the privacy computing network, other communication parties in the privacy computing network adopt a network topology connection establishing program which is the same as the first communication party P0 and the second communication party P1 to carry out mutual pairwise connection, and after two or more communication parties participating in the computing task are connected pairwise in any computing task, a plurality of communication parties can establish a small-sized topology network.
The first communication party P0 and the second communication party P1 are both nodes in a distributed privacy computing network, and when participating in a computing task, the first communication party P0 and the second communication party P1 are clients or servers, and both the first communication party P0 and the second communication party P1 are internally provided with a Channel SDK and a ZeroC ICE architecture, so that the first high-performance network communication library of the first communication party P0 is connected with the second Glacier2 of the second communication party P1 through a firewall, and the second Glacier2 finds the corresponding ip and port of the second communication party P1 on the second IceGrid service, so that the message and data of the first communication party P0 can be forwarded to the second communication party P1; similarly, the second high-performance network communication library of the second communication party P1 is connected with the first Glacier2 routing firewall of the first communication party P0, and the first Glacier2 routing firewall finds the corresponding ip and port of the first communication party P0 on the first IceGrid service, so that the message and data of the second communication party P1 can be forwarded to the first communication party P0; therefore, after the network topology connection of all nodes is completed at home, the mutual transmission of messages and data among all the participants can be realized.
In addition, in the process of computing the task, the integrity of the data must be ensured, otherwise, the computing task cannot be successfully executed; each transmission of data must be guaranteed to be successful; as shown in fig. 3, the data retransmission procedure includes the following steps: step A1: the client sends data and a service request to the server, and the step A2: after receiving the sent data and the service request, the server processes the service request, stores the data and returns a response state 0 to the client; step A3: the client waits for a response, calculates the residual overtime time T of the periodic task, then judges whether the T is 0 or not, and if not, continuously judges whether the response state is 0 or not, if so, the transmission is successful. The periodic task time is set according to actual needs, and in step A3, when the response status is judged not to be 0, the system sleeps for a set time period T, and then returns to continue to execute step a1, wherein T < T. In this way, by supporting the functions of data retransmission, timeout and the like during abnormal conditions, the integrity of data and the robustness of a program are ensured.
In addition, as shown in fig. 4, the data reading and storing program includes: after each server receives data from the client, storing the data in a cache according to a preset cache rule; and each server side acquires data from the cache according to the requirement, and deletes the acquired data from the cache. The preset caching rule is that nodeid + msgid of a corresponding client is stored as a key word; and when the data is obtained from the cache, a Receive interface of the server side is called, and the nodeid + msgid is used as a key word key to obtain the data from the cache. Thus, each server may need to receive data sent by multiple clients or a single client for multiple times, and the server stores the data regularly, for example, nodeid + msgid as a key in a cache; the server side calls a Receive interface of the server side according to the requirement, and acquires data from the cache by taking the nodeid + msgid as a key; the retrieved data will be deleted from the cache. The node ID and the msgid are two IDs of the client, wherein the node ID is a node identifier, and the msgid is a communication type identifier; therefore, a plurality of clients can be supported to simultaneously send data to one server, and after the rpc interface of the server responds, the data is cached, and the data in the cache is read according to the requirement; the user only needs to realize data transmission and access as conveniently as a function adjusting function.
In addition, the data retransmission program is a mode of creating a zeroC ICE stub object, long connection between the client and the server is saved, and the functions of efficient communication between the client and the server and automatic recovery after abnormal disconnection of a network are realized.
And the data reading and storing program uses the combination of the Hash Map and the Queue container to store the data, so that the receiving sequence of the data can be ensured, and the efficient storage and reading of the data can be realized.
In addition, the high-performance network communication library of the embodiment is set as a pluggable component, so that the high-performance network communication library can be directly used in a node process in a pluggable mode, and is very convenient to install and use.
The high-performance network communication library is packaged on a zeroC ICE frame, a network topology is established through configuration, data in a communication process are stored, and the data are taken out when the high-performance network communication library is used; in addition, the proxy service in the zeroC ICE framework is combined and used in a component form, and under the existing framework, a user can realize the proxy service only by adding a combined code; the Channel SDK well solves the problem, supports a plurality of clients to send data to one server at the same time, caches the data after the rpc interface of the server responds, and reads the data in the cache according to the requirement; the Channel SDK is used only by configuring information such as IP and Port of Glacier2 and IceGrid, and the like, and the code is not required to be modified, so that a user can use the proxy service without deeply knowing the principle of the Glacer 2 and the IceGrid.
The high-performance network communication library of this embodiment implements point-to-point high-performance communication by means of a stable and efficient rpc framework, and in the whole communication process, it is necessary to ensure the integrity, security and network robustness of data and implement an efficient caching mechanism for data. Where ZeroC ICE uses a compact, efficient binary protocol (gRPC uses proto protocol, affecting efficiency in message data serialization and deserialization) to minimize CPU and bandwidth consumption, and IDL Slice of ICE is used to define the communication specification between the client and the server.
The high-performance network communication library of the embodiment is properly expanded and can be used in any scene of point-to-point high-performance communication; moreover, the network communication service is widely used in the IT industry, and by using the Channel SDK of the high-performance network communication library, a programmer can realize high-performance network communication without concerning the problems of loss, disconnection and the like of sending data in a multi-concurrency and complex network environment.
The present invention is not limited to the above preferred embodiments, and any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A ZeroC ICE-based high performance network communication library, comprising:
using a communication architecture of a zeroC ICE as a bottom layer framework of a high-performance network communication library;
a network topology connection creation program for creating a network topology connection according to the configuration;
a data retransmission program for ensuring the integrity of data in the communication process;
and storing the data in the communication process, and taking out the data reading and storing program of the data when in use.
2. The ZeroC ICE implementation based high performance network communication library of claim 1, wherein: the communication architecture of the zeroC ICE comprises an IceGrid service and a Glactor 2 routing firewall, when a calculation task is started, a service of a node process automatically allocates ports and starts monitoring, an ip and a port of a server are bound on a specified AdapterId and a servantId and then are registered on the IceGrid service, and the Glactor 2 finds a specified corresponding server from the IceGrid service through the specified AdapterId and the servantId to forward messages.
3. The ZeroC ICE implementation based high performance network communication library according to claim 2, characterized in that: the network topology connection creation program includes: and connecting the first high-performance network communication library of the first communication party with a second Glaciar 2 routing firewall of the second communication party, and connecting the second network communication library of the second communication party with a first Glaciar 2 routing firewall of the first communication party, so as to establish network topological connection between the first communication party and the second communication party, wherein the first communication party and the second communication party are any communication parties in the privacy calculation network respectively.
4. The ZeroC ICE implementation based high performance network communication library according to claim 1, characterized in that: the data retransmission procedure comprises the following steps:
step A1: the client sends data and service requests to the server,
step A2: after receiving the data and the service request, the server processes the service request, stores the data and returns a response state 0 to the client;
step A3: the client waits for a response, calculates the residual overtime time T of the periodic task, then judges whether the T is 0 or not, and if not, continuously judges whether the response state is 0 or not, if so, the transmission is successful.
5. The ZeroC ICE implementation based high performance network communication library of claim 3, wherein: in step A3, when the response status is not 0, the sleep is performed for a set time period T, and then the process returns to step a1, where T < T.
6. The ZeroC ICE implementation based high performance network communication library according to claim 1, characterized in that: the data reading and storing program includes: after each server receives data from the client, storing the data in a cache according to a preset cache rule; and each server side acquires data from the cache according to the requirement, and deletes the acquired data from the cache.
7. The ZeroC ICE implementation based high performance network communication library of claim 6, wherein: the preset caching rule is that nodeid + msgid of a corresponding client is used as a key word for storage; and when the data is obtained from the cache, a Receive interface of the server side is called, and the nodeid + msgid is used as a key word key to obtain the data from the cache.
8. The ZeroC ICE implementation based high performance network communication library according to claim 1, characterized in that: the data retransmission program is used for storing long connection between a client and a server in a mode of creating a zeroC ICE stub object.
9. The ZeroC ICE implementation based high performance network communication library according to claim 1, characterized in that: the data reading and storing program uses a combination of hash Map and Queue container to save data.
10. A ZeroC ICE based high performance network communication library according to any of claims 1-9, characterized in that: the high performance network communication library is configured as a pluggable component.
CN202210453577.2A 2022-04-27 2022-04-27 High-performance network communication library realized based on zeroC ICE Withdrawn CN114785695A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210453577.2A CN114785695A (en) 2022-04-27 2022-04-27 High-performance network communication library realized based on zeroC ICE

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210453577.2A CN114785695A (en) 2022-04-27 2022-04-27 High-performance network communication library realized based on zeroC ICE

Publications (1)

Publication Number Publication Date
CN114785695A true CN114785695A (en) 2022-07-22

Family

ID=82433780

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210453577.2A Withdrawn CN114785695A (en) 2022-04-27 2022-04-27 High-performance network communication library realized based on zeroC ICE

Country Status (1)

Country Link
CN (1) CN114785695A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115277846A (en) * 2022-07-27 2022-11-01 上海阵方科技有限公司 Dynamic data fragmentation sending system based on ICE (Integrated Circuit emphasis) implementation

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115277846A (en) * 2022-07-27 2022-11-01 上海阵方科技有限公司 Dynamic data fragmentation sending system based on ICE (Integrated Circuit emphasis) implementation

Similar Documents

Publication Publication Date Title
CN111565229B (en) Communication system distributed method based on Redis
KR100990415B1 (en) Method for synchronizing connection state in data communication, and communication node using the same
CN111479121B (en) Live broadcasting method and system based on streaming media server
CN104811459A (en) Processing method, processing device and system for message services and message service system
CN107528891B (en) Websocket-based automatic clustering method and system
CN111615066A (en) Distributed micro-service registration and calling method based on broadcast
CN107517227B (en) Session implementation method and device for distributed consistency system
CN112631788B (en) Data transmission method and data transmission server
CN113572835B (en) Data processing method, network element equipment and readable storage medium
WO2011015020A1 (en) File transmission method, system and client
WO2023000936A1 (en) Data processing method, function device and readable storage medium
CN116048538B (en) Service grid deployment method and device for DPU
CN112838940A (en) Network controller framework and data processing method
CN112671554A (en) Node fault processing method and related device
CN113259423A (en) Method and device for client networking access in P2P system
CN114025002A (en) MQTT information transmission method, system and communication equipment
CN114785695A (en) High-performance network communication library realized based on zeroC ICE
JP4373060B2 (en) Distributed processing system and proxy node, user side node and method in distributed processing system
CN110771117B (en) Session layer communication using ID-oriented network
CN114090342A (en) Storage disaster tolerance link management method, message execution node and storage control cluster
CN109413142B (en) Method for realizing iSCSI virtual agent under L inux
CN110661836B (en) Message routing method, device and system, and storage medium
CN116132530A (en) Method for realizing MQTT Broker server by applying Raft algorithm based on Netty framework
CN111083182B (en) Distributed Internet of things equipment management method and device
CN114844847A (en) High-reliability real-time message distribution method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication
WW01 Invention patent application withdrawn after publication

Application publication date: 20220722