CN115277846A

CN115277846A - Dynamic data fragmentation sending system based on ICE (Integrated Circuit emphasis) implementation

Info

Publication number: CN115277846A
Application number: CN202210889127.8A
Authority: CN
Inventors: 彭飞; 李升林
Original assignee: Shanghai Qianfang Technology Co ltd
Current assignee: Shanghai Qianfang Technology Co ltd
Priority date: 2022-07-27
Filing date: 2022-07-27
Publication date: 2022-11-01

Abstract

The invention relates to the technical field of universal data interaction, and particularly discloses a dynamic data fragmentation sending system realized based on ICE (Internet communications and Internet Explorer), which comprises: the system comprises a plurality of node modules, a plurality of server-side modules and a plurality of client-side modules, wherein an ICE communication frame is used as the node modules, the node modules are respectively provided with a plurality of server-side and a plurality of client-side, and the node modules are respectively provided with a glacier2 module; the glacier2 module is used as a firewall-passing and request routing tool to realize that client requests are concentrated on one port and then forwarded to a corresponding server. In the invention, each node in the private computing network is dynamically loaded, and the nodes need to communicate frequently in the process of computing tasks, and often need to mutually access the internal services of the private network through an untrusted network environment.

Description

Dynamic data fragmentation sending system based on ICE (Integrated Circuit emphasis) implementation

Technical Field

The invention relates to the technical field of data interaction, in particular to a dynamic data fragmentation sending system realized based on an ICE.

Background

ICE is an open source communication protocol product of ZEROC, and the full name is as follows: the Internet Communications Engine, which translates into chinese, is an Internet Communications Engine and an object-oriented middleware, enabling us to build distributed applications at minimal cost. ICE focuses on the development of application logic that deals with all underlying network interface programming, so we do not consider such details: open network connections, serialization and deserialization of network data transfers, number of attempts to fail a connection, etc. The Ice is an object-oriented middleware platform, which means that it provides tools, APIs, and library support for building object-oriented client-server applications. To communicate with an object held by Ice, the client must hold a proxy for the object (the same meaning as the CORBA reference), where the proxy refers to an instance of the object, ice locates the object at runtime, then finds or activates it, passes the In parameter to the remote object, and gets the return result through the Out parameter.

The proxy mentioned here is divided into direct proxy and indirect proxy, the direct proxy stores the mark of some object and the operation address of its server; the indirect proxy refers to a proxy which stores an identification of a certain object and an object adapter name (object adapter name), the indirect proxy does not contain addressing information, in order to correctly locate the server, the client uses the object adapter name inside the proxy during operation and transmits the object adapter name to a locator service, such as an Iceppack service, then the locator uses the adapter name as a keyword, the lookup is carried out in a table containing a server address, the current server address is returned to the client, the client run time knows how to contact the server, and the client request is dispatched (dispatch) as usual.

In a traditional private computing network, nodes are often classified into different private network environments, frequent network communication interaction between the nodes is required based on the requirement of computing tasks, and the services of the nodes are often classified into respective private networks to ensure network security through firewalls. The requests between nodes are mapped to specific service ports through firewall NAT of each private network.

In a distributed privacy computing network, each node is a server or a client, and each node is often in an untrusted network and has its own independent private network environment. Each private network is provided with a firewall to ensure the security, data interaction is frequently carried out between nodes under an untrusted network, and if data is lost or incomplete in the task calculation process, the task fails. It is therefore necessary to ensure both data security and data integrity in the above network environment.

The prior art has the following defects: the network operation and maintenance cost is high, and the network operation and maintenance cost is difficult to be continued in a complex network environment; the RPC frame is directly based on, and is often a 'question and answer' mode, and data caching is lacked; therefore, we propose a dynamic data fragmentation delivery system based on ICE implementation.

Disclosure of Invention

The invention aims to provide a dynamic data fragment sending system based on ICE realization, each node in a privacy computing network in the system is dynamically loaded, frequent communication is needed among the nodes in the computing task process, and the nodes often need to mutually access the internal service of the private network through an untrusted network environment; the difficulties of network disconnection, data loss, callback addressing and the like are faced under the condition of the complicated network; the invention can solve the problems only by little modification in the implementation of a specific scheme, thereby greatly simplifying the problem of information transmission in the complex network.

In order to achieve the purpose, the invention provides the following technical scheme:

a dynamic data fragmentation delivery system implemented on an ICE, the system comprising:

the system comprises a plurality of node modules, a plurality of server-side modules and a plurality of client-side modules, wherein an ICE communication frame is used as the node modules, the node modules are respectively provided with a plurality of server-side and a plurality of client-side, and the node modules are respectively provided with a glacier2 module; the glacier2 module is used as a firewall-passing and request routing tool, a client request is concentrated on one port and then forwarded to a corresponding server, the glacier2 module is provided with a client port and a service port, the service port is used for forwarding request information to the server and the client, and the client port is used for achieving bidirectional communication between the glacier2 module of different nodes and the client and the server.

In a preferred embodiment of the invention, the program interface in the system is designed to be idempotent.

As a preferred embodiment of the invention, the system supports the ssl protocol in a configuration mode to ensure the security of communication.

As a preferred embodiment of the present invention, the system sets the maximum thread number of the dynamic thread pool of the server and the idle thread recovery time in a configuration manner.

As a preferred embodiment of the present invention, a heartbeat mechanism module is disposed between different nodes in the system, and links between different nodes assist communication through the heartbeat mechanism module.

The system also comprises a Map + Queue container module used for caching data.

As a preferred embodiment of the present invention, the bidirectional communication interaction process is as follows:

a) The client configures a client port of the router, initiates a call request and establishes connection;

b) The router establishes connection with a corresponding server through the information provided by the client proxy and forwards the request;

c) The server side initiates callback information to the client side through a client port of the router;

d) The router forwards the callback request to the corresponding client through the bidirectional connection established in the first step;

e) The channel SDK uses a HashMap interface and queue container combination to cache data, and efficient storage and reading of the data are achieved.

As a preferred embodiment of the invention, the method uses nodeid + msgid as key to buffer the received data into a Map + Queue container combination to store the data.

Compared with the prior art, the invention has the beneficial effects that: in the invention, each node in the privacy computing network is dynamically loaded, and the nodes need to communicate frequently in the computing task process, and often need to mutually access the internal service of the private network through an untrusted network environment; the network disconnection, data loss, call-back addressing and other difficulties are faced under the complicated network condition; the invention can solve the problems only by little modification in the implementation of a specific scheme, thereby greatly simplifying the problem of information transmission in the complex network.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention.

Fig. 1 is an architecture diagram of a dynamic data slicing transmission system implemented based on ICE.

Fig. 2 is a block diagram of a procedure for guaranteeing a callback of a server using bidirectional communication and Glacier2 in a dynamic data fragmentation transmission system implemented based on ICE.

Detailed Description

In order to make the technical problems, technical solutions and advantageous effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The invention provides a dynamic data fragmentation sending system based on ICE realization, which comprises:

Further, the program interfaces in the present system are designed to be idempotent.

Furthermore, the system supports the ssl protocol in a configuration mode to ensure the security of communication.

Furthermore, the maximum thread number and the idle thread recovery time of the dynamic thread pool of the server are set in the system in a configuration mode.

Furthermore, a heartbeat mechanism module is arranged between different nodes in the system, and links between different nodes assist in communication through the heartbeat mechanism module.

Furthermore, the system also comprises a Map + Queue container module for data caching.

Furthermore, the two-way communication interaction process is as follows,

(1) the client configures a client port of the router, initiates a call request and establishes connection;

(2) the router establishes connection with a corresponding server through the information provided by the client proxy and forwards the request;

(3) the server side initiates callback information to the client side through a client port of the router;

(4) the router forwards the callback request to a corresponding client through the bidirectional connection established in the first step;

(5) the channel SDK uses a HashMap interface and queue container combination to cache data, and efficient storage and reading of the data are achieved.

Further, the method uses nodeid + msgid as key, and caches the received data in a Map + Queue container combination to store the data.

Example one

Referring to fig. 1-2, a dynamic data fragmentation delivery system implemented on ICE includes:

the method comprises the following steps that a plurality of ICE communication frames are taken as node modules, a plurality of service terminals and a plurality of client terminals are arranged in the node modules, and a glacier2 module is arranged in the node modules; wherein, the glacier2 module is used for realizing that the client request concentrates on a port and then forwards corresponding server as wearing firewall and request routing tool, the glacier2 module all is provided with customer port and service port, service port is used for giving server and client with the request information that forwards, customer port is used for realizing the glacier2 module of different nodes and the both way communication of client and server. Wherein the ICE can ensure that under any network environment or operating system, a successful call is only once, it will try to locate the remote server at runtime, and try to connect repeatedly in case of connection failure, and indeed even no longer give a prompt to the user. When the client calls the method of the server, the method can be realized in a synchronous or asynchronous mode, synchronous calling is equivalent to calling a local method, and other actions can be blocked; asynchronous calling is a very useful calling mode, for example, data required to be prepared by a server comes from other asynchronous interfaces, the client does not need to wait at this time, after the data of the server is sufficiently prepared, the client is notified in a message mode, the server can do other things, and the client can also obtain the data from the server.

In this embodiment, the program interface in the system is designed to be idempotent, and by means of the way that the program interface is designed to be idempotent, when ice runs, under abnormal conditions such as network disconnection and the like, automatic retry is performed to ensure the robustness and data integrity of the program.

In this embodiment, the system supports the SSL protocol to ensure the Security of communication through a configuration manner, where the system defaults to the TCP protocol, SSL is Secure Sockets Layer protocol, and Transport Layer Security (TLS) thereof is a Security protocol for providing Security and data integrity for network communication; TLS and SSL encrypt the network connection between the transmission layer and the application layer, which can effectively ensure the security of communication; the SSL protocol is positioned between the TCP/IP protocol and various application layer protocols and provides safety support for data communication. The SSL protocol can be divided into two layers: SSL recording Protocol (SSL Record Protocol): it is built on top of reliable transmission protocol (such as TCP), and provides basic functions of data encapsulation, compression, encryption and the like for higher-layer protocol. SSL Handshake Protocol (SSL Handshake Protocol): it is established on SSL record protocol, and is used for making identity authentication, negotiation encryption algorithm and exchange encryption key, etc. by two communication parties before actual data transmission is started.

In this embodiment, the system sets the maximum thread number of the dynamic thread pool of the server and the idle thread recovery time (ice.

In this embodiment, a heartbeat mechanism module is arranged between different nodes in the system, and links between different nodes assist communication through the heartbeat mechanism module, wherein the heartbeat mechanism is a mechanism that regularly sends a self-defined structure (heartbeat packet) to let the other side know that the other side is still alive so as to ensure the validity of connection, and the heartbeat packet is a mechanism that a client regularly sends simple information to a server side to tell that the client side still stays; the code is that a fixed message is sent to the server every few minutes, the server replies a fixed message after receiving the fixed message, and if the server does not receive the client message within a few minutes, the client is disconnected; for example, some communication software is not used for a long time, a heartbeat packet is needed to know whether the state of the communication software is online or offline, and a packet receiving packet is sent regularly; a bag sending party: the client can be the client or the server side, and the realization of which is convenient and reasonable is realized; typically a client; the server can also poll the heartbeat down at regular time; the heartbeat packet is called as the heartbeat packet because: it sends the message once every fixed time like heartbeat, so as to tell the server that the client is still alive; this is in fact to maintain a long connection, as far as the content of this packet is not specified, but is generally a very small packet, or an empty packet containing only the header; in this embodiment, different nodes pass through while communicating.

In this embodiment, the system further includes a Map + Queue container module for performing data caching.

Example two

Referring to fig. 2, the two-way communication interaction process is as follows,

(1) a Client configures a Client port of a Router, initiates a call request and establishes connection, wherein the Router is a Router, and the Client port is Client Endpoints;

(3) the server side initiates callback information to the client side through a client port of a Router, wherein the Router is a Router, and the service port is server Endpoints;

(4) the router forwards the callback request to the corresponding client through the bidirectional connection established in the first step;

(5) the Channel SDK caches data by using a HashMap interface and Queue container combination to realize efficient storage and reading of the data, wherein the Channel SDK is a Channel SDK, and the HashMap interface and Queue container combination is a HashMap and Queue container which can guarantee the data receiving sequence.

In this embodiment, the method uses nodeid + msgid as a key, and buffers the received data into a Map + Queue container combination to store the data, where nodeid + msgid is a node number and a message identifier, respectively.

To sum up, each node in the private computing network is dynamically loaded, frequent communication is required among the nodes in the computing task process, and the nodes often need to mutually access the internal services of the private network through an untrusted network environment; the difficulties of network disconnection, data loss, callback addressing and the like are faced under the condition of the complicated network; the invention can solve the problems only by little modification in the implementation of a specific scheme, thereby greatly simplifying the problem of information transmission in the complex network.

The processor fetches instructions and analyzes the instructions one by one from the memory, then completes corresponding operations according to the instruction requirements, generates a series of control commands, enables all parts of the computer to automatically, continuously and coordinately act to form an organic whole, realizes the input of programs, the input of data, the operation and the output of results, and the arithmetic operation or the logic operation generated in the process is completed by the arithmetic unit; the Memory comprises a Read-Only Memory (ROM) for storing a computer program, and a protection device is arranged outside the Memory.

Illustratively, a computer program can be partitioned into one or more modules, which are stored in memory and executed by a processor to implement the present invention. One or more of the modules may be a series of computer program instruction segments capable of performing certain functions, which are used to describe the execution of the computer program in the terminal device.

Those skilled in the art will appreciate that the above description of the service device is merely exemplary and not limiting of the terminal device, and may include more or less components than those described, or combine certain components, or different components, such as may include input output devices, network access devices, buses, etc.

The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like, which is the control center of the terminal equipment and connects the various parts of the entire user terminal using various interfaces and lines.

The memory may be used to store computer programs and/or modules, and the processor may implement various functions of the terminal device by operating or executing the computer programs and/or modules stored in the memory and calling data stored in the memory. The memory mainly comprises a storage program area and a storage data area, wherein the storage program area can store an operating system, application programs (such as an information acquisition template display function, a product information publishing function and the like) required by at least one function and the like; the storage data area may store data created according to the use of the berth-state display system (e.g., product information acquisition templates corresponding to different product types, product information that needs to be issued by different product providers, etc.), and the like. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.

The terminal device integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the modules/units in the system according to the above embodiment may be implemented by a computer program, which may be stored in a computer-readable storage medium and used by a processor to implement the functions of the embodiments of the system. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signal, telecommunications signal, software distribution medium, etc.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a component of' 8230; \8230;" does not exclude the presence of another like element in a process, method, article, or apparatus that comprises the element.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all equivalent structures or equivalent processes performed by the present invention or directly or indirectly applied to other related technical fields are also included in the scope of the present invention.

Claims

1. A dynamic data slicing transmission system implemented based on ICE, the system comprising:

the system comprises a plurality of node modules, a plurality of server-side modules and a plurality of client-side modules, wherein an ICE communication frame is used as the node modules, the node modules are respectively provided with a plurality of server-side and a plurality of client-side, and the node modules are respectively provided with a glacier2 module;

the glacier2 module is used as a firewall-passing and request routing tool to realize that client requests are concentrated on one port and then forwarded to the corresponding server, and the glacier2 module is provided with a client port and a service port; the service port is used for transmitting the request information to the service end and the client, and the client port is used for realizing the two-way communication between the glacier2 modules of different nodes and the client and the service end.

2. The ICE-based implementation of dynamic data slicing transmission system according to claim 1, wherein the program interface in the transmission system is designed to be idempotent.

3. The ICE-based dynamic data slicing sending system of claim 2, wherein the sending system supports ssl protocol to ensure communication security by way of configuration.

4. The ICE-based dynamic data slicing sending system of claim 3, wherein the sending system sets the maximum thread number of the server dynamic thread pool and the idle thread recovery time in a configuration manner.

5. The ICE-based dynamic data fragmentation delivery system of claim 4 in which a heartbeat mechanism module is provided between different nodes of the delivery system, and links between different nodes facilitate communication via the heartbeat mechanism module.

6. The ICE-based dynamic data fragment sending system of claim 5, further comprising a to Map + Queue container module for data caching.

7. The ICE-based dynamic data slicing sending system of claim 6, wherein the bidirectional communication interaction flow is as follows:

the client configures a client port of the router, initiates a call request and establishes connection;

the router establishes connection with a corresponding server through the information provided by the client proxy and forwards the request;

the server side initiates callback information to the client side through a client port of the router;

the router forwards the callback request to the corresponding client through the bidirectional connection established in the first step;

the channel SDK uses a HashMap interface and queue container combination to cache data, and efficient storage and reading of the data are achieved.

8. The ICE-based dynamic data slicing sending system of claim 7, wherein the system uses nodeid + msgid as key to buffer the received data into Map + Queue container combination to save the data.