CN114024986B

CN114024986B - Lightweight network simulation method using node proxy

Info

Publication number: CN114024986B
Application number: CN202111298005.3A
Authority: CN
Inventors: 周国强; 许张贤; 张卫丰
Original assignee: Nanjing University of Posts and Telecommunications
Current assignee: Nanjing University of Posts and Telecommunications
Priority date: 2021-11-04
Filing date: 2021-11-04
Publication date: 2023-07-11
Anticipated expiration: 2041-11-04
Also published as: CN114024986A

Abstract

The invention discloses a lightweight network simulation method using a node proxy. The method comprises the following steps: the network simulation configuration center reads json configuration files and converts the contents of the configuration files into corresponding network configuration parameters; creating a proxy node for each node in the process; the connection between nodes is completed by an RPC framework; each node in the process establishes connection with all proxy nodes; initializing node number and node socket mapping and providing the node number and node socket mapping for the proxy node to search; the mapping of the node numbers and the sockets is stored in a network simulation configuration center; inter-node communication is diverted by proxy nodes; the network simulation is implemented by the proxy node according to the network simulation configuration center parameters; the network data is counted by the proxy node; the network data is written into a network simulation configuration center; the network simulation configuration center receives a network configuration parameter modification request; and the network simulation configuration center receives the test ending command and closes the whole process. The invention effectively simulates the production network environment in a single-process multi-node test environment, and provides a lightweight network environment simulation for the distributed system.

Description

Lightweight network simulation method using node proxy

Technical Field

The invention relates to the field of distributed technology, in particular to a lightweight network simulation method in a test environment.

Background

With the increasing speed of single equipment performance and the increasing size of data processing of large enterprises, more and more organizations begin to utilize distributed technology to complete the computation and storage of large-scale data. To verify that the program is in line with expectations, developers must conduct continuous integration testing during development, however distributed nodes in the production environment may reach hundreds. In order to simulate a production network, the large-scale test is required to call very much hardware resources to achieve the effect, so that not only is the resources heavy, but also the management is difficult, and therefore, how to have a lightweight network simulation test environment is urgent for research personnel.

Most distributed technologies employ remote procedure calls, i.e., RPC technology. The current RPC framework, such as GRPC, BRPC, etc., cannot simulate network interruption, delay, packet loss and network partition, and all lack network data statistics means, so we cannot implement network environment simulation and statistics from the RPC level. At the same time, it is also unacceptable if the original distributed program is invasively altered in order to achieve network simulation and network data statistics.

Disclosure of Invention

The invention aims at overcoming the defects of the prior art and providing a lightweight network simulation method using a node proxy.

The lightweight network simulation method using the node proxy is characterized by comprising the following steps of:

s1: the network simulation configuration center reads json configuration files and converts the contents of the configuration files into corresponding network configuration parameters;

s2: creating a proxy node for each node in the process;

s3: initializing an RPC framework of the node;

s4: each node in the process establishes connection with all proxy nodes;

s5: initializing node number and node socket mapping and providing the node number and node socket mapping for the proxy node to search;

s6: the mapping between the node numbers and the sockets is stored in a network simulation configuration center;

s7: inter-node communication is diverted by proxy nodes;

s8: the network simulation is implemented by the proxy node according to the network simulation configuration center parameters;

s9: the network data is counted by the proxy node;

s10: the network data is written into a network simulation configuration center;

s11: the network simulation configuration center receives a network configuration parameter modification request;

s12: and the network simulation configuration center receives the test ending command and closes the whole process.

Preferably, in the step S1, the network simulation configuration center reads a json configuration file, where the configuration file includes an overall network delay size, a packet loss rate, and a network partition condition, includes a node network delay size, a packet loss rate, whether blocking occurs, and whether out-of-order arrival occurs, and these parameter information is read for future viewing.

Preferably, in the step S2, the proxy node is only a node that receives data instead, and does not actively send data to other nodes.

Preferably, the proxy node receives data for the node, then looks up the current network simulation parameters, and finally decides to perform operations such as normal forwarding, delay, packet loss or out-of-order arrival on the data.

Preferably, in the step S3, it is characterized in that: the same RPC framework should be used between nodes, such as grpc or brpc; and initializing the RPC framework and determining an externally provided calling interface.

Preferably, in the step S4, the node establishes connection with all the proxy nodes, the node wants to send data to a certain node, the proxy node of the target should be taken as a sending target, and the proxy node takes charge of transferring the data as an intermediate role.

Preferably, in the step S5, the node obtains a node number when initializing, and the number starts from 0 and then increases backward in sequence with 1. Meanwhile, the node sets a port for RPC communication and can be connected with the IP address to form a socket.

Preferably, the proxy node receives an RPC that can determine the source by looking at the socket of the RPC initiator and then looking at the network configuration, determining how to forward the data.

Preferably, in the step S6, the network simulation configuration center stores the node number and the socket by using the map data type in c++, and the proxy node needs to apply to the network simulation configuration center when needing to look up the socket and the number mapping, and the result is returned from the latter.

Preferably, in the step S7, the method specifically includes the following steps:

s701: the node sends RPC request to the proxy node of the target node

S702: the proxy node receives the RPC request and acquires the current network configuration from the configuration center

S703: the proxy node obtains the node number of the RPC initiator from the configuration center and obtains the current state of the node

S704: and according to the RPC initiating node state, deciding to discard, delay or disorder operation on the data packet.

S705: discarding, delaying or out-of-order operations are performed on the data packets according to the network environment.

S706: and according to the state of the target node, deciding to discard, delay or disorder the data packet.

S707: and recording the size of the transferring data, updating the quantity of the transferring RPCs, and storing the quantity in the agent local.

S708: the proxy node initiates an RPC request to the target node.

S709: the proxy node receives the RPC response of the target node and returns the RPC response to the RPC initiator node.

Preferably, in the step S8, the method specifically includes the following steps:

s801: checking whether packet loss operation is performed on the data packet.

S802: and discarding the RPC when the packet is lost, directly ending the operation, otherwise, turning to the next step.

S803: see if delay action is taken on the data packet.

S804: and after delay, sleeping randomly for a period of time, and after awakening, putting the data packet into a transmission queue, otherwise, directly transferring to the next step.

S805: see if out-of-order send operations are to be taken on the data packets.

S806: transmitting all data packets in the transmission queue to the target node

Preferably, in the step S9, the method specifically includes the following steps:

s901: the proxy node receives the RPC request

S902: the proxy node calculates the data size carried by the RPC

S903: proxy node updating total forwarding data size

S904: proxy node updating forwarding RPC quantity

S904: the proxy node stores the statistical data in the memory in the form of long type

Preferably, in the step S10, the network data is counted in the proxy node, and the proxy node periodically writes the data back to the network simulation configuration center, and the latter periodically stores the data in json format in the disk.

Preferably, the proxy node updates the data to the network simulation configuration center by using an incremental update method.

Preferably, in the step S11, the method specifically includes the following steps:

s1101: and the user sends a request for modifying the network simulation parameters to the network simulation configuration center by using RPC, HTTP and other modes.

S1102: the network simulation configuration center modifies the local parameters stored in the memory.

S1103: the network simulation configuration center sends notification that the network configuration parameters are modified to the proxy node through RPC, HTTP and the like.

S1104: the agent node receives the notification and adopts the latest network simulation configuration.

Preferably, in step S12, the network simulation configuration center sends the tested request to all nodes including proxy nodes by using RPC, HTTP, etc., waits for the replies of all nodes, and then ends the process after confirming that all nodes are closed.

Compared with the prior art, the invention has the following beneficial effects:

the invention solves the network simulation problem in the distributed test environment, and successfully simulates the phenomena of delay, packet loss, disorder, network partition and the like in the production environment network.

The invention can be implemented on the premise of no invasion to the prior art, effectively reduces the complexity of work and reduces the maintenance workload of the simulation environment.

Drawings

FIG. 1 is a schematic illustration of test environment initialization for a lightweight network simulation approach of the present invention using node agents

FIG. 2 is a block diagram of forwarding details for a lightweight network modeling approach of the present invention using node proxies

FIG. 3 illustrates parameter adjustment and system exit for a lightweight network simulation approach using node agents in accordance with the present invention

Detailed Description

The invention will be described in further detail below with reference to the drawings and examples, but the invention is not limited to the examples

As shown in fig. 1, the initialization of the test environment of the lightweight network simulation method using the node agent in the invention comprises the following specific steps:

step S1: the network simulation configuration center reads corresponding configuration files through a system call or network remote access mode, wherein the configuration files comprise the whole network delay size, the packet loss rate and the network partition condition, the node network delay size, the packet loss rate, whether blocking occurs or not and whether out-of-order arrival occurs or not, and all the parameter information is recorded in the configuration files in json format.

The configuration center can analyze the configuration file through a json analysis tool such as simplejson and the like, then creates a class instance of the management configuration file locally, and transmits the parameters successfully analyzed as real parameters to the configuration file class constructor parameters to initialize instance classes in sequence.

Step S2: the network configuration center creates a corresponding proxy node for each node. The proxy node is responsible for receiving data for the node to the foreign agent, and is also the only visible to other nodes. In order to be completely transparent to the outside, the proxy node registers all the RPC functions registered by the node, but the internal logic of the RPC functions registered by the proxy node is completely different from that of the node, which is also the key point that the proxy node can achieve non-invasive network simulation.

Step S3: the RPC frame of all nodes is initialized, which includes registering the services provided by the nodes to the outside, and then waiting for the external call request.

Step S4: because the receipt of node data is done by means of proxy nodes, each node in the process establishes a connection with all proxy nodes. The proxy node is transparent to other nodes, and the other nodes consider the proxy node to be the target node, so that the non-invasive effect of the test environment on the original program can be achieved.

The proxy node is responsible for receiving data, and refers to the configuration of the network simulation center, and performs corresponding actions on the received data according to the configuration. In order to achieve the network interruption effect, the agent directly discards the data packet; to achieve the network delay effect, the proxy will sleep for a period of time before forwarding the packet; in order to achieve the purpose of achieving the out-of-order arrival of the data packets, the agent places all the data packets to be forwarded into a queue and breaks the data packets; to achieve the network partition effect, the proxy looks at the source and destination of the data packet and decides to forward or discard the data packet according to the network partition configuration.

Step S5: when initializing the node, the network configuration center will do mapping operation to the node number and node socket. The node numbers start from 0 and then increment backward in sequence 1,2,3 and …, so that the node stores the entity communicating with other nodes in the array, and the communication object can be found rapidly through the target node number.

For the proxy node to only acquire the IP address and port number of the sender upon receipt of the RPC forwarding request, in order to determine the sender at the network simulation level, the proxy node must query the network simulation configuration center, which will look up the mapping and then return the RPC request forwarder number to the proxy node.

Step S6: the node numbers and node socket mappings may be stored in a network simulation configuration center, which may store two parts using map functions in a c++ standard library, one part having the node numbers and node sockets as key value pairs and the other part having the node sockets and node numbers as key value pairs. The map function is internally implemented by a red-black tree, which ensures that the longest path length does not exceed twice either path length. This is a balanced binary tree that relaxes the requirements. Because the regulation of tree adjustment is relaxed, the red-black tree is not adjusted as frequently as the balanced binary tree, and the performance of data deletion and insertion can be improved.

As shown in fig. 2, the forwarding details of the lightweight network simulation method using the node proxy in the invention are as follows:

step S7: the node sends an RPC request, which specifically comprises the following steps:

s701: the node requires some RPC function of the external node.

S702: the node takes an array in which the external node communication instance is stored.

S703: the node takes the communication instance of the target node from the array according to the number.

S703: the node uses the communication instance to call the corresponding request function and send the RPC request.

Step S8: inter-node communication is diverted by the proxy node, and the RPC request issued in S7 is destined for some external node from a logical point of view, but is actually received by the proxy node of the corresponding node from a practical point of view.

The proxy node performs network simulation actions after receiving the RPC request and then selects to discard or forward the RPC to the target node.

Step S9: the proxy node reads the configuration from the network simulation configuration center during initialization, and performs corresponding actions according to the network configuration stored locally after receiving the RPC, and the specific steps are as follows:

s901: checking whether the network is not available, if yes, discarding the RPC, otherwise, going to the next step.

S902: checking whether the source node or the target node is in a different network partition, if so, discarding the RPC, otherwise, turning to the next step.

S903: checking whether the source node or the target node is blocked, if so, discarding the RPC, otherwise, going to the next step.

S904: checking whether the network has delay, if so, randomly sleeping for a period of time and then turning to the next step, otherwise, directly turning to the next step.

S905: checking whether the source node or the target node is started for delay, if yes, randomly sleeping for a period of time and then turning to the next step, otherwise, directly turning to the next step.

S906: checking whether the data disorder is started, if so, disturbing the data in the sending queue, otherwise, turning to the next step.

Step S10: the agent node counts the network data, and the specific steps are as follows:

s101: looking at the source and target nodes of the RPC.

S102: and calculating the data size of the RPC data packet.

S103: the total data volume of the network, the data volume sent by the RPC sender and the data volume received by the RPC receiver are updated.

S104: the time of the RPC from receipt of the successful forwarding is tracked and this time is taken as a delay into the statistics of the average delay of the network.

S105: periodically synchronizing data in incremental updates to a network simulation configuration center

As shown in fig. 3, the method for adjusting parameters and exiting the system by using the lightweight network simulation method of the node agent comprises the following specific steps:

step S11: and starting the test system, initializing the system, and then automatically running the system, and waiting for a user instruction in the period.

Step S12: the user sends the instruction by means of RPC or HTTP.

Step S13: the system checks whether it is a network parameter modification instruction, if so, it goes to S14, otherwise it goes to S15.

Step S14: after receiving the modification request, the test system modifies the network parameters stored in the memory, and specifically comprises the following steps:

s1401: the type of parameter that needs to be modified is checked.

S1402: network parameters stored in the memory are modified.

S1403: the proxy node is informed that the network parameters are changed through communication modes such as RPC or HTTP and the like, and the parameters needing to be modified are carried in the message.

Step S15: the system checks whether it is the instruction to close the test, if so, it goes to S16, otherwise it goes to S12 to wait for the user to issue the command.

Step S16: and the network simulation configuration center sends a closing command to all nodes through the RPC, waits for the closed replies of all nodes, and exits the process after receiving the closed replies of all nodes.

The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and variations could be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and variations should also be regarded as being within the scope of the invention.

Claims

1. A lightweight network simulation approach using node agents, comprising the steps of:

s2: creating a proxy node for each node in the process;

s3: initializing a Remote Procedure Call (RPC) framework of a node;

s4: each node in the process establishes connection with all proxy nodes;

s7: inter-node communication is diverted by proxy nodes;

s9: the network data is counted by the proxy node;

s12: the network simulation configuration center receives the test ending command and closes the whole process;

in the step S1, the network simulation configuration center reads json configuration files, wherein the configuration files comprise the overall network delay size, the packet loss rate and the network partition condition, and the configuration files comprise the node network delay size, the packet loss rate, whether blocking occurs and whether out-of-order arrival occurs, and the parameter information is read for future checking;

in the step S2, the node agent is only a node agent for receiving data, and it will not actively send data to other nodes;

the node agent receives data for the node, then checks the current network simulation parameters, and finally decides to adopt normal forwarding, delay, packet loss or out-of-order arrival operation for the data.

2. The lightweight network simulation approach using node agents according to claim 1, wherein: in the step S3, the method is characterized in that: the same RPC framework is used among nodes; and initializing the RPC framework and determining an externally provided calling interface.

3. The lightweight network simulation approach using node agents according to claim 1, wherein: in step S4, the node establishes connection with all node agents, the node wants to send data to a certain node, the node agent of the target should be used as a sending target, and the node agent is used as a man-in-the-middle role to take charge of transferring data.

4. The lightweight network simulation approach using node agents according to claim 1, wherein: in step S5, the node obtains a node number when initializing, the number starts from 0, then 1, and increases backwards in sequence, and meanwhile, the node sets a port for RPC communication, and is connected with the IP address to form a socket.

5. The lightweight network simulation approach using node agents of claim 4, wherein: the node proxy receives an RPC, determines the source by looking at the socket of the RPC initiator, and then looks at the network configuration to determine how to forward the data.

6. The lightweight network simulation approach using node agents according to claim 1, wherein: in step S6, the network simulation configuration center stores the node number and the socket by using the map data type in c++, and the node agent needs to apply to the network simulation configuration center when the socket and the number need to be checked, and the result is returned by the network simulation configuration center.

7. The lightweight network simulation method using node proxy according to claim 1, wherein in step S7, the method specifically comprises the following steps of inter-node communication, and further comprises:

s701: the node sends an RPC request to a proxy node of the target node;

s702: the agent node receives the RPC request and acquires the current network configuration from the configuration center;

s703: the agent node obtains the node number of the RPC initiator from the configuration center and obtains the current state of the node;

s704: determining to discard, delay or disorder operation on the data packet according to the RPC initiating node state;

s705: discarding, delaying or out-of-order operation is adopted for the data packets according to the network environment;

s706: determining to discard, delay or disorder the data packet according to the state of the target node;

s707: recording the size of the transferring data, updating the quantity of the transferring RPCs, and storing the quantity in a node agent local;

s708: the node agent initiates an RPC request to the target node;

s709: the node proxy receives the RPC response of the target node and returns the RPC response to the RPC initiator node.

8. The lightweight network simulation approach using node agents according to claim 1, wherein: the step S8 specifically includes the following steps:

s801: checking whether packet loss operation is adopted for the data packet;

s802: discarding the RPC when the packet is lost, directly ending the operation, otherwise, turning to the next step;

s803: checking whether delay operation is performed on the data packet;

s804: after delay, sleeping randomly for a period of time, and after awakening, putting the data packet into a transmission queue, otherwise, directly transferring to the next step;

s805: checking whether out-of-order sending operation is adopted for the data packet;

s806: and transmitting all the data packets in the transmission queue to the target node.

9. The lightweight network simulation approach using node agents according to claim 1, wherein: in the step S10, the network statistics are counted in the node agent, and the node agent periodically stores the network statistics in the disk according to json format.

10. The lightweight network simulation approach using node agents according to claim 9, wherein: the node proxy update data adopts an incremental update method.

11. The lightweight network simulation method using node agents according to claim 1, wherein ≡s11 comprises the steps of:

s1101: the user sends a request for modifying the network simulation parameters to the network simulation configuration center in an RPC mode;

s1102: the network simulation configuration center modifies the local parameters stored in the memory;

s1103: the network simulation configuration center sends a notification that the network configuration parameters are modified to the node agent in an RPC mode;

s1104: the node agent receives the notification and adopts the latest network simulation configuration.

12. The lightweight network simulation approach using node agents according to claim 1, wherein: in step S12, the network simulation configuration center sends a request of completing the test to all nodes by using an RPC method, waits for replies of all nodes, and finishes the process after confirming that all nodes are closed.

13. The lightweight network simulation method using node agents according to claim 1, wherein in step S9, the method specifically comprises the following steps:

s901: the node agent receives the RPC request;

s902: the node agent calculates the data size carried by the RPC;

s903: the node agent updates the total forwarding data size;

s904: the node agent updates the number of forwarding RPCs;

s905: the node agent stores the statistics in memory in long type.