CN114024986A

CN114024986A - Lightweight network simulation method using node agent

Info

Publication number: CN114024986A
Application number: CN202111298005.3A
Authority: CN
Inventors: 周国强; 许张贤; 张卫丰
Original assignee: Nanjing University of Posts and Telecommunications
Current assignee: Nanjing University of Posts and Telecommunications
Priority date: 2021-11-04
Filing date: 2021-11-04
Publication date: 2022-02-08
Anticipated expiration: 2041-11-04
Also published as: CN114024986B

Abstract

The invention discloses a lightweight network simulation method using a node agent. The method comprises the following steps: a network simulation configuration center reads a json configuration file and converts the content of the configuration file into corresponding network configuration parameters; creating a proxy node for each node in the process; the connection between the nodes is completed by an RPC frame; each node in the process establishes connection with all the agent nodes; initializing node number and node socket mapping and providing the mapping for the proxy node for searching; the mapping of the node number and the socket is stored in a network simulation configuration center; the communication between the nodes is transferred by the proxy node; the network simulation is implemented by the agent node according to the parameters of the network simulation configuration center; network data is counted by the agent node; network data is written into a network simulation configuration center; a network simulation configuration center receives a network configuration parameter modification request; and the network simulation configuration center receives the test ending command and closes the whole process. The invention effectively simulates the production network environment in the single-process multi-node test environment and provides a lightweight network environment simulation for the distributed system.

Description

Lightweight network simulation method using node agent

Technical Field

The invention relates to the field of distributed technology, in particular to a method for simulating a lightweight network in a test environment.

Background

With the slow performance improvement of a single device and the continuous increase of the data processing scale of a large-scale enterprise, more and more organizations begin to utilize a distributed technology to complete the calculation and storage of large-scale data. To verify that a program is as expected, developers must conduct ongoing integration tests during development, however, distributed nodes in a production environment can reach hundreds. In order to simulate a production network, such a large-scale test inevitably requires to call a great number of hardware resources to achieve an effect, which is not only heavy resources but also difficult to manage, so that how to have a lightweight network simulation test environment is a critical task for research and development personnel.

Most distributed technologies employ remote procedure calls, i.e., RPC technology. The current RPC frameworks such as GRPC, BRPC, etc. cannot simulate the network interruption, delay, packet loss and network partition, and all lack network data statistics means, so we cannot implement network environment simulation and statistics from the RPC level. At the same time, it is unacceptable to make invasive changes to the original distributed program in order to implement network simulation and network data statistics.

Disclosure of Invention

The invention aims to provide a lightweight network simulation method using a node agent aiming at the defects of the prior art.

The lightweight network simulation method using the node agent is characterized by comprising the following steps of:

s1: a network simulation configuration center reads a json configuration file and converts the content of the configuration file into corresponding network configuration parameters;

s2: creating a proxy node for each node in the process;

s3: initializing an RPC frame of a node;

s4: each node in the process establishes connection with all the agent nodes;

s5: initializing node number and node socket mapping and providing the mapping for the proxy node for searching;

s6: the mapping of the node number and the socket is stored in a network simulation configuration center;

s7: the communication between the nodes is transferred by the proxy node;

s8: the network simulation is implemented by the agent node according to the parameters of the network simulation configuration center;

s9: network data is counted by the agent node;

s10: network data is written into a network simulation configuration center;

s11: a network simulation configuration center receives a network configuration parameter modification request;

s12: and the network simulation configuration center receives the test ending command and closes the whole process.

Preferably, in step S1, the network simulation configuration center reads the json configuration file, where the configuration file includes the overall network delay size, the packet loss rate, and the network partition condition, and includes the node network delay size, the packet loss rate, whether to block, and whether to arrive out of order, and these parameter information are read for future viewing.

Preferably, in the step S2, the proxy node only receives data for the node instead, and it does not actively send data to other nodes.

Preferably, the proxy node receives data for the node, then checks the current network simulation parameter, and finally determines to perform operations such as normal forwarding, delay, packet loss or out-of-order arrival on the data.

Preferably, in step S3, the method further includes: the same RPC frame, such as grpc or brpc, etc., should be used between nodes; initializing the RPC framework and determining a calling interface provided externally.

Preferably, in step S4, the node establishes connections with all proxy nodes, and when the node intends to send data to a certain node, the target proxy node should be used as a sending target, and the proxy node is responsible for transferring data as a broker role.

Preferably, in step S5, the node obtains a node number at initialization, and the number increases from 0 to 1. Meanwhile, the node sets a port for RPC communication, and can become a socket when being connected with the IP address of the node.

Preferably, the proxy node receives an RPC, which can determine the source by looking at the socket of the RPC initiator, and then looking at the network configuration, determine how to forward the data.

Preferably, in step S6, the network simulation configuration center stores the node number and the socket by using the map data type in C + +, and the agent node needs to apply for the network simulation configuration center when needing to check the mapping between the socket and the number, and the result is returned by the latter.

Preferably, the step S7 specifically includes the following steps:

s701: node sends RPC request to proxy node of target node

S702: the agent node receives the RPC request and obtains the current network configuration from the configuration center

S703: the agent node obtains the node number of the RPC initiator from the configuration center and obtains the current state of the node

S704: and deciding to discard, delay or carry out-of-order operation on the data packet according to the state of the RPC initiating node.

S705: the data packets are discarded, delayed or out of order according to the network environment.

S706: and deciding to take discarding, delaying or out-of-order operation on the data packet according to the state of the target node.

S707: and recording the size of the transfer data, updating the number of the transfer RPC, and storing the transfer RPC in the local agent.

S708: the proxy node initiates an RPC request to the target node.

S709: and the proxy node receives the RPC response of the target node and returns the RPC response to the RPC initiator node.

Preferably, the step S8 specifically includes the following steps:

s801: and checking whether packet loss operation is carried out on the data packet.

S802: and if the packet is lost, discarding the RPC, and directly ending the operation, otherwise, turning to the next step.

S803: to see if a delay action is taken on the packet.

S804: and if the delay is delayed, randomly sleeping for a period of time, awakening and then putting the data packet into a sending queue, otherwise, directly switching to the next step.

S805: and checking whether out-of-order sending operation is adopted for the data packet.

S806: all the data packets in the sending queue are sent to the target node

Preferably, the step S9 specifically includes the following steps:

s901: the proxy node receives the RPC request

S902: the proxy node calculates the size of data carried by RPC

S903: proxy node updating total forwarding data size

S904: proxy node updates the number of RPC forwarding

S904: the agent node stores the statistical data into the memory in the long type

Preferably, in step S10, the network data is counted in the proxy node, and the proxy node periodically writes the data back to the network simulation configuration center, which also periodically stores the data in the disk according to json format.

Preferably, the agent node updates data to the network simulation configuration center by adopting an incremental updating method.

Preferably, the step S11 specifically includes the following steps:

s1101: the user sends a request for modifying the network simulation parameters to the network simulation configuration center by means of RPC, HTTP and the like.

S1102: and the network simulation configuration center modifies the local parameters stored in the memory.

S1103: and the network simulation configuration center sends a notice that the network configuration parameters are modified to the proxy node in an RPC (remote procedure call), HTTP (hyper text transport protocol) mode and other modes.

S1104: the agent node receives the notification and adopts the latest network simulation configuration.

Preferably, in step S12, the network simulation configuration center sends a test-completed request to all nodes, including the proxy node, by using RPC, HTTP, and the like, waits for replies from all nodes, and ends the process after confirming that all nodes are closed.

Compared with the prior art, the invention has the following beneficial effects:

the invention solves the problem of network simulation in a distributed test environment, and successfully simulates the phenomena of delay, packet loss, disorder, network partition and the like in a production environment network.

The invention can be implemented on the premise of no invasion to the prior art, thereby effectively reducing the complexity of work and reducing the maintenance workload of the simulation environment.

Drawings

FIG. 1 is a test environment initialization for lightweight network simulation using node proxies in accordance with the present invention

FIG. 2 is a forwarding detail of the lightweight network simulation approach using node proxies according to the present invention

FIG. 3 shows parameter adjustment and system exit for lightweight network simulation using node proxies according to the present invention

Detailed Description

The invention is explained in more detail below with reference to the drawings and the exemplary embodiments, without thereby restricting the invention to the scope of the exemplary embodiments described

As shown in fig. 1, the test environment initialization of the lightweight network simulation method using the node agent of the present invention includes the following specific steps:

step S1: the network simulation configuration center reads corresponding configuration files through a system call or network remote access mode, the configuration files comprise the whole network delay size, the packet loss rate and the network partition condition, the node network delay size, the packet loss rate, whether the node network is blocked or not and whether the node network arrives out of order, and the parameter information is recorded in the configuration files in a json format.

The configuration center can analyze the configuration file through a json analysis tool such as simplejson and the like, then create a class instance for managing the configuration file locally, and transmit parameters which are successfully analyzed as actual parameters to the configuration file class and construct function parameters to initialize the instance class in sequence.

Step S2: the network configuration center creates a corresponding proxy node for each node. The proxy node is responsible for receiving data for the node to the foreign agent and is also the only visible node to other nodes. In order to be completely transparent to the outside, the proxy node registers all RPC functions registered by the node, but the internal logic of the RPC functions registered by the proxy node is completely different from that of the node, which is the key point that the proxy node can perform non-intrusive network simulation.

Step S3: initializing RPC frameworks of all nodes, wherein services provided by registered nodes to the outside are contained, and then waiting for external call requests.

Step S4: because the reception of node data is done by means of the proxy nodes, each node in the process establishes a connection with all the proxy nodes. The agent node is transparent to other nodes, and the other nodes consider the agent node to be the target node, so that the non-invasive effect of the test environment on the original program can be achieved.

The agent node is responsible for receiving data, and the agent node can make corresponding action on the received data according to the configuration of the network simulation center. In order to achieve the network interruption effect, the agent directly discards the data packet; to achieve the network delay effect, the agent sleeps for a period of time before forwarding the packet; in order to achieve the purpose of out-of-order arrival of data packets, an agent puts all data packets to be forwarded into a queue and scrambles the data packets; to achieve the network partition effect, the agent looks at the source and destination of the packet and decides to forward or drop the packet according to the network partition configuration.

Step S5: when initializing the node, the network configuration center performs mapping operation on the node number and the node socket. The node number is sequentially increased backwards from 0 and then 1, 2 and 3 …, so that the node stores the entities communicating with other nodes in the array, and the communication object can be quickly found by the target node number.

For the proxy node, it can only obtain the sender's IP address and port number when it receives the RPC forwarding request, and in order to determine the sender at the network emulation level, the proxy node must query the network emulation configuration center, which looks at the mapping and then returns the RPC request forwarder's number to the proxy node.

Step S6: the node number and the node socket mapping can be stored in a network simulation configuration center, the configuration center can use a map function in a C + + standard library to store two parts, one part uses the node number and the node socket as a key-value pair, and the other part uses the node socket and the node number as the key-value pair. The inside of the map function is realized by a red and black tree, and the longest path length can be ensured not to exceed twice of any path length. This is a balanced binary tree that relaxes the requirements. Because the regulation of tree adjustment is relaxed, the red-black tree is not adjusted as frequently as a balanced binary tree, and the performance of data deletion and insertion can be improved.

As shown in fig. 2, the forwarding details of the lightweight network simulation method using the node proxy of the present invention include the following steps:

step S7: the method for transmitting the RPC request by the node specifically comprises the following steps:

s701: a node needs some RPC function of an external node.

S702: the node takes an array storing external node communication instances.

S703: the node takes the communication instance of the target node from the array according to the number.

S703: and the node calls a corresponding request function by using the communication instance and sends the RPC request.

Step S8: the inter-node communication is transferred by the proxy node, and the RPC request issued in S7 is logically destined to some external node, but is actually received by the proxy node of the corresponding node from the implementation point of view.

After receiving the RPC request, the proxy node will do network simulation action and then choose to discard or forward the RPC to the target node.

Step S9: the agent node reads the configuration from the network simulation configuration center during initialization, and performs corresponding actions according to the locally stored network configuration after receiving the RPC, and the specific steps are as follows:

s901: and checking whether the network is unavailable, if so, discarding the RPC, and otherwise, turning to the next step.

S902: and checking whether the source node or the target node is in different network partitions, if so, discarding the RPC, and otherwise, turning to the next step.

S903: and checking whether the source node or the target node is blocked, if so, discarding the RPC, and otherwise, turning to the next step.

S904: and checking whether the network has delay, if so, randomly sleeping for a period of time and then switching to the next step, otherwise, directly switching to the next step.

S905: and checking whether the starting of the source node or the target node is delayed, if so, randomly sleeping for a period of time and then switching to the next step, and if not, directly switching to the next step.

S906: and checking whether data disorder is started, if so, disordering the data in the sending queue, and otherwise, turning to the next step.

Step S10: the agent node counts network data, and the specific steps are as follows:

s101: and checking the source and target nodes of the RPC.

S102: and calculating the size of the RPC data packet data.

S103: and updating the total data volume of the network, the data volume sent by the RPC sender and the data volume received by the RPC receiver.

S104: the time that the RPC has since received a successful forward is tracked and taken as the delay into the statistics of the average delay of the network.

S105: periodically synchronizing data to a network simulation configuration center in the form of incremental updates

As shown in fig. 3, the parameter adjustment and system exit of the lightweight network simulation method using the node agent of the present invention specifically include the following steps:

step S11: starting the test system, initializing the system, automatically running, and waiting for a user instruction.

Step S12: the user sends the instruction through RPC or HTTP and other modes.

Step S13: the system checks whether the command is a network parameter modification command, if so, the process goes to S14, otherwise, the process goes to S15.

Step S14: the test system modifies the network parameters stored in the memory after receiving the modification request, and the specific steps are as follows:

s1401: see the type of parameter that needs to be modified.

S1402: modifying the network parameters stored in the memory.

S1403: and informing the proxy node that the network parameters have changed through communication modes such as RPC or HTTP and the like, and carrying the parameters needing to be modified in the message.

Step S15: the system checks whether it is an instruction to close the test, if so, it goes to S16, otherwise, it goes to S12 to wait for the user to issue a command.

Step S16: and the network simulation configuration center sends a closing command to all the nodes through the RPC, waits for the closed replies of all the nodes, and exits the process after receiving the closed replies of all the nodes.

The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.

Claims

1. A lightweight network simulation approach using node proxies, comprising the steps of:

s2: creating a proxy node for each node in the process;

s3: initializing an RPC frame of a node;

s4: each node in the process establishes connection with all the agent nodes;

s7: the communication between the nodes is transferred by the proxy node;

s9: network data is counted by the agent node;

s10: network data is written into a network simulation configuration center;

2. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S1, the network simulation configuration center reads the json configuration file, where the configuration file includes the overall network delay size, the packet loss rate, and the network partition condition, and includes the node network delay size, the packet loss rate, whether the node is blocked, and whether the node arrives out of order, and these parameter information are read for future viewing.

3. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S2, the proxy node only receives data for the node, and it does not actively send data to other nodes.

4. The lightweight network simulation approach using node proxies of claim 3, wherein: the proxy node receives data for the node, then checks the current network simulation parameter, and finally determines to adopt normal forwarding, delay, packet loss or out-of-order arrival and other operations on the data.

5. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S3, the method includes: the same RPC frame, such as grpc or brpc, etc., should be used between nodes; initializing the RPC framework and determining a calling interface provided externally.

6. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S4, the node establishes connections with all the proxy nodes, and when the node intends to send data to a certain node, the target proxy node should be used as a sending target, and the proxy node is responsible for transferring data as a broker role.

7. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S5, the node obtains a node number at initialization, and the number increases from 0 to 1. Meanwhile, the node sets a port for RPC communication, and can become a socket when being connected with the IP address of the node.

8. The lightweight network simulation approach using node proxies of claim 7, wherein: the proxy node receives an RPC, which can determine the source by looking at the socket of the RPC initiator, and then looking at the network configuration, determine how to forward the data.

9. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S6, the network simulation configuration center stores the node number and the socket by using the map data type in C + +, and the proxy node needs to apply for the network simulation configuration center when it needs to check the mapping between the socket and the number, and the result is returned by the latter.

10. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S7, the method specifically includes the following steps:

s701: node sends RPC request to proxy node of target node

S708: the proxy node initiates an RPC request to the target node.

11. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S8, the method specifically includes the following steps:

S803: to see if a delay action is taken on the packet.

S806: and all the data packets in the sending queue are sent to the target node.

12. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S9, the method specifically includes the following steps:

s901: the proxy node receives the RPC request

S902: the proxy node calculates the size of data carried by RPC

S903: proxy node updating total forwarding data size

S904: proxy node updates the number of RPC forwarding

13. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S10, the network data is counted in the proxy node, and the proxy node periodically writes the data back to the network simulation configuration center, which also periodically stores the data in a json format to a disk.

14. The lightweight network simulation approach using node proxies of claim 13, wherein: the agent node updates data to the network simulation configuration center by adopting an incremental updating method.

15. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S11, the method specifically includes the following steps:

16. The lightweight network simulation approach using node proxies of claim 1, wherein: in step S12, the network simulation configuration center sends a test-completed request to all nodes, including the proxy node, by using RPC, HTTP, and the like, waits for replies from all nodes, and ends the process after confirming that all nodes are closed.