CN114007246B - Method, apparatus, computer device and medium for reducing network congestion - Google Patents
Method, apparatus, computer device and medium for reducing network congestion Download PDFInfo
- Publication number
- CN114007246B CN114007246B CN202111275790.0A CN202111275790A CN114007246B CN 114007246 B CN114007246 B CN 114007246B CN 202111275790 A CN202111275790 A CN 202111275790A CN 114007246 B CN114007246 B CN 114007246B
- Authority
- CN
- China
- Prior art keywords
- network
- data
- map
- data storage
- packet loss
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 148
- 230000008569 process Effects 0.000 claims abstract description 95
- 238000013500 data storage Methods 0.000 claims abstract description 90
- 230000002159 abnormal effect Effects 0.000 claims abstract description 11
- 230000004044 response Effects 0.000 claims description 28
- 230000007246 mechanism Effects 0.000 claims description 15
- 230000005856 abnormality Effects 0.000 claims description 8
- 238000007619 statistical method Methods 0.000 claims description 8
- 238000012935 Averaging Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 238000007726 management method Methods 0.000 description 8
- 238000001228 spectrum Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008439 repair process Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- HOKKHZGPKSLGJE-GSVOUGTGSA-N N-Methyl-D-aspartic acid Chemical compound CN[C@@H](C(O)=O)CC(O)=O HOKKHZGPKSLGJE-GSVOUGTGSA-N 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/0231—Traffic management, e.g. flow control or congestion control based on communication conditions
- H04W28/0236—Traffic management, e.g. flow control or congestion control based on communication conditions radio quality, e.g. interference, losses or delay
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/0289—Congestion control
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present disclosure relates to a method, apparatus, computer device and medium for reducing network congestion; wherein the method comprises the following steps: constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map; determining a target data service with network congestion based on the delay profile and the packet loss rate profile; and determining a data reconstruction strategy of the target data service according to the application scene of the data storage process. The embodiment of the disclosure can timely find out network congestion, and after the network is abnormal, control the flow of data reconstruction, and reduce the network congestion.
Description
Technical Field
The present disclosure relates to the field of communications technologies, and in particular, to a method, an apparatus, a computer device, and a medium for reducing network congestion.
Background
With the rapid development of the mobile internet, the generated data is more and more, and the traditional storage system adopts a centralized storage server to store all the data, but the storage server cannot meet the requirement of large-scale storage application. The distributed storage system adopts a soft-hard decoupling mode, realizes data storage, improves the reliability, availability and access efficiency of the system, and is easy to expand.
In a high-performance distributed storage system, an access service often needs to forward tens of thousands or even hundreds of thousands of input output (i.e. IO) to a data service every second, and a congested or unreachable network may cause a large number of blocked IOs, which results in that an application program perceives that storage is blocked, and at the same time, network congestion may reduce the performance of the distributed storage system, which causes inconvenience to users, so it is very necessary to reduce network congestion.
Disclosure of Invention
To solve or at least partially solve the above technical problems, the present disclosure provides a method, apparatus, computer device, and medium for reducing network congestion.
In a first aspect, the present disclosure provides a method of reducing network congestion, comprising:
constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map;
determining a target data service with network congestion based on the delay profile and the packet loss rate profile;
and determining a data reconstruction strategy of the target data service according to the application scene of the data storage process.
Optionally, the determining the data reconstruction policy of the target data service according to the application scenario of the data storage process includes:
acquiring the type of the application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is the operation priority, stopping the data reconstruction operation of the target data service when the data storage quantity exceeds a first preset threshold value;
and if the type is the data security priority, controlling the data reconstruction operation of the target data service according to the corresponding flow information when the data storage quantity exceeds a second preset threshold value.
Optionally, before constructing the network condition map according to the delay condition and the packet loss rate in the data storage process, the method further comprises:
and obtaining the delay condition and the packet loss rate in the data storage process.
Optionally, the obtaining the delay condition and the packet loss rate in the data storage process includes:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output IO request packets and a second number of received IO response packets sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and second number.
Optionally, the method further comprises:
acquiring the network card utilization rate in the data storage process;
and when the network card utilization rate exceeds a target threshold value, suspending the working process of the heartbeat mechanism.
Optionally, the method further comprises:
and under a preset triggering condition, updating the network condition map.
Optionally, the updating the network condition map under the preset trigger condition includes:
when detecting that the network is abnormal, updating the network condition map;
or,
when no abnormality of the network is detected, the network condition map is updated by a method of weighted averaging of the newly constructed network condition map and the historical network condition map within a preset time.
In a second aspect, the present disclosure provides an apparatus for reducing network congestion, comprising:
the map construction module is used for constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map;
a first determining module, configured to determine, based on the delay profile and the packet loss rate profile, a target data service in which network congestion occurs;
and the second determining module is used for determining the data reconstruction strategy of the target data service according to the application scene of the data storage process.
Optionally, the second determining module is specifically configured to:
acquiring the type of the application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is the operation priority, stopping the data reconstruction operation of the target data service when the data storage quantity exceeds a first preset threshold value;
and if the type is the data security priority, controlling the data reconstruction operation of the target data service according to the corresponding flow information when the data storage quantity exceeds a second preset threshold value.
Optionally, the apparatus further includes:
the acquisition module is used for acquiring the delay condition and the packet loss rate in the data storage process before constructing the network condition map according to the delay condition and the packet loss rate in the data storage process.
Optionally, the acquiring module is specifically configured to:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output IO request packets and a second number of received IO response packets sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and second number.
Optionally, the apparatus further includes:
the utilization rate acquisition module is used for acquiring the utilization rate of the network card in the data storage process;
and the pause module is used for pausing the working process of the heartbeat mechanism when the network card utilization rate exceeds a target threshold value.
Optionally, the apparatus further includes:
and the updating module is used for updating the network condition map under the preset triggering condition.
Optionally, the updating module is specifically configured to:
when detecting that the network is abnormal, updating the network condition map;
or,
when no abnormality of the network is detected, the network condition map is updated by a method of weighted averaging of the newly constructed network condition map and the historical network condition map within a preset time.
In a third aspect, the present disclosure also provides a computer device comprising:
one or more processors;
storage means for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of reducing network congestion as described in any of the embodiments of the present disclosure.
In a fourth aspect, the present disclosure also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of reducing network congestion as described in any of the embodiments of the present disclosure.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages: firstly, constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises the delay map and the packet loss rate map, then determining a target data service with network congestion based on the delay map and the packet loss rate map, and finally determining a data reconstruction strategy of the target data service according to the application scene of the data storage process.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic diagram of a distributed storage system provided by an embodiment of the present disclosure;
fig. 2 is a flow chart of a method for reducing network congestion provided by an embodiment of the present disclosure;
FIG. 3 is a flow chart of another method for reducing network congestion provided by embodiments of the present disclosure;
fig. 4 is a schematic structural diagram of an apparatus for reducing network congestion according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, a further description of aspects of the present disclosure will be provided below. It should be noted that, without conflict, the embodiments of the present disclosure and features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced otherwise than as described herein; it will be apparent that the embodiments in the specification are only some, but not all, embodiments of the disclosure.
Fig. 1 is a schematic structural diagram of a distributed storage system according to an embodiment of the present disclosure, as shown in fig. 1:
the distributed storage system is mainly used for data storage, and usually adopts a soft and hard decoupling mode, the hardware can be a general server, and the software can be divided into three types of services, namely: the communication of the access service (front-end), the data service (back-end) and the management service (management end) may be implemented based on a transmission control protocol/internet protocol (Transmission Control Protocol/Internet Protocol, abbreviated as TCP/IP) network.
Illustratively, three services are all connected into two switches (switch a and switch B) through which the stack is made, achieving high availability of links. The access service transmits the data to the data service of the back end, and no traffic exists between the access services. The data services are different, and when the storage data needs to be migrated and rebuilt, the data services need to be accessed mutually.
Two access services are exemplarily shown in fig. 1, respectively: the access service A comprises a virtual disk A and a virtual disk B, and the access service B comprises a virtual disk C;1 management service; 3 data services, respectively: data service a, data service B and data service C, wherein data service a includes physical disk 1, physical disk 2 and physical disk 3, data service B includes physical disk 4, physical disk 5 and physical disk 6, and data service C includes physical disk 7, physical disk 8 and physical disk 9.
The access service is understood as a host responsible for providing a storage interface to the outside, and is divided into block storage, file storage and object storage according to the type of the interface. A data service is understood to mean a host responsible for the storage of data, storing data transmitted by an access service to a physical disk. The management service is mainly responsible for storing metadata, scheduling storage resources, copy management and the like. Metadata is used to describe the distributed storage clusters.
It should be noted that: the number of access services, data services, switches, virtual disks, and physical disks is not limited in this embodiment, and the structure of the distributed storage system is mainly illustrated in fig. 1.
The virtual disk is used for storing data in a scattered manner in a plurality of physical disks of a plurality of data services through the management and mapping of distributed storage, and the virtual disk needs to establish TCP connection with the management process of each physical disk storing own data. Access services typically allow multiple virtual disks to be mounted, resulting in a typically large number of connections.
With the popularization of high-speed storage devices, such as serial advanced technology attachment (Serial Advanced Technology Attachment, SATA for short), serial small computer system interface (Serial Attached SCSI, SAS for short), small computer system interface (Small ComputerSystem Interface, SCSI for short), solid State Disk (SSD for short), and Non-volatile memory management host system controller (nmda) SSD through interface technical specifications (Non-Volatile Memory Express, NVME for short), the performance of a distributed storage system is further improved, so that the flow rate through a storage network is also greater and greater, when the whole storage system is in high-load operation, the abnormality of the network easily causes the performance avalanche of the whole cluster.
Fig. 2 is a flow chart of a method for reducing network congestion according to an embodiment of the present disclosure. The present embodiment is applicable to a case of reducing network congestion in a distributed storage system. The method of the present embodiment may be performed by an apparatus for reducing network congestion, which may be implemented in hardware and/or software, and may be configured in a computer device. As shown in fig. 2, the method specifically includes the following steps:
s110, constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map.
The delay pattern may be understood as a pattern formed by delay time of the access service receiving the heartbeat response packet after the access service sends the heartbeat packet to the data service. The packet loss rate map may be understood as a map formed by packet loss rates calculated by the access service when the access service does not receive the IO response packet after sending the IO request packet to the data service.
In the prior art, a distributed storage system needs to detect the network connection state of the whole storage cluster, for example, a regular heartbeat mode is adopted to judge whether a heartbeat response packet is received within a specified time, and the method can judge whether the network is reachable or not, and cannot comprehensively consider the situation of network congestion, so that the problem of performance reduction caused by the network congestion in the distributed storage system cannot be solved.
In this embodiment, two important indicators of network conditions are measured, namely: the delay condition and the packet loss rate in the data storage process can construct a network condition map, and the network condition map can be specifically constructed according to the delay condition in the data storage process, namely: after a process of a virtual disk in the access service sends a heartbeat packet to a process of a physical disk corresponding to the data service, the process of the virtual disk receives delay time of the heartbeat response packet, and then the virtual disk and the physical disk are combined together to construct a two-dimensional map, so that the delay map can be obtained; according to the packet loss rate in the data storage process, namely: after the access service sends the IO request packet to the data service, the access service calculates the packet loss rate when not receiving the IO response packet, and then the two-dimensional map is constructed by combining the virtual disk in the access service and the physical disk in the data service together, so that the packet loss rate map can be obtained.
For example, assuming that there are 3 virtual disks in the access service and 5 physical disks in the data service, table 1 below is a schematic diagram of the network status map, as shown in table 1: the row represents a virtual disk in the access service, the column represents a physical disk in the switch or the data service, and the blank cell represents a value of delay time or packet loss rate to be filled.
TABLE 1
| Device/indicator | Physical disk 1 | Physical disk 2 | Physical disk 3 | Physical disk 4 | Physical disk 5 |
| Virtual disk A | |||||
| Virtual disk B | |||||
| Virtual disk C |
Taking the delay chart table 2 as an example, the average value of delay in a period of time during the data storage process is recorded, -it indicates that no data is stored on the physical disk by the virtual disk, and OT (timeout) indicates that the heartbeat response packet returns to timeout, that is: the time for which the heartbeat response packet is returned exceeds a preset timeout time.
TABLE 2
| Device/indicator | Physical disk 1 | Physical disk 2 | Physical disk 3 | Physical disk 4 | Physical disk 5 |
| Virtual disk A | Undelayed | - | Undelayed | - | Undelayed |
| Virtual disk B | Undelayed | OT | Undelayed | - | - |
| Virtual disk C | Undelayed | Undelayed | Undelayed | Undelayed | - |
And S120, determining a target data service with network congestion based on the delay profile and the packet loss rate profile.
The target data service may be understood as a data service where network congestion occurs.
After the delay map and the packet loss rate map are constructed, which data service and which physical disk in the data service have network congestion can be determined according to specific data in the two maps, so that the network congestion condition can be found in time, and the problem can be relieved in a targeted manner.
S130, determining a data reconstruction strategy of the target data service according to the application scene of the data storage process.
After determining the target data service with network congestion, indicating that a certain network link is not reachable, the storage traffic will be transferred to other network links, and the damaged or congested link will lead to a data copy to be behind due to the data cannot be transmitted, so that the storage of multiple copies is in a degraded state, i.e. the set available copy number is not reached. The distributed storage system can actively reconstruct or repair abnormal copies at the moment, so that the safety of data is ensured. The process of rebuilding or repairing, however, also relies on the TCP/IP network, which will further lead to network congestion, resulting in reduced overall storage system performance.
In order to solve the above problems, in combination with different application scenarios of the data storage process, for example, website application and database storage, a corresponding data reconstruction policy may be formulated in advance according to different application scenarios, so that an appropriate data reconstruction policy is determined for a target data service according to a current application scenario, and traffic of data reconstruction or repair is managed and controlled, so that network congestion caused by data reconstruction or repair can never be relieved.
In this embodiment, firstly, a network condition map is constructed according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises the delay map and the packet loss rate map, then, a target data service with network congestion is determined based on the delay map and the packet loss rate map, and finally, a data reconstruction strategy of the target data service is determined according to the application scenario of the data storage process.
In this embodiment, optionally, the method may further specifically include:
and under a preset triggering condition, updating the network condition map.
The preset triggering condition may be preset, or may be determined according to the specific situation, which is not limited in this embodiment.
In this embodiment, since the network status of the data storage process is continuously changed, the network status map needs to be updated under a preset trigger condition, so as to facilitate the subsequent better determination of the target data service with network congestion.
In this embodiment, optionally, updating the network condition map under a preset trigger condition includes:
when detecting that the network is abnormal, updating the network condition map;
or,
when no abnormality of the network is detected, the network condition map is updated by a method of weighted averaging of the newly constructed network condition map and the historical network condition map within a preset time.
The preset time may be set in advance, or may be determined according to actual situations, which is not limited in this embodiment.
In this embodiment, when an abnormality in the network is detected, the network condition map may be updated in time, so that the updated network condition map may reflect the latest network congestion situation; and when the network is abnormal, the network condition spectrum is not updated in real time, but the network condition spectrum is updated by a method of carrying out weighted average on the newly constructed network condition spectrum and the historical network condition spectrum within the preset time, for example, the data of the newly constructed network condition spectrum is Dnew, the data of the historical network condition spectrum is Dtable, and then the data of the updated network condition spectrum can be 0.3 times Dnew+0.7 times Dtable, wherein 0.3 and 0.7 are weighting coefficients, and the network condition spectrum can be adjusted in real time. The weighted average method is particularly suitable for the situation that the network link is just recovered from the abnormality, and the situation is not reflected in the network condition map immediately at the moment, but the weighted average is gradually adjusted after a period of time, so that the situation that the network is abnormal again due to large flow is avoided when the network congestion is recovered.
Fig. 3 is a flow chart of another method for reducing network congestion provided by an embodiment of the present disclosure. This embodiment is optimized based on the above embodiment. Optionally, this embodiment explains in detail the process of determining the data reconstruction policy of the target data service.
As shown in fig. 3, the method specifically includes the following steps:
s210, obtaining the delay condition and the packet loss rate in the data storage process.
In order to obtain the network congestion situation in the data storage process in time, two important indexes of the network situation need to be obtained, namely: delay condition and packet loss rate. Because the access service is the originating terminal of the network traffic, the delay condition and the packet loss rate in the data storage process can be obtained through the access service.
S220, constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map.
And S230, determining a target data service with network congestion based on the delay profile and the packet loss rate profile.
S240, acquiring the type of the application scene of the data storage process, wherein the type comprises operation priority and data security priority.
The operation priority can be understood as the storage of the priority guarantee data in the application scene. The data security priority can be understood as ensuring the security of the data preferentially in the application scene, so that when network congestion occurs, data reconstruction is needed for the abnormal copy.
After determining the target data service where network congestion occurs, the type of application scenario of the data storage process needs to be acquired, so that the corresponding data reconstruction policy is determined according to different application scenario types.
S250, determining whether the type of the application scene is the operation priority.
If yes, executing S260; if not, S270 is performed.
And S260, stopping the data reconstruction operation of the target data service when the data storage quantity exceeds a first preset threshold value.
The first preset threshold may be preset, or may be determined according to circumstances, which is not limited in this embodiment.
If the type of the application scene is the operation priority, the data can be smoothly stored mainly. Therefore, when the data storage amount exceeds the first preset threshold, since the data reconstruction operation also depends on the network, in order not to increase the burden on the network, it is necessary to stop the data reconstruction operation of the target data service.
And S270, when the data storage quantity exceeds a second preset threshold value, controlling the data reconstruction operation of the target data service according to the corresponding flow information.
The second preset threshold may be preset, or may be determined according to circumstances, which is not limited in this embodiment. The magnitude relation between the first preset threshold and the second preset threshold is not limited in this embodiment.
If the type of the application scene is data security priority, the data security is mainly ensured. Therefore, when the data storage amount exceeds the second preset threshold, the data reconstruction operation of the target data service needs to be controlled according to the corresponding flow information, namely the flow information of the scene, and the progress of the data reconstruction operation can be controlled, so that the data security is ensured while the data is stored.
In this embodiment, firstly, a delay condition and a packet loss rate in a data storage process are obtained, a network condition map is constructed according to the delay condition and the packet loss rate in the data storage process, then, a target data service with network congestion is determined based on the delay map and the packet loss rate map, then, the type of an application scene in the data storage process is obtained, finally, whether the type of the application scene is operation priority is determined, and if the type of the application scene is operation priority, the data reconstruction operation of the target data service is stopped when the data storage quantity exceeds a first preset threshold value. If the type of the application scene is data security priority, when the data storage quantity exceeds a second preset threshold value, controlling data reconstruction operation of the target data service according to the corresponding flow information, and determining corresponding data reconstruction strategies according to the types of different application scenes, thereby being beneficial to relieving network congestion caused by the data reconstruction process, avoiding occurrence of storage blocking and further improving the data storage performance.
In this embodiment, optionally, the obtaining the delay condition and the packet loss rate in the data storage process may specifically include:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output IO request packets and a second number of received IO response packets sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and second number.
Specifically, after a heartbeat packet is sent by a process of a virtual disk in an access service to a process of a physical disk storing virtual disk data at regular time by using a heartbeat mechanism, the process of the physical disk returns a heartbeat response packet to the virtual disk process after receiving the heartbeat packet, and if the time of the virtual disk process receiving the heartbeat response packet exceeds a preset timeout time, the heartbeat response packet returns to be overtime; if the time of the virtual disk process receiving the heartbeat response packet does not exceed the preset timeout time, the heartbeat response packet returns without timeout, so that the specific delay condition in the data storage process can be determined according to the time of the virtual disk process receiving the heartbeat response packet.
The packet loss rate can be judged based on the stored IO request, and the packet loss usually occurs in a busy network state, so that the stored IO times per second is very high, and a large amount of CPU resources are consumed based on the statistics of the packet loss number of each IO. Therefore, the embodiment calculates the packet loss rate of the current distributed storage system through a window statistical method.
The time length of the window statistics time window setting may be equal to the IO request timeout time set by the distributed storage system, seq-s: indicating the total number of IO request packets that have been successfully sent at a certain time. seq-r: indicating the total number of IO response packets that have been successfully received at the base instant. For example, the IO request packet for which time window 1 opens a record is seq-s1, the IO request packet for which time window 2 opens a record is seq-s2, and so on; the IO response packet for time window 1 open record is seq-r1, time window 2 open record is seq-r2, and so on.
For example, when seq-s2-seq-r3< = 0, it indicates that all the IO requests sent by the time window 1 are received, where there is no packet loss in the time window 1, and the packet loss rate at the opening time of the time window 4 is 0; when seq-s2-seq-r3>0, it indicates that there is a packet loss in the time window 1, and the total number of packets lost in this period is seq-s2-seq-r3, and the number of transmitted IO requests is: seq-s2-seq-s1, so the packet loss rate at the opening time of the time window 4 is: (seq-s 2-seq-r 3)/(seq-s 2-seq-s 1).
Therefore, the packet loss rate in the data storage process can be calculated by combining the formula through the first number of IO request packets sent by each access service and the second number of received IO response packets.
In this embodiment, although each virtual disk calculates its own packet loss rate, the calculation method is simple and convenient, the overhead of the system can be ignored, and the statistics of the packet loss rate has no influence on the load of the network itself, and when the load of the network is high, the use of the above-mentioned packet loss rate statistical method will not cause further congestion of the network.
In this embodiment, optionally, each IO request packet needs to be written/read sequentially.
In this embodiment, in order to ensure that the writing order of the stored data is not disturbed by the asynchronous network transmission, each IO request packet may be numbered, for example, with a positive integer seq of 64 bits as the identifier, the seq increases by 1 with each IO request packet. The data service writes/reads the received IO request packets into/from the disk according to the seq order, and sends IO response packets to the access service. The access service assembles the acknowledgements in order of increasing 1 according to the seq of the IO response packet. Meanwhile, to avoid the case where old data overlaps new data, seq-r acknowledges by incremental reception.
In this embodiment, optionally, the method further includes:
acquiring the network card utilization rate in the data storage process;
and when the network card utilization rate exceeds a target threshold value, suspending the working process of the heartbeat mechanism.
The target threshold may be preset, or may be determined according to circumstances, which is not limited in this embodiment.
Specifically, when the number of virtual disks in the access service is large, the number of TCP connections to be detected is also large, and the time interval for sending the heartbeat packet based on the heartbeat mechanism is determined by the following formula:
time interval for sending heartbeat packet = preset time interval (1 + network card usage)/(1-network card usage)
According to the formula, when the network card usage rate is high, the heartbeat packet may aggravate congestion of the network, so that the network card usage rate in the data storage process can be obtained through the operating system, and when the network card usage rate exceeds the target threshold, the working process of the heartbeat mechanism is suspended. Thereby avoiding further congestion of the network due to the transmission of heartbeat packets.
In this embodiment, the heartbeat mechanism is combined with the window statistics method, so that it can determine whether network congestion occurs or not and determine a link of the network congestion under extremely low network overhead. Particularly, under the condition that the network is congested, the working process of a heartbeat mechanism can be automatically suspended, and only a window statistical method is adopted, so that the overtime detection frequency is greatly reduced, and the system performance is improved.
Fig. 4 is a schematic structural diagram of an apparatus for reducing network congestion according to an embodiment of the present disclosure; the device is configured in the computer equipment, and can realize the method for reducing network congestion in any embodiment of the application. The device specifically comprises the following steps:
the map construction module 310 is configured to construct a network condition map according to the delay condition and the packet loss rate in the data storage process, where the network condition map includes a delay map and a packet loss rate map;
a first determining module 320, configured to determine, based on the delay profile and the packet loss rate profile, a target data service where network congestion occurs;
a second determining module 330, configured to determine a data reconstruction policy of the target data service according to an application scenario of the data storage process.
In this embodiment, optionally, the second determining module 330 is specifically configured to:
acquiring the type of the application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is the operation priority, stopping the data reconstruction operation of the target data service when the data storage quantity exceeds a first preset threshold value;
and if the type is the data security priority, controlling the data reconstruction operation of the target data service according to the corresponding flow information when the data storage quantity exceeds a second preset threshold value.
In this embodiment, optionally, the apparatus further includes:
the acquisition module is used for acquiring the delay condition and the packet loss rate in the data storage process before constructing the network condition map according to the delay condition and the packet loss rate in the data storage process.
In this embodiment, optionally, the obtaining module is specifically configured to:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output IO request packets and a second number of received IO response packets sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and second number.
In this embodiment, optionally, the apparatus further includes:
the utilization rate acquisition module is used for acquiring the utilization rate of the network card in the data storage process;
and the pause module is used for pausing the working process of the heartbeat mechanism when the network card utilization rate exceeds a target threshold value.
In this embodiment, optionally, the apparatus further includes:
and the updating module is used for updating the network condition map under the preset triggering condition.
In this embodiment, optionally, the update module is specifically configured to:
when detecting that the network is abnormal, updating the network condition map;
or,
when no abnormality of the network is detected, the network condition map is updated by a method of weighted averaging of the newly constructed network condition map and the historical network condition map within a preset time.
According to the device for reducing network congestion, firstly, a network condition map is constructed according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises the delay map and the packet loss rate map, then, a target data service with network congestion is determined based on the delay map and the packet loss rate map, finally, a data reconstruction strategy of the target data service is determined according to the application scene of the data storage process, the network congestion can be found in time through the method, and after the network is abnormal, the flow of data reconstruction is controlled, so that the network congestion is reduced.
The device for reducing network congestion provided by the embodiment of the disclosure can execute the method for reducing network congestion provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of executing the method.
Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure. As shown in fig. 5, the computer device includes a processor 410 and a storage 420; the number of processors 410 in the computer device may be one or more, one processor 410 being taken as an example in fig. 5; the processor 410 and the memory device 420 in the computer device may be connected by a bus or other means, for example in fig. 5.
The storage 420 is a computer readable storage medium, and may be used to store a software program, a computer executable program, and modules, such as program instructions/modules corresponding to the method for reducing network congestion in the embodiments of the present disclosure. The processor 410 executes various functional applications of the computer device and data processing, i.e., implements the method for reducing network congestion provided by embodiments of the present disclosure, by running software programs, instructions, and modules stored in the storage 420.
The storage device 420 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application program required for functions; the storage data area may store data created according to the use of the terminal, etc. In addition, the storage 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some examples, storage 420 may further include memory remotely located relative to processor 410, which may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The computer device provided in this embodiment may be used to execute the method for reducing network congestion provided in any of the foregoing embodiments, and has corresponding functions and beneficial effects.
The disclosed embodiments also provide a storage medium containing computer-executable instructions that when executed by a computer processor are for implementing the method of reducing network congestion provided by the disclosed embodiments.
Of course, a storage medium containing computer-executable instructions provided by the embodiments of the present disclosure is not limited to the method operations described above, but may also perform related operations in the method for reducing network congestion provided by any embodiment of the present disclosure.
From the above description of embodiments, it will be apparent to those skilled in the art that the present disclosure may be implemented by means of software and necessary general purpose hardware, but may of course also be implemented by means of hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present disclosure may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk, or an optical disk of a computer, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments of the present disclosure.
It should be noted that, in the embodiment of the apparatus for reducing network congestion, each unit and module included are only divided according to the functional logic, but not limited to the above division, so long as the corresponding functions can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present disclosure.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown and described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (9)
1. A method of reducing network congestion, the method comprising:
constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map;
determining a target data service with network congestion based on the delay profile and the packet loss rate profile;
determining a data reconstruction strategy of the target data service according to the application scene of the data storage process;
the determining the data reconstruction policy of the target data service according to the application scenario of the data storage process includes:
acquiring the type of the application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is the operation priority, stopping the data reconstruction operation of the target data service when the data storage quantity exceeds a first preset threshold value;
and if the type is the data security priority, controlling the data reconstruction operation of the target data service according to the corresponding flow information when the data storage quantity exceeds a second preset threshold value.
2. The method of claim 1, further comprising, prior to constructing the network condition map based on delay conditions and packet loss rates in the data storage process:
and obtaining the delay condition and the packet loss rate in the data storage process.
3. The method of claim 2, wherein the obtaining the delay condition and the packet loss rate in the data storage process comprises:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output IO request packets and a second number of received IO response packets sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and second number.
4. A method according to claim 3, further comprising:
acquiring the network card utilization rate in the data storage process;
and when the network card utilization rate exceeds a target threshold value, suspending the working process of the heartbeat mechanism.
5. The method as recited in claim 1, further comprising:
and under a preset triggering condition, updating the network condition map.
6. The method of claim 5, wherein updating the network status map under the preset trigger condition comprises:
when detecting that the network is abnormal, updating the network condition map;
or,
when no abnormality of the network is detected, the network condition map is updated by a method of weighted averaging of the newly constructed network condition map and the historical network condition map within a preset time.
7. An apparatus for reducing network congestion, the apparatus comprising:
the map construction module is used for constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map;
a first determining module, configured to determine, based on the delay profile and the packet loss rate profile, a target data service in which network congestion occurs;
the second determining module is used for determining a data reconstruction strategy of the target data service according to the application scene of the data storage process;
the second determining module is specifically configured to:
acquiring the type of the application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is the operation priority, stopping the data reconstruction operation of the target data service when the data storage quantity exceeds a first preset threshold value;
and if the type is the data security priority, controlling the data reconstruction operation of the target data service according to the corresponding flow information when the data storage quantity exceeds a second preset threshold value.
8. A computer device, comprising:
one or more processors;
storage means for storing one or more programs,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111275790.0A CN114007246B (en) | 2021-10-29 | 2021-10-29 | Method, apparatus, computer device and medium for reducing network congestion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111275790.0A CN114007246B (en) | 2021-10-29 | 2021-10-29 | Method, apparatus, computer device and medium for reducing network congestion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114007246A CN114007246A (en) | 2022-02-01 |
| CN114007246B true CN114007246B (en) | 2024-02-02 |
Family
ID=79925648
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111275790.0A Active CN114007246B (en) | 2021-10-29 | 2021-10-29 | Method, apparatus, computer device and medium for reducing network congestion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114007246B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2922347A1 (en) * | 2007-10-11 | 2009-04-17 | Bouchaib Hoummady | Vehicle e.g. car, mobility and traffic guiding managing method, involves simulating temporal spatial load of traffic in mobility space, evaluating pertinent situations in traffic, and transferring data corresponding to circulation situation |
| CN103428240A (en) * | 2012-05-18 | 2013-12-04 | 深圳市腾讯计算机系统有限公司 | Data transmission method and data transmission terminal |
| CN109088794A (en) * | 2018-08-20 | 2018-12-25 | 郑州云海信息技术有限公司 | A kind of fault monitoring method and device of node |
| CN111510338A (en) * | 2020-03-09 | 2020-08-07 | 苏州浪潮智能科技有限公司 | Distributed block storage network sub-health test method, device and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112425122A (en) * | 2018-07-17 | 2021-02-26 | 脸谱公司 | Communication network insight to detect alerts |
| US10764184B2 (en) * | 2018-07-17 | 2020-09-01 | Facebook, Inc. | Detecting communication network insights of alerts |
-
2021
- 2021-10-29 CN CN202111275790.0A patent/CN114007246B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2922347A1 (en) * | 2007-10-11 | 2009-04-17 | Bouchaib Hoummady | Vehicle e.g. car, mobility and traffic guiding managing method, involves simulating temporal spatial load of traffic in mobility space, evaluating pertinent situations in traffic, and transferring data corresponding to circulation situation |
| CN103428240A (en) * | 2012-05-18 | 2013-12-04 | 深圳市腾讯计算机系统有限公司 | Data transmission method and data transmission terminal |
| CN109088794A (en) * | 2018-08-20 | 2018-12-25 | 郑州云海信息技术有限公司 | A kind of fault monitoring method and device of node |
| CN111510338A (en) * | 2020-03-09 | 2020-08-07 | 苏州浪潮智能科技有限公司 | Distributed block storage network sub-health test method, device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于场景模型的视频无线信道鲁棒传输;周有喜;肖嵩;吴成柯;;宇航学报(第01期);第108-112页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114007246A (en) | 2022-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11886731B2 (en) | Hot data migration method, apparatus, and system | |
| CN108062285B (en) | NVMe storage device and method for accessing same | |
| KR20210042161A (en) | Data synchronization method and apparatus, medium, and electronic device for distributed system | |
| US7945736B2 (en) | Dynamic load management of network memory | |
| CN111522636B (en) | Application container adjusting method, application container adjusting system, computer readable medium and terminal device | |
| US9027025B2 (en) | Real-time database exception monitoring tool using instance eviction data | |
| US8593948B1 (en) | Network device and method of controlling network device | |
| WO2011059837A1 (en) | Allocating storage memory based on future use estimates | |
| CN107204875B (en) | Data reporting link monitoring method and device, electronic equipment and storage medium | |
| EP2723017A1 (en) | Method, apparatus and system for implementing distributed auto-incrementing counting | |
| US7804775B2 (en) | Apparatus and computer program for identifying selected applications utilizing a single existing available bit in frame headers | |
| US10275303B2 (en) | Mechanism for running diagnostic rules across multiple nodes | |
| US9621438B2 (en) | Network traffic management | |
| CN109274550B (en) | iSCSI self-adaptive IO queue depth matching method | |
| CN109756429A (en) | Bandwidth allocation methods and equipment | |
| CN114007246B (en) | Method, apparatus, computer device and medium for reducing network congestion | |
| US9893972B1 (en) | Managing I/O requests | |
| CN105893150B (en) | Interface calling frequency control method and device and interface calling request processing method and device | |
| CN116405546A (en) | Data pushing method and terminal | |
| US20170269864A1 (en) | Storage Array Operation Method and Apparatus | |
| US10554511B2 (en) | Information processing apparatus, method and non-transitory computer-readable storage medium | |
| CN115333917A (en) | CDN anomaly detection method and device | |
| CN111158899A (en) | Data acquisition method, data acquisition device, task management center and task management system | |
| CN114064362B (en) | Data recovery method, system and computer readable storage medium for distributed storage | |
| CN112698927A (en) | Bidirectional communication method, device, electronic equipment and machine-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |