CN114007246A - Method, apparatus, computer device and medium for reducing network congestion - Google Patents
Method, apparatus, computer device and medium for reducing network congestion Download PDFInfo
- Publication number
- CN114007246A CN114007246A CN202111275790.0A CN202111275790A CN114007246A CN 114007246 A CN114007246 A CN 114007246A CN 202111275790 A CN202111275790 A CN 202111275790A CN 114007246 A CN114007246 A CN 114007246A
- Authority
- CN
- China
- Prior art keywords
- map
- network
- data
- data storage
- packet loss
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 149
- 230000008569 process Effects 0.000 claims abstract description 95
- 238000013500 data storage Methods 0.000 claims abstract description 88
- 230000002159 abnormal effect Effects 0.000 claims abstract description 13
- 230000004044 response Effects 0.000 claims description 28
- 230000007246 mechanism Effects 0.000 claims description 15
- 238000007619 statistical method Methods 0.000 claims description 8
- 230000005856 abnormality Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008439 repair process Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000593 degrading effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/0231—Traffic management, e.g. flow control or congestion control based on communication conditions
- H04W28/0236—Traffic management, e.g. flow control or congestion control based on communication conditions radio quality, e.g. interference, losses or delay
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/0289—Congestion control
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present disclosure relates to a method, apparatus, computer device and medium for reducing network congestion; wherein, the method comprises the following steps: constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map; determining a target data service with network congestion based on the delay map and the packet loss rate map; and determining a data reconstruction strategy of the target data service according to the application scene of the data storage process. The embodiment of the disclosure can timely find network congestion, and control the flow of data reconstruction after the network is abnormal, thereby reducing network congestion.
Description
Technical Field
The present disclosure relates to the field of communications technologies, and in particular, to a method, an apparatus, a computer device, and a medium for reducing network congestion.
Background
With the rapid development of the mobile internet, more and more data are generated, and the traditional storage system adopts a centralized storage server to store all data, but the storage server cannot meet the requirement of large-scale storage application. The distributed storage system adopts a soft and hard decoupling mode to realize data storage, improves the reliability, the availability and the access efficiency of the system and is easy to expand.
In a high-performance distributed storage system, an access service often needs to forward tens of thousands or even hundreds of thousands of input/output (IO) to a data service every second, a congested or unreachable network may cause a large number of blocked IO, resulting in that an application program senses the storage stuck, and meanwhile, network congestion may reduce the performance of the distributed storage system, causing inconvenience to users, and therefore, reducing network congestion is very necessary.
Disclosure of Invention
To address the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, apparatus, computer device, and medium for reducing network congestion.
In a first aspect, the present disclosure provides a method for reducing network congestion, comprising:
constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map;
determining a target data service with network congestion based on the delay map and the packet loss rate map;
and determining a data reconstruction strategy of the target data service according to the application scene of the data storage process.
Optionally, the determining a data reconstruction policy of the target data service according to the application scenario of the data storage process includes:
acquiring the type of an application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is operation priority, stopping data reconstruction operation of the target data service when the data storage exceeds a first preset threshold;
and if the type is data security priority, controlling data reconstruction operation of the target data service according to the corresponding flow information when the data storage exceeds a second preset threshold.
Optionally, before constructing the network condition map according to the delay condition and the packet loss rate in the data storage process, the method further includes:
and acquiring the delay condition and the packet loss rate in the data storage process.
Optionally, the obtaining of the delay condition and the packet loss rate in the data storage process includes:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output (IO) request packets and a second number of received IO response packets which are sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and the second number.
Optionally, the method further includes:
acquiring the network card utilization rate in the data storage process;
and when the utilization rate of the network card exceeds a target threshold value, suspending the working process of the heartbeat mechanism.
Optionally, the method further includes:
and updating the network condition map under a preset triggering condition.
Optionally, the updating the network condition map under the preset trigger condition includes:
when the network is detected to be abnormal, updating the network condition map;
or,
when no network abnormality is detected, the network condition map is updated by a method of carrying out weighted average on the newly constructed network condition map and the historical network condition map within the preset time.
In a second aspect, the present disclosure provides an apparatus for reducing network congestion, comprising:
the map building module is used for building a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map;
a first determining module, configured to determine, based on the delay map and the packet loss rate map, a target data service in which network congestion occurs;
and the second determining module is used for determining a data reconstruction strategy of the target data service according to the application scene of the data storage process.
Optionally, the second determining module is specifically configured to:
acquiring the type of an application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is operation priority, stopping data reconstruction operation of the target data service when the data storage exceeds a first preset threshold;
and if the type is data security priority, controlling data reconstruction operation of the target data service according to the corresponding flow information when the data storage exceeds a second preset threshold.
Optionally, the apparatus further comprises:
the acquisition module is used for acquiring the delay condition and the packet loss rate in the data storage process before constructing the network condition map according to the delay condition and the packet loss rate in the data storage process.
Optionally, the obtaining module is specifically configured to:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output (IO) request packets and a second number of received IO response packets which are sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and the second number.
Optionally, the apparatus further comprises:
the utilization rate acquisition module is used for acquiring the network card utilization rate in the data storage process;
and the pause module is used for pausing the working process of the heartbeat mechanism when the utilization rate of the network card exceeds a target threshold value.
Optionally, the apparatus further comprises:
and the updating module is used for updating the network condition map under the preset triggering condition.
Optionally, the update module is specifically configured to:
when the network is detected to be abnormal, updating the network condition map;
or,
when no network abnormality is detected, the network condition map is updated by a method of carrying out weighted average on the newly constructed network condition map and the historical network condition map within the preset time.
In a third aspect, the present disclosure also provides a computer device, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method for reducing network congestion of any of the embodiments of the present disclosure.
In a fourth aspect, the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the method for reducing network congestion according to any one of the embodiments of the present disclosure.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages: the method comprises the steps of firstly, constructing a network condition map according to delay conditions and packet loss rates in a data storage process, determining a target data service with network congestion based on the delay map and the packet loss rate map, and finally determining a data reconstruction strategy of the target data service according to an application scene of the data storage process.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a schematic structural diagram of a distributed storage system provided by an embodiment of the present disclosure;
fig. 2 is a flowchart illustrating a method for reducing network congestion according to an embodiment of the present disclosure;
fig. 3 is a flowchart illustrating another method for reducing network congestion according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an apparatus for reducing network congestion according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
Fig. 1 is a schematic structural diagram of a distributed storage system provided in an embodiment of the present disclosure, as shown in fig. 1:
the distributed storage system is mainly used for data storage, and usually adopts a software and hardware decoupling mode, hardware can be a general server, and software can be divided into three types of services, namely: the communication of the three services can be realized based on a Transmission Control Protocol/Internet Protocol (Transmission Control Protocol/Internet Protocol, TCP/IP for short) network.
Illustratively, all three services are connected to two switches (switch a and switch B), and by stacking the two switches, high availability of links is achieved. The access service transmits data to the data service of the back end, and no flow exists between the access services. The data services are different, and when the stored data needs to be migrated and rebuilt, the data services need to be accessed with each other.
Two access services are exemplarily shown in fig. 1, which are: the access service A comprises a virtual disk A and a virtual disk B, and the access service B comprises a virtual disk C; 1 management service; 3 data services, respectively: the data service A comprises a physical disk 1, a physical disk 2 and a physical disk 3, the data service B comprises a physical disk 4, a physical disk 5 and a physical disk 6, and the data service C comprises a physical disk 7, a physical disk 8 and a physical disk 9.
The access service may be understood as a host responsible for providing an external storage interface, and may be divided into block storage, file storage, and object storage according to the interface type. A data service may be understood as a host responsible for the storage of data, storing data transmitted by an access service in a physical disk. The management service is mainly responsible for storing metadata, scheduling storage resources, copy management and the like. The metadata is used to describe the distributed storage cluster.
It should be noted that: the number of access services, data services, switches, virtual disks, and physical disks is not limited in this embodiment, and the structure shown in fig. 1 is mainly used to explain the structure of the distributed storage system.
Through management and mapping of distributed storage, data is stored in a plurality of physical disks of a plurality of data services in a scattered manner by the virtual disk, and the virtual disk needs to establish a TCP connection with a management process of each physical disk storing data of the virtual disk. Access services typically allow multiple virtual disks to be mounted, resulting in a typically large number of connections.
With the popularization of high-speed storage devices, such as Serial Advanced Technology Attachment (SATA), Serial Attached Small computer system Interface (SAS), Small Computer System Interface (SCSI), Solid State Disk (SSD), and Non-Volatile Memory management host system controller (Non-Volatile Memory Express (NVME)), the performance of a distributed storage system is further improved, and thus, the traffic through a storage network is also increased, and when the entire storage system is in high-load operation, the network is abnormal, which is likely to cause the performance of the entire cluster, the present disclosure provides a method for reducing network congestion, which can discover network congestion in time, and control the traffic of data reconstruction after the network is abnormal, network congestion is reduced.
Fig. 2 is a flowchart illustrating a method for reducing network congestion according to an embodiment of the present disclosure. The present embodiments may be applicable to reducing network congestion in a distributed storage system. The method of the embodiment may be performed by an apparatus for reducing network congestion, which may be implemented in hardware and/or software and may be configured in a computer device. As shown in fig. 2, the method specifically includes the following steps:
s110, constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map.
The delay map may be understood as a map formed by delay time of the access service receiving the heartbeat response packet after the access service sends the heartbeat packet to the data service. The packet loss rate map may be understood as a map formed by the packet loss rate calculated when the access service does not receive the IO response packet after the access service sends the IO request packet to the data service.
In the prior art, a distributed storage system needs to detect a network connection state of a whole storage cluster, for example, a periodic heartbeat mode is adopted to judge whether a heartbeat response packet is received within a specified time.
In this embodiment, two important indicators of the network condition are measured, that is: the network condition map can be constructed according to the delay condition and the packet loss rate in the data storage process, and specifically, according to the delay condition in the data storage process, the network condition map can be constructed by: after the process of the virtual disk in the access service sends a heartbeat packet to the process of the physical disk corresponding to the data service, the process of the virtual disk receives the delay time of a heartbeat response packet, and then a two-dimensional map is jointly constructed by combining the virtual disk and the physical disk, so that a delay map can be obtained; according to the packet loss rate in the data storage process, namely: after the access service sends the IO request packet to the data service, the packet loss rate calculated when the access service does not receive the IO response packet is combined with the virtual disk in the access service and the physical disk in the data service to jointly construct a two-dimensional map, and then the packet loss rate map can be obtained.
For example, assuming that there are 3 virtual disks in the access service and 5 physical disks in the data service, table 1 below is a schematic structural diagram of a network condition map, as shown in table 1: the rows represent virtual disks in the access service, the columns represent physical disks in the switch or data service, and the blank cells represent values of delay time or packet loss rate to be filled.
TABLE 1
| Equipment/index | Physical disk 1 | Physical disk 2 | Physical disk 3 | Physical disk 4 | Physical disk 5 |
| Virtual disk A | |||||
| Virtual disk B | |||||
| Virtual disk C |
Taking the delay map table 2 as an example, the average value of the delay in a period of time during the data storage process is recorded, indicating that no data is stored in the virtual disk in the physical disk, and OT (timeout) indicates that the heartbeat response packet returns to timeout, that is: the time for returning the heartbeat response packet exceeds the preset timeout time.
TABLE 2
| Equipment/index | Physical disk 1 | Physical disk 2 | Physical disk 3 | Physical disk 4 | Physical disk 5 |
| Virtual disk A | Without delay | - | Without delay | - | Without delay |
| Virtual disk B | Without delay | OT | Without delay | - | - |
| Virtual disk C | Without delay | Without delay | Without delay | Without delay | - |
And S120, determining a target data service with network congestion based on the delay map and the packet loss rate map.
The target data service may be understood as a data service in which network congestion occurs.
After the delay map and the packet loss rate map are constructed, which data service and which physical disk in the data service has network congestion can be determined according to specific data in the two maps, so that the network congestion condition can be found in time, and the problem can be favorably and specifically alleviated in the follow-up process.
S130, determining a data reconstruction strategy of the target data service according to the application scene of the data storage process.
After the target data service with network congestion is determined, it is indicated that a certain network link is not reachable, the storage traffic will be transferred to other network links, and due to the fact that data cannot be transmitted, the damaged or congested link will cause data copy to be backward, so that the storage of multiple copies is in a degraded state, that is, the set available copy number is not reached. The distributed storage system can actively rebuild or repair the abnormal copy at the moment, and the safety of data is guaranteed. But the process of rebuilding or repairing, also depending on the TCP/IP network, will further cause network congestion, degrading the overall storage system performance.
In order to solve the above problems, in combination with different application scenarios of the data storage process, for example, website application and database storage, corresponding data reconstruction strategies may be formulated in advance according to different application scenarios, so that a suitable data reconstruction strategy is determined for a target data service according to a current application scenario, and the flow of data reconstruction or repair is controlled, which never can alleviate a network congestion situation caused by data reconstruction or repair.
In this embodiment, a network condition map is first constructed according to a delay condition and a packet loss rate in a data storage process, wherein the network condition map includes a delay map and a packet loss rate map, then a target data service with network congestion is determined based on the delay map and the packet loss rate map, and finally a data reconstruction strategy of the target data service is determined according to an application scenario of the data storage process.
In this embodiment, optionally, the method may further specifically include:
and updating the network condition map under a preset triggering condition.
The preset trigger condition may be preset, or may be determined according to a specific situation, and this embodiment is not limited.
In this embodiment, since the network condition in the data storage process is constantly changing, the network condition map needs to be updated under the preset trigger condition, so as to facilitate better determining the target data service with network congestion in the following.
In this embodiment, optionally, the updating the network condition map under the preset trigger condition includes:
when the network is detected to be abnormal, updating the network condition map;
or,
when no network abnormality is detected, the network condition map is updated by a method of carrying out weighted average on the newly constructed network condition map and the historical network condition map within the preset time.
The preset time may be set in advance, or may be determined according to an actual situation, and this embodiment is not particularly limited.
In the embodiment, when the network is detected to be abnormal, the network condition map can be updated in time, so that the updated network condition map can reflect the latest network congestion condition; when the network is not abnormal, the network condition map is not updated in real time, but the network condition map is updated by a method of performing weighted average on the newly constructed network condition map and the historical network condition map within preset time, for example, the data of the newly constructed network condition map is Dnew, and the data of the historical network condition map is Dtable, so that the data of the updated network condition map can be 0.3 Dnew +0.7 Dtable, wherein 0.3 and 0.7 are weighting coefficients, and the adjustment can be performed in real time. The weighted average method is particularly suitable for the situation that when a network link is just recovered from an abnormality, the network link is not reflected in a network condition map immediately at the moment, but is gradually adjusted through weighted average for a period of time, so that the phenomenon that the network is abnormal again due to large flow when network congestion is recovered is avoided.
Fig. 3 is a flowchart illustrating another method for reducing network congestion according to an embodiment of the present disclosure. The embodiment is optimized on the basis of the embodiment. Optionally, the present embodiment explains in detail a process of determining a data reconstruction policy of a target data service.
As shown in fig. 3, the method specifically includes the following steps:
s210, obtaining the delay condition and the packet loss rate in the data storage process.
In order to obtain the network congestion condition in the data storage process in time, two important indexes of the network condition need to be obtained, namely: delay conditions and packet loss rates. Since the access service is an initiating end of the network traffic, the delay condition and the packet loss rate in the data storage process can be obtained through the access service.
S220, constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map.
And S230, determining a target data service with network congestion based on the delay map and the packet loss rate map.
S240, acquiring the type of the application scene in the data storage process, wherein the type comprises operation priority and data security priority.
The running priority can be understood as priority guarantee of data storage in the application scenario. Data security priority can be understood as priority guarantee of data security in the application scenario, and therefore when network congestion occurs, data reconstruction needs to be performed on an abnormal copy.
After determining a target data service with network congestion, the type of the application scenario of the data storage process needs to be acquired, so as to determine a corresponding data reconstruction policy according to the types of different application scenarios in the following.
And S250, determining whether the type of the application scene is the operation priority.
If yes, go to S260; if not, go to step S270.
And S260, stopping the data reconstruction operation of the target data service when the data storage amount exceeds a first preset threshold value.
The first preset threshold may be preset, or may be determined according to specific situations, and this embodiment is not limited.
If the type of the application scene is operation priority, it is mainly ensured that the data can be smoothly stored at the moment. Therefore, when the data storage amount exceeds the first preset threshold, since the data reconstruction operation is also dependent on the network, the data reconstruction operation of the target data service needs to be stopped in order not to increase the load on the network.
And S270, when the data storage amount exceeds a second preset threshold value, controlling data reconstruction operation of the target data service according to the corresponding flow information.
The second preset threshold may be preset, or may be determined according to a specific situation, and this embodiment is not limited. In this embodiment, the magnitude relationship between the first preset threshold and the second preset threshold is not limited.
If the type of the application scene is data security priority, the data security is mainly guaranteed at the moment. Therefore, when the data storage amount exceeds the second preset threshold, the data reconstruction operation of the target data service needs to be controlled according to the corresponding traffic information, that is, the traffic information of the scene, and the progress of the data reconstruction operation can be specifically controlled, so that the data security is ensured while the data is stored.
In this embodiment, a delay condition and a packet loss rate in a data storage process are first obtained, a network condition map is constructed according to the delay condition and the packet loss rate in the data storage process, then, a target data service with network congestion is determined based on the delay map and the packet loss rate map, then, a type of an application scenario in the data storage process is obtained, and finally, whether the type of the application scenario is operation priority or not is determined, and if the type of the application scenario is operation priority, when a data storage amount exceeds a first preset threshold, a data reconstruction operation of the target data service is stopped. If the type of the application scenario is data security priority, when the data storage amount exceeds a second preset threshold, controlling data reconstruction operation of a target data service according to corresponding flow information, and determining a corresponding data reconstruction strategy according to the types of different application scenarios, so that network congestion caused by a data reconstruction process is relieved, memory card pause is avoided, and the data storage performance is further improved.
In this embodiment, optionally, the obtaining of the delay condition and the packet loss rate in the data storage process may specifically include:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output (IO) request packets and a second number of received IO response packets which are sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and the second number.
Specifically, by using a heartbeat mechanism, after a process of a virtual disk in an access service sends a heartbeat packet to a process of a physical disk storing virtual disk data at regular time, the process of the physical disk returns a heartbeat response packet to the virtual disk process after receiving the heartbeat packet, and if the time for the virtual disk process to receive the heartbeat response packet exceeds preset timeout time, the heartbeat response packet is returned to be overtime; if the time for the virtual disk process to receive the heartbeat response packet does not exceed the preset timeout time, the heartbeat response packet is not returned overtime, so that the specific delay condition in the data storage process can be determined according to the time for the virtual disk process to receive the heartbeat response packet.
The packet loss rate can be judged based on the stored IO request, the packet loss usually occurs in a busy network state, the number of stored IO times per second is very high, and a large amount of CPU resources are consumed based on counting the number of lost packets by each IO. Therefore, in the embodiment, the packet loss rate of the current distributed storage system is counted by a window counting method.
The time length set by the time window of the window statistical method may be equal to the IO request timeout time set by the distributed storage system, seq-s: indicating the total number of IO request packets that have been successfully transmitted at a certain time. seq-r: indicating the total number of IO response packets that have been successfully received at the base time. For example, the IO request packet recorded by the time window 1 is seq-s1, the IO request packet recorded by the time window 2 is seq-s2, and so on; the IO response packet for which time Window 1 opens a record is seq-r1, for which time Window 2 opens a record is seq-r2, and so on.
For example, when seq-s2-seq-r3< ═ 0, it indicates that all IO requests sent by time window 1 have been received, at this time, there is no packet loss in time window 1, and the packet loss rate at the time when time window 4 is opened is 0; when seq-s2-seq-r3>0, it indicates that there is packet loss in the time window 1, the total number of packet loss in this time is seq-s2-seq-r3, and the number of IO requests sent is: seq-s2-seq-s1, therefore, the packet loss rate at the time when the time window 4 is opened is: (seq-s2-seq-r3)/(seq-s2-seq-s 1).
Therefore, the packet loss rate in the data storage process can be calculated by combining the first number of the IO request packets and the second number of the received IO response packets sent by each access service with the above formula.
In this embodiment, although each virtual disk calculates its own packet loss rate, the calculation method is simple and convenient, the overhead of the system can be ignored, and the statistics of the packet loss rate has no influence on the load of the network itself.
In this embodiment, optionally, each IO request packet needs to be written/read in sequence.
In this embodiment, in order to ensure that the writing order of the storage data is not disturbed by asynchronous network transmission, each IO request packet may be numbered, for example, by using a 64-bit positive integer seq as an identifier, and the seq is incremented by 1 with each IO request packet. And the data service writes the received IO request packets into/reads the disk according to the seq sequence and sends IO response packets to the access service. The access service assembles acknowledgements in order of increasing 1 according to the seq of the IO response packet. Meanwhile, in order to avoid the situation that old data overwrites new data, seq-r receives acknowledgements by incrementing.
In this embodiment, optionally, the method further includes:
acquiring the network card utilization rate in the data storage process;
and when the utilization rate of the network card exceeds a target threshold value, suspending the working process of the heartbeat mechanism.
The target threshold may be preset, or may be determined according to specific situations, and this embodiment is not limited.
Specifically, when the number of virtual disks in the access service is large, the number of TCP connections to be detected is also large, and the time interval for sending the heartbeat packet based on the heartbeat mechanism is determined by the following formula:
time interval for sending heartbeat packet is preset time interval (1+ network card utilization rate)/(1-network card utilization rate)
According to the formula, when the network card utilization rate is high, the heartbeat packet may aggravate network congestion, so that the network card utilization rate in the data storage process can be obtained through the operating system, and when the network card utilization rate exceeds a target threshold value, the working process of the heartbeat mechanism is suspended. Thereby avoiding further congestion of the network due to the sending of heartbeat packets.
In this embodiment, the heartbeat mechanism is combined with a window statistical method, so that whether network congestion occurs or not can be determined with extremely low network overhead, and a link with the network congestion can be determined. Especially, under the condition that the network is congested, the working process of a heartbeat mechanism can be automatically suspended, and only a window statistical method is adopted, so that the overtime detection frequency is greatly reduced, and the system performance is favorably improved.
Fig. 4 is a schematic structural diagram of an apparatus for reducing network congestion according to an embodiment of the present disclosure; the device is configured in computer equipment, and can realize the method for reducing network congestion in any embodiment of the application. The device specifically comprises the following steps:
the map building module 310 is configured to build a network condition map according to a delay condition and a packet loss rate in a data storage process, where the network condition map includes a delay map and a packet loss rate map;
a first determining module 320, configured to determine, based on the delay map and the packet loss rate map, a target data service where network congestion occurs;
a second determining module 330, configured to determine a data reconstruction policy of the target data service according to an application scenario of the data storage process.
In this embodiment, optionally, the second determining module 330 is specifically configured to:
acquiring the type of an application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is operation priority, stopping data reconstruction operation of the target data service when the data storage exceeds a first preset threshold;
and if the type is data security priority, controlling data reconstruction operation of the target data service according to the corresponding flow information when the data storage exceeds a second preset threshold.
In this embodiment, optionally, the apparatus further includes:
the acquisition module is used for acquiring the delay condition and the packet loss rate in the data storage process before constructing the network condition map according to the delay condition and the packet loss rate in the data storage process.
In this embodiment, optionally, the obtaining module is specifically configured to:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output (IO) request packets and a second number of received IO response packets which are sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and the second number.
In this embodiment, optionally, the apparatus further includes:
the utilization rate acquisition module is used for acquiring the network card utilization rate in the data storage process;
and the pause module is used for pausing the working process of the heartbeat mechanism when the utilization rate of the network card exceeds a target threshold value.
In this embodiment, optionally, the apparatus further includes:
and the updating module is used for updating the network condition map under the preset triggering condition.
In this embodiment, optionally, the update module is specifically configured to:
when the network is detected to be abnormal, updating the network condition map;
or,
when no network abnormality is detected, the network condition map is updated by a method of carrying out weighted average on the newly constructed network condition map and the historical network condition map within the preset time.
According to the device for reducing network congestion provided by the embodiment of the disclosure, a network condition map is firstly constructed according to delay conditions and packet loss rates in a data storage process, wherein the network condition map comprises a delay map and a packet loss rate map, then a target data service with network congestion is determined based on the delay map and the packet loss rate map, and finally a data reconstruction strategy of the target data service is determined according to an application scene in the data storage process.
The device for reducing network congestion provided by the embodiment of the disclosure can execute the method for reducing network congestion provided by any embodiment of the disclosure, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure. As shown in fig. 5, the computer apparatus includes a processor 410 and a storage device 420; the number of the processors 410 in the computer device may be one or more, and one processor 410 is taken as an example in fig. 5; the processor 410 and the storage 420 in the computer device may be connected by a bus or other means, and fig. 5 illustrates the connection by a bus as an example.
The storage device 420 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the method for reducing network congestion in the embodiments of the present disclosure. The processor 410 executes various functional applications and data processing of the computer device by executing software programs, instructions and modules stored in the storage 420, namely, implements the method for reducing network congestion provided by the embodiments of the present disclosure.
The storage device 420 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the storage 420 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the storage 420 may further include memory located remotely from the processor 410, which may be connected to a computer device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The computer device provided by the present embodiment can be used to execute the method for reducing network congestion provided by any of the above embodiments, and has corresponding functions and advantages.
The disclosed embodiments also provide a storage medium containing computer-executable instructions for implementing the method for reducing network congestion provided by the disclosed embodiments when executed by a computer processor.
Of course, the storage medium provided by the embodiments of the present disclosure contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the method for reducing network congestion provided by any embodiments of the present disclosure.
From the above description of the embodiments, it is obvious for a person skilled in the art that the present disclosure can be implemented by software and necessary general hardware, and certainly can be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present disclosure.
It should be noted that, in the embodiment of the apparatus for reducing network congestion, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only used for distinguishing one functional unit from another, and are not used for limiting the protection scope of the present disclosure.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method of reducing network congestion, the method comprising:
constructing a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map;
determining a target data service with network congestion based on the delay map and the packet loss rate map;
and determining a data reconstruction strategy of the target data service according to the application scene of the data storage process.
2. The method of claim 1, wherein determining the data reconstruction policy of the target data service according to the application scenario of the data storage process comprises:
acquiring the type of an application scene of the data storage process, wherein the type comprises operation priority and data security priority;
if the type is operation priority, stopping data reconstruction operation of the target data service when the data storage exceeds a first preset threshold;
and if the type is data security priority, controlling data reconstruction operation of the target data service according to the corresponding flow information when the data storage exceeds a second preset threshold.
3. The method according to claim 1, before constructing the network condition map according to the delay condition and the packet loss rate in the data storage process, further comprising:
and acquiring the delay condition and the packet loss rate in the data storage process.
4. The method according to claim 3, wherein the obtaining of the delay condition and the packet loss rate in the data storage process includes:
based on a heartbeat mechanism, determining the time when each access service respectively receives a heartbeat response packet sent by a physical disk in the corresponding data service, and determining the delay condition in the data storage process according to all the time;
determining a first number of input/output (IO) request packets and a second number of received IO response packets which are sent by each access service when each time window is opened according to a window statistical method;
and calculating the packet loss rate in the data storage process according to the corresponding first number and the second number.
5. The method of claim 4, further comprising:
acquiring the network card utilization rate in the data storage process;
and when the utilization rate of the network card exceeds a target threshold value, suspending the working process of the heartbeat mechanism.
6. The method of claim 1, further comprising:
and updating the network condition map under a preset triggering condition.
7. The method according to claim 6, wherein the updating the network condition map under a preset trigger condition comprises:
when the network is detected to be abnormal, updating the network condition map;
or,
when no network abnormality is detected, the network condition map is updated by a method of carrying out weighted average on the newly constructed network condition map and the historical network condition map within the preset time.
8. An apparatus for reducing network congestion, the apparatus comprising:
the map building module is used for building a network condition map according to the delay condition and the packet loss rate in the data storage process, wherein the network condition map comprises a delay map and a packet loss rate map;
a first determining module, configured to determine, based on the delay map and the packet loss rate map, a target data service in which network congestion occurs;
and the second determining module is used for determining a data reconstruction strategy of the target data service according to the application scene of the data storage process.
9. A computer device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111275790.0A CN114007246B (en) | 2021-10-29 | 2021-10-29 | Method, apparatus, computer device and medium for reducing network congestion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111275790.0A CN114007246B (en) | 2021-10-29 | 2021-10-29 | Method, apparatus, computer device and medium for reducing network congestion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114007246A true CN114007246A (en) | 2022-02-01 |
| CN114007246B CN114007246B (en) | 2024-02-02 |
Family
ID=79925648
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111275790.0A Active CN114007246B (en) | 2021-10-29 | 2021-10-29 | Method, apparatus, computer device and medium for reducing network congestion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114007246B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2922347A1 (en) * | 2007-10-11 | 2009-04-17 | Bouchaib Hoummady | Vehicle e.g. car, mobility and traffic guiding managing method, involves simulating temporal spatial load of traffic in mobility space, evaluating pertinent situations in traffic, and transferring data corresponding to circulation situation |
| CN103428240A (en) * | 2012-05-18 | 2013-12-04 | 深圳市腾讯计算机系统有限公司 | Data transmission method and data transmission terminal |
| CN109088794A (en) * | 2018-08-20 | 2018-12-25 | 郑州云海信息技术有限公司 | A kind of fault monitoring method and device of node |
| US20200028782A1 (en) * | 2018-07-17 | 2020-01-23 | Facebook, Inc. | Detecting Communication Network Insights Of Alerts |
| CN111510338A (en) * | 2020-03-09 | 2020-08-07 | 苏州浪潮智能科技有限公司 | Distributed block storage network sub-health test method, device and storage medium |
| US20200351201A1 (en) * | 2018-07-17 | 2020-11-05 | Facebook, Inc. | Detecting communication network insights of alerts |
-
2021
- 2021-10-29 CN CN202111275790.0A patent/CN114007246B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2922347A1 (en) * | 2007-10-11 | 2009-04-17 | Bouchaib Hoummady | Vehicle e.g. car, mobility and traffic guiding managing method, involves simulating temporal spatial load of traffic in mobility space, evaluating pertinent situations in traffic, and transferring data corresponding to circulation situation |
| CN103428240A (en) * | 2012-05-18 | 2013-12-04 | 深圳市腾讯计算机系统有限公司 | Data transmission method and data transmission terminal |
| US20200028782A1 (en) * | 2018-07-17 | 2020-01-23 | Facebook, Inc. | Detecting Communication Network Insights Of Alerts |
| US20200351201A1 (en) * | 2018-07-17 | 2020-11-05 | Facebook, Inc. | Detecting communication network insights of alerts |
| CN109088794A (en) * | 2018-08-20 | 2018-12-25 | 郑州云海信息技术有限公司 | A kind of fault monitoring method and device of node |
| CN111510338A (en) * | 2020-03-09 | 2020-08-07 | 苏州浪潮智能科技有限公司 | Distributed block storage network sub-health test method, device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 周有喜;肖嵩;吴成柯;: "基于场景模型的视频无线信道鲁棒传输", 宇航学报, no. 01, pages 108 - 112 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114007246B (en) | 2024-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11467975B2 (en) | Data processing method and NVMe storage device | |
| US11886731B2 (en) | Hot data migration method, apparatus, and system | |
| JP7039685B2 (en) | Traffic measurement methods, devices, and systems | |
| US10768857B2 (en) | Storage system having a controller that selects a die of a solid state disk to store data | |
| KR20210042161A (en) | Data synchronization method and apparatus, medium, and electronic device for distributed system | |
| EP3637708B1 (en) | Network congestion processing method, device, and system | |
| US9374425B2 (en) | Behavior based client selection for disparate treatment | |
| US11888744B2 (en) | Spin-leaf network congestion control method, node, system, and storage medium | |
| US11003359B2 (en) | Method and device for managing disk pool | |
| CN110659151A (en) | Data verification method and device and storage medium | |
| WO2018068714A1 (en) | Deduplication processing method, and storage device | |
| US20180088955A1 (en) | Method and system for managing data access in storage system | |
| EP4407957A1 (en) | Data processing method and apparatus, computer device and readable storage medium | |
| CN117370138A (en) | High capacity distributed storage system | |
| CN109274550B (en) | iSCSI self-adaptive IO queue depth matching method | |
| CN114785739A (en) | Method, device, equipment and medium for controlling service quality of logical volume | |
| CN114007246A (en) | Method, apparatus, computer device and medium for reducing network congestion | |
| US20170269864A1 (en) | Storage Array Operation Method and Apparatus | |
| CN110677463B (en) | Parallel data transmission method, device, medium and electronic equipment | |
| CN104506663B (en) | A kind of intelligent cloud computing operation management system | |
| EP4325374A1 (en) | Data traffic control method and related device | |
| CN113849129B (en) | IO request forwarding method, device and equipment among storage controllers | |
| CN114647632B (en) | Method, system and storage medium for database load control | |
| US12147845B2 (en) | Virtual machine migration based on network usage | |
| CN105391765B (en) | A kind of method and system of content storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |