CN111641698A - Data statistical method, system, equipment and storage medium - Google Patents
Data statistical method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN111641698A CN111641698A CN202010442521.8A CN202010442521A CN111641698A CN 111641698 A CN111641698 A CN 111641698A CN 202010442521 A CN202010442521 A CN 202010442521A CN 111641698 A CN111641698 A CN 111641698A
- Authority
- CN
- China
- Prior art keywords
- gateway
- object storage
- data
- address information
- service quality
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/142—Network analysis or design using statistical or mathematical methods
Abstract
The application discloses a data statistical method, a system, equipment and a storage medium. In the method, each object storage gateway directionally sends the service quality data to the main gateway in the object storage gateways for statistics, thereby avoiding network delay caused by the generation of broadcast message storm and relatively ensuring the reliability and accuracy of the process of summarizing the service quality data of the object storage gateways. In addition, the application also provides a data statistical system, equipment and a storage medium, and the beneficial effects are as described above.
Description
Technical Field
The present application relates to the field of distributed object storage, and in particular, to a data statistics method, system, device, and storage medium.
Background
Currently, with the advent of the internet of things, millions of devices in an enterprise create, receive, and send data at all times, and the data level is also stepped up from PB to ZB level.
In the face of massive unstructured data, enterprises need to effectively retain, protect and analyze the data, and exert the value of the data to the maximum extent. As a novel network storage architecture, distributed object storage adopts an object data organization, and a distributed storage architecture with high performance, high reliability, cross-platform and safe data sharing is provided.
Although the distributed object storage has the capability of storing massive unstructured data, in the distributed object storage, how to aggregate the service quality data of the object storage gateways in the device nodes in the distributed cluster so as to better control the service quality of the distributed object storage is a technical problem in the current industry. According to the current scheme, after each object storage gateway generates service quality data, the service quality data are respectively sent to other object storage gateways in a broadcast mode, so that the service quality data are collected and counted.
Therefore, it is a problem to be solved by those skilled in the art to provide a data statistics method to relatively ensure the reliability and accuracy of the summarizing process of the quality of service data of the object storage gateway.
Disclosure of Invention
The application aims to provide a data statistical method, a system, equipment and a storage medium, so as to relatively ensure the reliability and the accuracy of the summarizing process of the service quality data of an object storage gateway.
In order to solve the above technical problem, the present application provides a data statistics method, applied to a monitoring end in a distributed object storage cluster, including:
acquiring gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster;
selecting a main gateway from the object storage gateways according to the gateway attribute information;
and sending the communication address information of the main gateway to each object storage gateway so as to control other gateways except the main gateway to send service quality data to the main gateway and control the main gateway to perform summary statistics on the service quality data of each object storage gateway.
Preferably, the gateway attribute information includes communication address information and a process number;
selecting a main gateway from the object storage gateways according to the gateway attribute information, comprising the following steps:
and selecting the target object storage gateway with the minimum process number value from the object storage gateways, and setting the target object storage gateway as a main gateway.
Preferably, the communication address information includes IP address information and port number information.
Preferably, the service quality data includes one or more of the number of processed requests, uploaded traffic, downloaded traffic, and traffic data generation amount per unit time.
In addition, the present application further provides a data statistics method, applied to an object storage gateway in a distributed object storage cluster, including:
sending gateway attribute information to a monitoring end in the distributed object storage cluster;
receiving communication address information transmitted by a monitoring end;
judging whether the communication address information is a local gateway address;
if the communication address information is the local gateway address, receiving service quality data transmitted by other gateways in the distributed object storage cluster, and performing summary statistics by combining the local service quality data;
and if the communication address information is not the local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information.
Preferably, after performing summary statistics in combination with the local quality of service data, the method further includes:
and sending the summarized data obtained after the summary statistics to other gateways.
Preferably, sending the gateway attribute information to the monitoring end in the distributed object storage cluster includes:
and sending gateway attribute information to the monitoring end according to the preset frequency.
In addition, the present application also provides a data statistics system, including:
the monitoring terminal is used for acquiring gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster; selecting a main gateway from the object storage gateways according to the gateway attribute information; sending the communication address information of the main gateway to each object storage gateway to control other gateways except the main gateway to send service quality data to the main gateway and control the main gateway to perform summary statistics on the service quality data of each object storage gateway;
the object storage gateway is used for sending gateway attribute information to a monitoring end in the distributed object storage cluster; receiving communication address information transmitted by a monitoring end; judging whether the communication address information is a local gateway address; if the communication address information is the local gateway address, receiving service quality data transmitted by other gateways in the distributed object storage cluster, and performing summary statistics by combining the local service quality data; and if the communication address information is not the local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information.
In addition, this application still provides a data statistics equipment, includes:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the data statistics method applied to the monitoring end in the distributed object storage cluster and/or implement the steps of the data statistics method applied to the object storage gateway in the distributed object storage cluster.
Furthermore, the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when being executed by a processor, implements the steps of the data statistics method applied to the monitoring end in the distributed object storage cluster and/or implements the steps of the data statistics method applied to the object storage gateway in the distributed object storage cluster.
The data statistical method includes the steps that a monitoring end in a distributed object storage cluster obtains gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster, a main gateway is selected from each object storage gateway according to the gateway attribute information, communication address information of the main gateway is sent to each object storage gateway, each object storage gateway receives the communication address information transmitted by the monitoring end, whether the communication address information is a local gateway address or not is further judged, when the communication address information is the local gateway address, service quality data transmitted by other gateways in the distributed object storage cluster are received, and summary statistics is carried out by combining the local service quality data; and when the communication address information is not the local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information so as to perform summary statistics through the target object storage gateway. In the method, each object storage gateway directionally sends the service quality data to the main gateway in the object storage gateways for statistics, so that network delay caused by the generation of broadcast message storm is avoided, and the reliability and the accuracy of the process of summarizing the service quality data of the object storage gateways are relatively ensured. In addition, the application also provides a data statistical system, equipment and a storage medium, and the beneficial effects are as described above.
Drawings
In order to more clearly illustrate the embodiments of the present application, the drawings needed for the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a data statistics method applied to a monitoring end in a distributed object storage cluster according to an embodiment of the present application;
FIG. 2 is a flowchart of a data statistics method applied to an object storage gateway in a distributed object storage cluster according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a distributed object storage cluster structure in an embodiment of the present application;
FIG. 4 is a timing diagram of a communication in an embodiment of the present application scenario;
fig. 5 is a schematic structural diagram of a data statistics system disclosed in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without any creative effort belong to the protection scope of the present application.
In the face of massive unstructured data, enterprises need to effectively retain, protect and analyze the data, and exert the value of the data to the maximum extent. As a novel network storage architecture, distributed object storage adopts an object data organization, and a distributed storage architecture with high performance, high reliability, cross-platform and safe data sharing is provided.
Although the distributed object storage has the capability of storing massive unstructured data, in the distributed object storage, how to aggregate the service quality data of the object storage gateways in the device nodes in the distributed cluster so as to better control the service quality of the distributed object storage is a technical problem in the current industry. According to the current scheme, after each object storage gateway generates service quality data, the service quality data are respectively sent to other object storage gateways in a broadcast mode, so that the service quality data are collected and counted.
Therefore, the core of the application is to provide a data statistical method to relatively ensure the reliability and accuracy of the summarizing process of the service quality data of the object storage gateway.
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings.
Referring to fig. 1, an embodiment of the present application discloses a data statistics method, which is applied to a monitoring end in a distributed object storage cluster, and includes:
step S10: and acquiring gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster.
It should be noted that the execution main body of this embodiment is a monitoring end in the distributed object storage cluster, and the purpose of the monitoring end is to select a main gateway for performing summary statistics on the service quality data of each object storage gateway from each object storage gateway in the object storage cluster. The monitoring end may be an equipment node in which a monitoring service process runs alone in the distributed object storage cluster, or an equipment node in which a monitoring service process and other business processes run simultaneously in the distributed object storage cluster, which is not specifically limited herein, that is, the monitoring end may be considered as an equipment node in which a monitoring service process runs in the distributed object storage cluster. In addition, the device node referred to herein may be a physical node, or may be a virtual node obtained by virtualization based on the physical node, also depending on the actual scene of the distributed object storage cluster.
In this step, each object storage gateway in the distributed object storage cluster transmits its own gateway attribute information to the monitoring end, where the gateway attribute information refers to description information of the object storage gateway, and the characteristics of the object storage gateway can be obtained according to the gateway attribute information.
Step S11: and selecting a main gateway from the object storage gateways according to the gateway attribute information.
After receiving gateway attribute information transmitted by the object storage gateways, a monitoring end in the distributed object storage cluster further selects a main gateway from the object storage gateways according to the gateway attribute information, wherein the main gateway is used for carrying out unified statistics on service quality data of all the object storage gateways in the object storage gateways.
Step S12: and sending the communication address information of the main gateway to each object storage gateway so as to control other gateways except the main gateway to send service quality data to the main gateway and control the main gateway to perform summary statistics on the service quality data of each object storage gateway.
After the monitoring end selects the main gateway from the object storage gateways according to the gateway attribute information, the step further sends the communication address information of the main gateway to each object storage gateway, so as to inform each object storage gateway of the communication address of the main gateway, ensure that the main gateway in each object storage gateway can normally perform statistics on the service quality data, so that other gateways except the main gateway normally send the service quality data to the main gateway, and further achieve the purposes of controlling other gateways except the main gateway in each object storage gateway to send the service quality data to the main gateway and controlling the main gateway to perform summary statistics on the service quality data of each object storage gateway.
The data statistical method includes the steps that a monitoring end in a distributed object storage cluster obtains gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster, a main gateway is selected from each object storage gateway according to the gateway attribute information, communication address information of the main gateway is sent to each object storage gateway, each object storage gateway receives the communication address information transmitted by the monitoring end, whether the communication address information is a local gateway address or not is further judged, when the communication address information is the local gateway address, service quality data transmitted by other gateways in the distributed object storage cluster are received, and summary statistics is carried out by combining the local service quality data; and when the communication address information is not the local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information so as to perform summary statistics through the target object storage gateway. In the method, each object storage gateway directionally sends the service quality data to the main gateway in the object storage gateways for statistics, so that network delay caused by the generation of broadcast message storm is avoided, and the reliability and the accuracy of the process of summarizing the service quality data of the object storage gateways are relatively ensured.
On the basis of the above embodiment, as a preferred implementation, the gateway attribute information includes communication address information and a process number;
selecting a main gateway from the object storage gateways according to the gateway attribute information, comprising the following steps:
and selecting the target object storage gateway with the minimum process number value from the object storage gateways, and setting the target object storage gateway as a main gateway.
It should be noted that the key point of this embodiment is that the gateway attribute information includes communication address information of the object storage gateway and a service process number of the object storage gateway, where the service process number refers to an identification number that a process running the object storage gateway has, and is used to uniquely characterize an identity of the process; the communication address information refers to information related to an access address of the object storage gateway.
In the embodiment, the target object storage gateway with the minimum server process number value in the object storage gateways is selected as the main gateway, so that the overall efficiency of the process of selecting the main gateway from the object storage gateways can be relatively improved, and the overall efficiency of the data statistics process is further improved.
Further, the communication address information includes IP address information and port number information.
In the present embodiment, the communication Address information is further subdivided into IP Address (internet protocol Address) information and port number information. The IP address is a uniform address format provided by the IP protocol, and the IP protocol allocates a logical address for each network and each host on the Internet, so that the IP address information can only correspond to the equipment node where the main gateway is located; furthermore, since the process of the object storage gateway may be run in the same device node, and at the same time, other service processes may also be run, the device node often distinguishes access paths of a plurality of services through different port numbers, and the communication address information of the present embodiment includes IP address information and port number information, and can relatively ensure reliability of a process of sending the quality of service data to the main gateway by other gateways other than the main gateway.
On the basis of the above series of embodiments, as a preferred embodiment, the quality of service data includes one or more of the number of processed requests, the uploaded traffic, the downloaded traffic, and the traffic data generation amount per unit time.
It should be noted that the present embodiment further refines the qos data, and includes one or more of the processed request number, the uploaded traffic number, the downloaded traffic number, and the traffic data generation amount per unit time, where the processed request number refers to the number of network access requests that the target storage gateway has responded to, the uploaded traffic number refers to the amount of data input through the target storage gateway, the downloaded traffic number refers to the amount of data output through the target storage gateway, and the traffic data generation amount per unit time refers to the amount of traffic data generated by the target storage gateway in unit time. The embodiment can relatively improve the overall accuracy of summarizing and counting the service quality data of each object storage gateway.
Referring to fig. 2, an embodiment of the present application discloses a data statistics method applied to an object storage gateway in a distributed object storage cluster, including:
step S20: and sending gateway attribute information to a monitoring end in the distributed object storage cluster.
It should be noted that the execution subject of this embodiment is an object storage gateway in a distributed object storage cluster, where the object storage gateway may be considered as a device node running an object storage gateway process in the distributed object storage cluster, and other service processes are often run in the device node running the object storage gateway, and the object storage gateway is used to receive and transmit service data of the device node, and is used to provide support for network communication of the service processes between the device nodes, so that the object storage gateway further generates service quality data according to relevant state information of the network communication.
In this step, the object storage gateway sends gateway attribute information to the monitoring terminal in the distributed object storage cluster, so that the monitoring terminal in the distributed object storage cluster can further select a master gateway in the object storage gateway according to the gateway attribute information, and further send the communication address information of the master gateway to each object storage gateway after the monitoring terminal selects the master gateway in the object storage gateway according to the gateway attribute information.
Step S21: and receiving communication address information transmitted by the monitoring terminal.
And sending gateway attribute information to a monitoring end in the distributed object storage cluster, wherein the object storage gateway further receives communication address information transmitted by the monitoring end.
Step S22: and judging whether the communication address information is a local gateway address, if so, executing the step S23, otherwise, executing the step S24.
Step S23: and receiving the service quality data transmitted by other gateways in the distributed object storage cluster, and performing summary statistics by combining the local service quality data.
Step S24: and sending the local service quality data to a target object storage gateway corresponding to the communication address information.
After receiving the communication address information transmitted by the monitoring end, the object storage gateway further judges whether the object storage gateway is a main gateway or not by judging whether the communication address information is a local gateway address or not, if the communication address information is the local gateway address, the object storage gateway judges that the object storage gateway is the main gateway, the object storage gateway further receives service quality data transmitted by other gateways in the distributed object storage cluster, and performs summary statistics by combining the local service quality data; if the communication address information is not the local gateway address, judging that the communication address information is not the main gateway, and sending the local service quality data to a target object storage gateway corresponding to the communication address information so as to be used by the target object storage gateway for summarizing and counting the service quality data of each object storage gateway.
The data statistical method includes the steps that a monitoring end in a distributed object storage cluster obtains gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster, a main gateway is selected from each object storage gateway according to the gateway attribute information, communication address information of the main gateway is sent to each object storage gateway, each object storage gateway receives the communication address information transmitted by the monitoring end, whether the communication address information is a local gateway address or not is further judged, when the communication address information is the local gateway address, service quality data transmitted by other gateways in the distributed object storage cluster are received, and summary statistics is carried out by combining the local service quality data; and when the communication address information is not the local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information so as to perform summary statistics through the target object storage gateway. In the method, each object storage gateway directionally sends the service quality data to the main gateway in the object storage gateways for statistics, so that network delay caused by the generation of broadcast message storm is avoided, and the reliability and the accuracy of the process of summarizing the service quality data of the object storage gateways are relatively ensured.
On the basis of the foregoing embodiment, as a preferred implementation manner, after performing summary statistics in combination with the local quality of service data, the method further includes:
and sending the summarized data obtained after the summary statistics to other gateways.
It should be noted that, in the embodiment, after the summarized data is obtained by performing summary statistics in combination with the local service quality data, the summarized data is further sent to other gateways, so as to ensure that other gateways can learn the overall network service quality, and facilitate adjustment of network services in a targeted manner by each object storage gateway, thereby further improving the storage performance of the distributed object storage cluster.
Further, as a preferred embodiment, sending gateway attribute information to a monitoring end in a distributed object storage cluster includes:
and sending gateway attribute information to the monitoring end according to the preset frequency.
In this embodiment, the sending of the gateway attribute information to the monitoring terminal at the preset frequency refers to sending the gateway attribute information to the monitoring terminal at the preset time interval, so as to ensure that the monitoring terminal can reliably receive the gateway attribute information, thereby ensuring the accuracy of data statistics.
In order to further deepen understanding of the technical scheme of the application, the application provides a scene embodiment in a specific application scene, and the scene embodiment is as follows:
the distributed object storage cluster structure in this scenario embodiment is shown in fig. 3, and the architecture of the distributed object storage cluster is that 1 or more distributed object storage gateways exist on each device node. A main cluster monitoring service and a plurality of standby cluster monitoring services exist in the whole distributed object storage cluster at the same time, the total number n of the cluster monitoring services is 1/2 which is less than or equal to the total number of the equipment nodes and is greater than the number of the equipment nodes, and odd numbers are taken.
When the main cluster monitoring service fails, other standby cluster monitoring services are selected as the main cluster monitoring service, so that the scheme always exists according to the main cluster monitoring service.
When the distributed object storage cluster starts to provide services, the distributed object storage gateways on the equipment nodes send respective IP, port and process number information to the main cluster monitoring service in a network data packet mode in a directional mode, the main cluster monitoring service selects the main distributed object storage gateway from the main distributed object storage gateway according to the rule of the minimum process number, and then the IP and the port number of the main distributed object storage gateway are sent to the distributed object storage gateways in a directional mode.
Each distributed object storage gateway sends service quality data (including the number of requests processed by the current gateway, the uploaded traffic, the downloaded traffic, the current amount of business data generated per second and the like) counted in each process to the main distributed object storage gateway service in a network data packet mode at regular time (user configurable), the main distributed object storage gateway service performs data counting and summarizing after receiving the data packets, and then replies the summarized data to each distributed object storage gateway in the network data packet mode.
Fig. 4 shows a timing chart of communication between the master cluster monitoring service and the distributed object storage gateway in this scenario embodiment.
The specific implementation represented by the timing diagram is as follows:
1) there are 3 nodes in the distributed object storage cluster, of which there are 1 master cluster monitoring service and 3 distributed object storage gateway services.
2) According to the time interval set by the user, for example, every 3 seconds, the distributed object storage gateways 1, 2 and n respectively send own IP, port and process number to the main cluster monitoring service in a network data packet mode in an oriented mode.
3) And the main cluster monitoring service selects the distributed object storage gateway 2 with the minimum process number as a main cluster according to the compared process numbers.
4) And the IP and the port of the distributed object storage gateway 2 are directionally sent to the distributed object storage gateways 1, 2 and n in the form of network data packets.
5) The distributed object storage gateways 1 and n directionally send the process service quality data which are independently counted to the distributed object storage gateway 2 in a network data packet mode; since the distributed object storage gateway 2 is the master, it does not send network packets to itself.
6) The distributed object storage gateway 2 counts the received 1 and n service quality data, and replies the aggregate data to the distributed object storage gateway 1 and n in the form of network data packets.
7) And repeating the steps 2-6 to count and summarize the service quality data of the distributed object storage gateway independently at different equipment nodes.
Referring to fig. 5, an embodiment of the present application provides a data statistics system, including:
the monitoring terminal 10 is configured to acquire gateway attribute information transmitted by each object storage gateway 11 in the distributed object storage cluster; selecting a main gateway from the object storage gateways 11 according to the gateway attribute information; sending the communication address information of the main gateway to each object storage gateway 11 to control other gateways except the main gateway to send service quality data to the main gateway and control the main gateway to perform summary statistics on the service quality data of each object storage gateway 11;
the object storage gateway 11 is used for sending gateway attribute information to the monitoring terminal 10 in the distributed object storage cluster; receiving communication address information transmitted by the monitoring terminal 10; judging whether the communication address information is a local gateway address; if the communication address information is the local gateway address, receiving service quality data transmitted by other gateways in the distributed object storage cluster, and performing summary statistics by combining the local service quality data; if the communication address information is not the local gateway address, the local quality of service data is transmitted to the target object storage gateway 11 corresponding to the communication address information.
The data statistical system comprises a monitoring end in a distributed object storage cluster, a monitoring end in the distributed object storage cluster and a plurality of object storage gateways, wherein the monitoring end acquires gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster, further selects a main gateway from each object storage gateway according to the gateway attribute information, transmits communication address information of the main gateway to each object storage gateway, each object storage gateway receives the communication address information transmitted by the monitoring end and further judges whether the communication address information is a local gateway address, and when the communication address information is the local gateway address, the data statistical system receives service quality data transmitted by other gateways in the distributed object storage cluster and performs summary statistics by combining the local service quality data; and when the communication address information is not the local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information so as to perform summary statistics through the target object storage gateway. In the system, each object storage gateway directionally sends the service quality data to the main gateway in the object storage gateways for statistics, so that network delay caused by the generation of broadcast message storm is avoided, and the reliability and the accuracy of the process of summarizing the service quality data of the object storage gateways are relatively ensured.
In addition, an embodiment of the present application further provides a data statistics apparatus, including:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the data statistics method applied to the monitoring end in the distributed object storage cluster and/or implement the steps of the data statistics method applied to the object storage gateway in the distributed object storage cluster.
According to the data statistics device, a monitoring end in a distributed object storage cluster acquires gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster, a main gateway is selected from each object storage gateway according to the gateway attribute information, communication address information of the main gateway is transmitted to each object storage gateway, each object storage gateway receives the communication address information transmitted by the monitoring end, whether the communication address information is a local gateway address or not is further judged, when the communication address information is the local gateway address, service quality data transmitted by other gateways in the distributed object storage cluster are received, and summary statistics is carried out by combining the local service quality data; and when the communication address information is not the local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information so as to perform summary statistics through the target object storage gateway. In the equipment, each object storage gateway directionally sends the service quality data to the main gateway in the object storage gateways for statistics, so that network delay caused by the generation of broadcast message storm is avoided, and the reliability and the accuracy of the process of summarizing the service quality data of the object storage gateways are relatively ensured.
Furthermore, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when being executed by a processor, the computer program implements the steps of the data statistics method applied to the monitoring end in the distributed object storage cluster and/or implements the steps of the data statistics method applied to the object storage gateway in the distributed object storage cluster.
The computer-readable storage medium provided by the application is characterized in that a monitoring end in a distributed object storage cluster acquires gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster, a main gateway is selected from each object storage gateway according to the gateway attribute information, communication address information of the main gateway is transmitted to each object storage gateway, each object storage gateway receives the communication address information transmitted by the monitoring end and further judges whether the communication address information is a local gateway address, when the communication address information is the local gateway address, service quality data transmitted by other gateways in the distributed object storage cluster are received, and the summary statistics is carried out by combining the local service quality data; and when the communication address information is not the local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information so as to perform summary statistics through the target object storage gateway. In the computer readable storage medium, each object storage gateway directionally sends the service quality data to a main gateway in the object storage gateways for statistics, so that network delay caused by broadcast message storm is avoided, and reliability and accuracy of the service quality data summarization process of the object storage gateways are relatively ensured.
The data statistics method, system, device and storage medium provided by the present application are described in detail above. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A data statistical method is applied to a monitoring end in a distributed object storage cluster, and comprises the following steps:
acquiring gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster;
selecting a main gateway from the object storage gateways according to the gateway attribute information;
and sending the communication address information of the main gateway to each object storage gateway so as to control other gateways except the main gateway to send service quality data to the main gateway and control the main gateway to perform summary statistics on the service quality data of each object storage gateway.
2. The data statistics method of claim 1, wherein the gateway attribute information comprises the communication address information and a process number;
the selecting a main gateway from the object storage gateways according to the gateway attribute information includes:
and selecting the target object storage gateway with the minimum process number value from the object storage gateways, and setting the target object storage gateway as the main gateway.
3. The data statistics method of claim 2, wherein the communication address information includes IP address information and port number information.
4. The data statistics method as claimed in any one of claims 1 to 3, wherein the QoS data comprises one or more of processed request number, uploaded traffic, downloaded traffic, and traffic data generation amount per unit time.
5. A data statistical method is applied to an object storage gateway in a distributed object storage cluster, and comprises the following steps:
sending gateway attribute information to a monitoring end in the distributed object storage cluster;
receiving communication address information transmitted by the monitoring terminal;
judging whether the communication address information is a local gateway address;
if the communication address information is a local gateway address, receiving service quality data transmitted by other gateways in the distributed object storage cluster, and performing summary statistics by combining the local service quality data;
and if the communication address information is not a local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information.
6. The data statistics method of claim 5, wherein after the performing the aggregated statistics in conjunction with the local quality of service data, the method further comprises:
and sending the summarized data obtained after the summary statistics to the other gateways.
7. The data statistics method of claim 5 or 6, wherein the sending gateway attribute information to the monitoring end in the distributed object storage cluster comprises:
and sending the gateway attribute information to the monitoring terminal according to a preset frequency.
8. A data statistics system, comprising:
the monitoring terminal is used for acquiring gateway attribute information transmitted by each object storage gateway in the distributed object storage cluster; selecting a main gateway from the object storage gateways according to the gateway attribute information; sending the communication address information of the main gateway to each object storage gateway to control other gateways except the main gateway to send service quality data to the main gateway and control the main gateway to perform summary statistics on the service quality data of each object storage gateway;
the object storage gateway is used for sending gateway attribute information to a monitoring end in the distributed object storage cluster; receiving communication address information transmitted by the monitoring terminal; judging whether the communication address information is a local gateway address; if the communication address information is a local gateway address, receiving service quality data transmitted by other gateways in the distributed object storage cluster, and performing summary statistics by combining the local service quality data; and if the communication address information is not a local gateway address, sending the local service quality data to a target object storage gateway corresponding to the communication address information.
9. A data statistics device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data statistics method applied to the monitoring end in the distributed object storage cluster according to any one of claims 1 to 4 and/or implementing the steps of the data statistics method applied to the object storage gateway in the distributed object storage cluster according to any one of claims 5 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program, which when executed by a processor implements the steps of the data statistics method applied to the monitoring end in a distributed object storage cluster according to any one of claims 1 to 4 and/or implements the steps of the data statistics method applied to the object storage gateway in a distributed object storage cluster according to any one of claims 5 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010442521.8A CN111641698B (en) | 2020-05-22 | 2020-05-22 | Data statistical method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010442521.8A CN111641698B (en) | 2020-05-22 | 2020-05-22 | Data statistical method, system, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111641698A true CN111641698A (en) | 2020-09-08 |
| CN111641698B CN111641698B (en) | 2023-04-18 |
Family
ID=72332881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010442521.8A Active CN111641698B (en) | 2020-05-22 | 2020-05-22 | Data statistical method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111641698B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20160025994A (en) * | 2014-08-28 | 2016-03-09 | 네이버 주식회사 | Cluster management method and data storage system for selecting gateway in distributed storage environment |
| US9559889B1 (en) * | 2012-10-31 | 2017-01-31 | Amazon Technologies, Inc. | Cache population optimization for storage gateways |
| US9639555B1 (en) * | 2013-06-21 | 2017-05-02 | EMC IP Holding Company LLC | Object storage server based gateway to network file system cluster |
| US20180004452A1 (en) * | 2016-06-30 | 2018-01-04 | Intel Corporation | Technologies for providing dynamically managed quality of service in a distributed storage system |
| CN108196788A (en) * | 2017-12-28 | 2018-06-22 | 新华三技术有限公司 | QoS index monitoring method, device, storage medium |
| US10084665B1 (en) * | 2017-07-25 | 2018-09-25 | Cisco Technology, Inc. | Resource selection using quality prediction |
| CN108989439A (en) * | 2018-07-26 | 2018-12-11 | 郑州云海信息技术有限公司 | A kind of distributed storage method for controlling quality of service and device |
| CN109361733A (en) * | 2018-09-14 | 2019-02-19 | 新华三技术有限公司成都分公司 | Method for controlling quality of service and device |
| CN109451029A (en) * | 2018-11-16 | 2019-03-08 | 浪潮电子信息产业股份有限公司 | A kind of data cache method, device, equipment and the medium of distributed objects storage |
| US20190243676A1 (en) * | 2016-09-30 | 2019-08-08 | Hangzhou Hikvision Digital Technology Co., Ltd. | Cloud Storage Based Data Processing Method and System |
| CN110233875A (en) * | 2019-05-10 | 2019-09-13 | 厦门网宿有限公司 | A kind of data processing method, object storage gateway pond and operation system |
| CN110519354A (en) * | 2019-08-16 | 2019-11-29 | 济南浪潮数据技术有限公司 | A kind of distributed objects storage system and its method for processing business and storage medium |
| CN111078651A (en) * | 2019-12-23 | 2020-04-28 | 浪潮云信息技术有限公司 | Method and device for counting usage amount of object storage |
-
2020
- 2020-05-22 CN CN202010442521.8A patent/CN111641698B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9559889B1 (en) * | 2012-10-31 | 2017-01-31 | Amazon Technologies, Inc. | Cache population optimization for storage gateways |
| US9639555B1 (en) * | 2013-06-21 | 2017-05-02 | EMC IP Holding Company LLC | Object storage server based gateway to network file system cluster |
| KR20160025994A (en) * | 2014-08-28 | 2016-03-09 | 네이버 주식회사 | Cluster management method and data storage system for selecting gateway in distributed storage environment |
| US20180004452A1 (en) * | 2016-06-30 | 2018-01-04 | Intel Corporation | Technologies for providing dynamically managed quality of service in a distributed storage system |
| US20190243676A1 (en) * | 2016-09-30 | 2019-08-08 | Hangzhou Hikvision Digital Technology Co., Ltd. | Cloud Storage Based Data Processing Method and System |
| US10084665B1 (en) * | 2017-07-25 | 2018-09-25 | Cisco Technology, Inc. | Resource selection using quality prediction |
| CN108196788A (en) * | 2017-12-28 | 2018-06-22 | 新华三技术有限公司 | QoS index monitoring method, device, storage medium |
| CN108989439A (en) * | 2018-07-26 | 2018-12-11 | 郑州云海信息技术有限公司 | A kind of distributed storage method for controlling quality of service and device |
| CN109361733A (en) * | 2018-09-14 | 2019-02-19 | 新华三技术有限公司成都分公司 | Method for controlling quality of service and device |
| CN109451029A (en) * | 2018-11-16 | 2019-03-08 | 浪潮电子信息产业股份有限公司 | A kind of data cache method, device, equipment and the medium of distributed objects storage |
| CN110233875A (en) * | 2019-05-10 | 2019-09-13 | 厦门网宿有限公司 | A kind of data processing method, object storage gateway pond and operation system |
| CN110519354A (en) * | 2019-08-16 | 2019-11-29 | 济南浪潮数据技术有限公司 | A kind of distributed objects storage system and its method for processing business and storage medium |
| CN111078651A (en) * | 2019-12-23 | 2020-04-28 | 浪潮云信息技术有限公司 | Method and device for counting usage amount of object storage |
Non-Patent Citations (3)
| Title |
|---|
| YUSUKE TANIMURA，HIDETAKA KOIE: "Operation-Level Performance Control in the Object Store for Distributed Storage Systems", 《 2015 IEEE INTERNATIONAL CONFERENCE ON DATA SCIENCE AND DATA INTENSIVE SYSTEMS》 * |
| 丘晖，胡军军，李嫚，胡颖茂: "一种存储网关介绍及提升数据读写性能策略研究", 《广东通信技术》 * |
| 杜振伟: "负载感知的基于对象存储的分布式混合存储系统", 《中国优秀硕士学位论文全文数据库》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111641698B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3528440B1 (en) | Path selection method and system, network acceleration node, and network acceleration system | |
| Reyes-Lecuona et al. | A page-oriented WWW traffic model for wireless system simulations | |
| CN107979592B (en) | Method and device for sending service request message | |
| CN101447895A (en) | Collocation method for synchronizing network management and network element and device thereof | |
| CN107872396B (en) | Communication method with timeout control, processing device, terminal device and server | |
| CN106330742B (en) | Flow control method and network controller | |
| WO2018112877A1 (en) | Path calculating and access request distributing methods, devices and systems | |
| CN102394944B (en) | Method and equipment for correcting IP (Internet Protocol) address in Web access | |
| WO2017148446A1 (en) | Network resource scheduling method, device and system, and network node | |
| CN110445723B (en) | Network data scheduling method and edge node | |
| CN105262836A (en) | Information push method of server and push information reception method of client | |
| Park et al. | Smart base station-assisted partial-flow device-to-device offloading system for video streaming services | |
| KR20100057885A (en) | Method and system for checking automatically connectivity status of an ip link on ip network | |
| EP3952233A1 (en) | Tcp congestion control method, apparatus, terminal, and readable storage medium | |
| CN112838989A (en) | Data stream management method, network equipment and storage medium | |
| WO2023125380A1 (en) | Data management method and corresponding apparatus | |
| CN112087382B (en) | Service routing method and device | |
| CN110380981B (en) | Flow distribution method and equipment | |
| CN111641698B (en) | Data statistical method, system, equipment and storage medium | |
| CN114024917B (en) | Method, device, equipment and storage medium for guaranteeing internet service bandwidth | |
| CN113055193B (en) | Data multicast transmission method, device, equipment and storage medium | |
| CN114489730A (en) | Remote upgrading method, terminal equipment and computer readable storage medium | |
| WO2017165999A1 (en) | Network service implementation method, service controller, and communication system | |
| CN109327398B (en) | Method and device for preventing packet loss | |
| CN113612698A (en) | Data packet sending method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |