WO2021017884A1 - Procédé et appareil de traitement de données, et serveur passerelle - Google Patents

Procédé et appareil de traitement de données, et serveur passerelle Download PDF

Info

Publication number
WO2021017884A1
WO2021017884A1 PCT/CN2020/102520 CN2020102520W WO2021017884A1 WO 2021017884 A1 WO2021017884 A1 WO 2021017884A1 CN 2020102520 W CN2020102520 W CN 2020102520W WO 2021017884 A1 WO2021017884 A1 WO 2021017884A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
processed
time difference
time
preset
Prior art date
Application number
PCT/CN2020/102520
Other languages
English (en)
Chinese (zh)
Inventor
王向前
Original Assignee
北京金山云网络技术有限公司
北京金山云科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京金山云网络技术有限公司, 北京金山云科技有限公司 filed Critical 北京金山云网络技术有限公司
Publication of WO2021017884A1 publication Critical patent/WO2021017884A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H04L67/5682—Policies or rules for updating, deleting or replacing the stored data

Definitions

  • This application relates to the field of data processing technology, and in particular to a data processing method, device and gateway server.
  • monitoring data can be collected in each container cluster, and the monitoring data can be temporarily stored in the gateway server (also called the data receiving gateway), and the central control platform (also called For monitoring data storage processing platform) read monitoring data from the gateway server to analyze the working status of the container cluster.
  • the gateway server also called the data receiving gateway
  • the central control platform also called For monitoring data storage processing platform
  • the purpose of this application is to provide a data processing method, device, and gateway server to ensure the timeliness of monitoring data and at the same time improve the utilization of the gateway storage space.
  • an embodiment of the present application provides a data processing method, which includes: obtaining time information of the data to be processed; time information is used to indicate the time when the data to be processed arrives at the current device; and determining the data to be processed according to the time information Whether the preset conditions are met; if the preset conditions are met, delete the pending data.
  • the embodiments of the present application provide the first possible implementation manner of the first aspect, wherein the step of determining whether the data to be processed meets a preset condition according to the time information includes: calculating the time information and the current time Time difference; if the time difference is greater than or equal to the preset time difference threshold, it is determined that the preset condition is satisfied; if the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.
  • the embodiments of the present application provide a second possible implementation manner of the first aspect, wherein the step of determining whether the data to be processed meets a preset condition according to the time information includes: according to the time information and the data to be processed The cluster status of the corresponding container cluster determines whether the data to be processed meets the preset conditions.
  • the embodiments of the present application provide a third possible implementation manner of the first aspect, wherein the above-mentioned to-be-processed data includes multiple; according to the time information and the corresponding to-be-processed data
  • the step of determining whether the data to be processed meets the preset conditions includes: calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to the preset time difference threshold, and obtaining each The judgment result of the data to be processed; obtain the cluster status of the container cluster corresponding to each data to be processed; for each data to be processed, if the judgment result of the data to be processed is greater than or equal to the preset time difference threshold, and the current to be processed
  • the cluster status of the container cluster corresponding to the data is deleted, and it is determined that the current to-be-processed data meets the preset conditions.
  • the embodiments of the present application provide the fourth possible implementation manner of the first aspect, wherein the above-mentioned to-be-processed data includes multiple; according to the time information and the corresponding to-be-processed data
  • the step of determining whether the data to be processed meets the preset conditions includes: calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to the preset time difference threshold, and obtaining each Judgment result of the data to be processed; determine the data to be processed whose time difference is greater than or equal to the preset time difference threshold as pending data; obtain the cluster status of the container cluster corresponding to each pending data; for each pending data, if the current If the cluster state of the container cluster corresponding to the pending data is deleted, it is determined that the current pending data meets the preset condition.
  • an embodiment of the present application also provides a data processing device, the device includes: an information acquisition module configured to acquire time information of the data to be processed; time information is used to indicate the time when the data to be processed arrives at the current device; a condition determination module , Is set to determine whether the data to be processed meets the preset conditions based on time information; the data deletion module is set to delete the data to be processed if the preset conditions are met.
  • the embodiments of the present application provide the first possible implementation manner of the second aspect, wherein the above-mentioned condition determination module includes: a time difference calculation unit configured to calculate the time difference between the time information and the current time; and a first threshold judgment The unit is configured to determine that the preset condition is satisfied if the time difference is greater than or equal to the preset time difference threshold; if the time difference is less than the preset time difference threshold, then it is determined that the preset condition is not satisfied.
  • the embodiments of the present application provide a second possible implementation manner of the second aspect, wherein the above-mentioned condition determining module includes: a first condition determining unit configured to correspond to the container cluster according to the time information and the data to be processed To determine whether the data to be processed meets the preset conditions.
  • an embodiment of the present application provides a third possible implementation manner of the second aspect, wherein the above-mentioned to-be-processed data includes multiple; the first condition determining unit is configured to: calculate The time difference between the time information of each data to be processed and the current time, to determine whether the time difference is greater than or equal to the preset time difference threshold, to obtain the judgment result of each data to be processed; to obtain the cluster status of the container cluster corresponding to each data to be processed; For each data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, it is determined that the current data to be processed meets the preset conditions .
  • an embodiment of the present application provides a fourth possible implementation manner of the second aspect, wherein the aforementioned data to be processed includes multiple; the aforementioned first condition determination unit is configured as: Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to the preset time difference threshold, and obtain the judgment result of each data to be processed; the judgment result is that the time difference is greater than or equal to the preset time difference threshold
  • the pending data is determined as pending data; the cluster status of the container cluster corresponding to each pending data is obtained; for each pending data, if the cluster status of the container cluster corresponding to the current pending data is deleted, it is determined that the current pending data meets the pre-determined data Set conditions.
  • an embodiment of the present application further provides a gateway server, including a processor and a memory.
  • the memory stores machine executable instructions that can be executed by the processor.
  • the processor executes the machine executable instructions to implement the above data processing method.
  • the embodiments of the present application also provide a machine-readable storage medium.
  • the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions cause the processor to implement the above Data processing method.
  • the above-mentioned data processing method, device and gateway server first obtain the time information of the data to be processed; then, according to the time information, determine whether the data to be processed meets a preset condition; if the preset conditions are met, delete the data to be processed.
  • This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
  • FIG. 1 is a schematic diagram of an application scenario of data processing provided by an embodiment of the application
  • FIG. 3 is a flowchart of another data processing method provided by an embodiment of the application.
  • FIG. 4 is a flowchart of another data processing method provided by an embodiment of the application.
  • FIG. 5 is a flowchart of another data processing method provided by an embodiment of the application.
  • FIG. 6 is a flowchart of another data processing method provided by an embodiment of the application.
  • FIG. 7 is a schematic diagram of a monitoring data receiving process of a multi-container cluster provided by an embodiment of this application.
  • FIG. 8 is a schematic structural diagram of a data processing device provided by an embodiment of this application.
  • Fig. 9 is a schematic structural diagram of a gateway server provided by an embodiment of the application.
  • FIG. 1 shows the application scenario, which includes multiple container clusters, a gateway server that receives data, and a central control platform; among them, the gateway server that receives data can be installed with Pushgateway components, and the central control platform can also be called To monitor the data storage and processing platform, the Prometheus system can be run on the platform.
  • the Prometheus system is an open source monitoring and alarm system that can be used to monitor the running status of the container cluster; the Prometheus system usually obtains data from the target device by pulling, but some special monitoring data may not be able to be pulled
  • the above-mentioned Pushgateway component can be used, and the Pushgateway component is used to receive the monitoring data pushed to it by the target device and to save and record the monitoring data. It can be seen that the target device can push and save the monitoring data to the Pushgateway component, and then the Prometheus system obtains the monitoring data from the Pushgateway component by pulling the data.
  • the data aggregation forwarding agent device in the container cluster can push and save the data collected by various data collectors in the container cluster to the Pushgateway component in the gateway server, and Prometheus in the central control platform will receive data from Pull data from Pushgateway component.
  • the data collector in Figure 1 takes the node monitoring data collector, container monitoring data collector, and Kubernetes (k8s) service monitoring data collector as examples. These three data collectors can collect the node processor utilization of the cluster where they are located. Monitoring data such as rate, Pod operating status, kube-apiserver request delay, etc.
  • Kubernetes is an open source container orchestration engine, through which Kubernetes can automatically deploy, expand, and monitor container clusters.
  • Pod is the basic unit of container cloud scheduling.
  • a Pod can be understood as a set of containers, and a container cluster can be divided into multiple Pods; the above-mentioned container monitoring data collector or The Kubernetes service monitoring data collector can monitor the running status of the Pod through related components.
  • the running status of the Pod can be running, waiting, normal termination, abnormal termination, unknown status, etc.
  • the above-mentioned kube-apiserver is a REST (Representational State Transfer) API (Application Programming Interface) interface used to provide container cluster management.
  • the kube-apiserver can be used for authentication, authorization, and data verification of container clusters. It can also be used for storage operations of the container cluster status, etc., so the request delay of kube-apiserver greatly affects the running status of the container cluster.
  • the above-mentioned Kubernetes service monitoring data collector can monitor kube through related components. -The request of apiserver is delayed.
  • the data summary forwarding agent device in each container cluster collects the data of each data collector in the current cluster, processes and pushes it to the gateway server that receives the data, and the gateway server receives and temporarily stores the data summary forwarding agent device of all container clusters and pushes it over The most recent monitoring data.
  • the Prometheus system in the central control platform regularly obtains the collected monitoring data from the gateway server, and analyzes and judges the status of the overall cluster and each cluster based on the monitoring data; in addition, the central control platform also has both retrieval and display functions, which can be used For the retrieval and display of monitoring data.
  • the gateway server due to the limited storage space of the gateway server, every time the data aggregation forwarding agent device sends a new piece of data to the gateway server, it will overwrite the previous piece of historical data corresponding to the new data, that is, only each piece of historical data is stored in the gateway server. The latest piece of data in the cluster. Based on this feature, if a cluster is deleted, the gateway will retain the last piece of historical data that the cluster uploaded to the gateway. This will cause two problems: First, when the amount of historical data left is large, It will occupy more storage space of the gateway server; second, because the Prometheus system of the central control platform regularly obtains data from the gateway server, as long as there is historical data in the gateway server, it will be taken away.
  • the embodiments of the present application provide a data processing method, device, and gateway server.
  • the technology can be applied to various scenarios that require data to be time-sensitive, such as monitoring data of container clusters, monitoring data of network systems, etc. Data processing scene.
  • the method can be applied to a gateway server, and the method includes:
  • Step S200 Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
  • the above-mentioned data to be processed can be various types of monitoring data, sensor data, etc.; for example, monitoring data collected by data collectors in each container cluster, such as network traffic, central processing unit (CPU) usage rate, and Memory usage, etc.; these monitoring data changes in real time according to the working status and system status of the container cluster.
  • monitoring data collected by data collectors in each container cluster such as network traffic, central processing unit (CPU) usage rate, and Memory usage, etc.
  • CPU central processing unit
  • Memory usage etc.
  • the aforementioned data to be processed can be actively sent to the current device by other devices.
  • the current device is a gateway server
  • the data aggregation forwarding agent device in each container cluster sends the data to be processed to the gateway server; the data to be processed can also be sent through the current device.
  • the device obtains from other devices.
  • the gateway server obtains the data to be processed from the data aggregation forwarding agent device in each container cluster.
  • the current device When the data to be processed reaches the current device, the current device usually adds time information to the data to be processed when storing the data to be processed, and the time information indicates the time when the data to be processed arrives at the current device.
  • the time information can be carried into the data to be processed and sent to the current device together.
  • the time information can be a string of pre-appointed characters representing the time, which are added at the agreed position corresponding to the data to be processed, such as adding a character representing time at the end of the data to be processed; the time information can also be a more common time Stamp, the time stamp is a credential document formed after encryption. Generally, the timestamp is more reliable than the conventional characters mentioned above.
  • Step S202 Determine whether the data to be processed meets a preset condition according to the above-mentioned time information.
  • the foregoing preset condition is used to indicate whether to delete the to-be-processed data;
  • the preset condition may be a condition associated with time information, for example, the preset condition may be a time difference threshold.
  • the preset condition can be related to the update cycle of the data to be processed; if the update cycle of the data to be processed is 24 hours, the preset condition can be set as follows: the storage time in the current device is greater than 24 hours; The storage time of the processed data in the current device is 25 hours, and the data to be processed meets the above preset conditions.
  • the preset condition can also be set as: the storage time in the current device is greater than n times the update period of the data to be processed, n is greater than or equal to An integer of 1.
  • the storage time of the data to be processed in the current device can be compared with the time difference threshold, and it is determined whether the preset condition is met according to the comparison result.
  • the storage time of the data to be processed in the current device can be obtained by calculating the time information of the data to be processed and the current time.
  • the state of the container cluster corresponding to the data to be processed can also be considered to set preset conditions. For example, if the storage time of the data to be processed in the current device is relatively long, and at the same time, the container cluster corresponding to the data to be processed is deleted, it can be determined that the data to be processed meets the preset condition.
  • Step S204 if the above-mentioned preset conditions are met, delete the data to be processed.
  • the data to be processed can be deleted from the current device to release the storage space for storing the data to be processed.
  • the time information of the data to be processed is obtained; then, according to the time information, it is determined whether the data to be processed meets a preset condition; if the preset conditions are met, the data to be processed is deleted.
  • This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
  • the embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific implementation process of determining whether the data to be processed meets preset conditions based on time information, such as As shown in Figure 3, the method includes the following steps:
  • Step S300 Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
  • Step S302 Calculate the time difference between the above-mentioned time information and the current time.
  • the current time can be obtained directly from the current device or the Internet; for example, if the current device is a gateway server, the gateway server is usually provided with an accurate timing device, such as a radio clock or an atomic clock.
  • the time difference is obtained by subtracting the above-mentioned time information from the current time, that is, the time when the data to be processed arrives at the current device; the time difference can also be understood as the storage time of the data to be processed in the current device.
  • Step S304 It is judged whether the above-mentioned time difference is greater than or equal to the preset time difference threshold; if so, step S306 is executed; if not, the end is ended.
  • the above-mentioned time difference threshold may be related to the cycle of reading the data to be processed by the central control platform, or may be related to the cycle of updating the data to be processed.
  • the central control platform reads the monitoring data in the gateway server (that is, the data to be processed above) in a preset first time period; the data aggregation forwarding agent device of each container cluster uses the preset second time period (that is, the data Update period)
  • the monitoring data collected by the data collector is sent to the gateway server, and the second time period corresponding to each container cluster may be different.
  • the above-mentioned first time period may be greater than the longest second time period in the container cluster.
  • the data aggregation forwarding agent device of each container cluster has sent the monitoring data to the gateway server at least once, that is, the data to be processed is at least Updated once.
  • the foregoing first time period may be set according to specific requirements, such as being set to several times the foregoing longest data update period.
  • the above-mentioned time difference threshold may be the same as the time period for the central control platform to read the monitoring data.
  • the monitoring data of each container cluster stored in the gateway server that is, the data to be processed has been updated at least once; when it is determined that the storage time of the pending data of a certain container cluster in the current device is greater than the preset time difference threshold If the data to be processed is not updated, the data to be processed needs to be deleted.
  • the aforementioned time difference threshold may also be the same as the aforementioned longest second time period. If the process of forwarding the monitoring data of each container cluster to the gateway server is normal, within the time difference threshold, the monitoring data of each container cluster stored in the gateway server, that is, the data to be processed, has been updated at least once; When the storage time of the data to be processed in the current device is greater than the preset time difference threshold, it indicates that the data to be processed has not been updated and the data to be processed needs to be deleted.
  • the time difference threshold is set to any length of time between the longest second time period and the first time period, if it can be determined that the data to be processed does not meet the preset conditions, it can be guaranteed that the central control platform is adjacent to each other twice
  • the read monitoring data are all updated monitoring data. If there are other requirements for the time interval for obtaining updated monitoring data, such as the central control platform cannot read the same monitoring data three times in a row, the time difference threshold can be set correspondingly according to the requirements.
  • Step S306 Delete the data to be processed.
  • the time difference between the time information of the data to be processed and the current time is calculated, and the time difference is compared with a preset time difference threshold.
  • the time difference is greater than the time difference threshold, the data to be processed is deleted;
  • the timely updated data to be processed ensures the timeliness of the monitoring data and releases the storage space of the gateway server.
  • the embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on determining whether to delete based on the time information of the data to be processed and the cluster status of the corresponding container cluster
  • the specific implementation process of the data is shown in Figure 4. The method includes the following steps:
  • Step S400 Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
  • Step S402 According to the above-mentioned time information and the cluster status of the container cluster corresponding to the data to be processed, it is determined whether the data to be processed meets a preset condition; if it is, step S404 is executed; if not, it ends.
  • the aforementioned data to be processed may be monitoring data of the corresponding container cluster; the cluster status of the aforementioned container cluster includes creating (Initial), running (Running), upgrading (Updating), deleted (Deleted), etc.
  • the aforementioned preset conditions can be set according to the time information and the cluster status of the corresponding container cluster.
  • the preset condition may be divided into a first preset sub-condition and a second preset sub-condition; wherein, the first preset sub-condition may be determining the storage time of the data to be processed according to the above time information, and the storage time is greater than the preset Set the time difference threshold; the above second preset sub-condition can be set as the cluster status corresponding to the data to be processed is deleted.
  • the data to be processed it is possible to determine whether the storage time of the data to be processed meets the first sub-preset condition, and obtain the first judgment result; to determine whether the cluster state of the container cluster corresponding to the data to be processed meets the second sub-preset condition, Obtain the second judgment result; then according to the first judgment result and the second judgment result, judge whether the data to be processed meets the first preset sub-condition and the second preset sub-condition at the same time, if yes, confirm that the data to be processed meets the preset condition.
  • Step S404 Delete the data to be processed.
  • the time information of the data to be processed is obtained, and the data to be processed is cleared when the time information and the state of the container cluster corresponding to the data to be processed meet the preset conditions; this method is in the process of deleting the data to be processed , While considering whether the data to be processed is updated in time, consider whether the container cluster corresponding to the data to be processed has been deleted, ensuring the timeliness of the data to be processed.
  • This embodiment of the application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; this method focuses on determining the data to be processed based on time information and cluster status when there are multiple data to be processed
  • the specific implementation process of processing whether the data meets the preset conditions is shown in Figure 5.
  • the method includes the following steps:
  • Step S500 Obtain time information of each data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
  • Step S502 Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed.
  • the judgment result may include two kinds, one is that the time difference is greater than or equal to the preset time difference threshold, and the other is that the time difference is less than the preset time difference threshold.
  • Step S504 Obtain the cluster status of the container cluster corresponding to each data to be processed.
  • the relational database of the container cluster management platform stores basic information of each cluster, including the cluster status; you can send a request to the container cluster management platform to inquire about the cluster status of the container cluster corresponding to the data to be processed; The request usually includes the identity of the container cluster in question. After obtaining the cluster state of the container cluster corresponding to the data to be processed, the cluster state can be saved in the storage space set by the current device.
  • Step S506 For each data to be processed, determine whether the judgment result of the current data to be processed is greater than or equal to a preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted; if so, perform step S508; If not, end.
  • the following processing can be performed on each data to be processed in a preset order: first, obtain the judgment result of the current data to be processed from the judgment result above; then obtain the result of the cluster state obtained above The cluster status of the container cluster corresponding to the current data to be processed; if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, you can determine the current The data to be processed meets preset conditions, and the data to be processed needs to be deleted.
  • Step S508 Delete the data to be processed.
  • the time information of the data to be processed and the cluster status of the corresponding container cluster are obtained, and the data to be processed is cleared when the time information and the container cluster status meet the preset conditions; this method is deleting the data to be processed In the process, while considering whether the data to be processed is updated in time, it is also considered whether the container cluster corresponding to the data to be processed has been deleted, which ensures the timeliness of the data to be processed.
  • This embodiment of the application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; this method focuses on determining the data to be processed based on time information and cluster status when there are multiple data to be processed
  • the specific implementation process of processing whether the data meets the preset conditions is shown in Figure 6.
  • the method includes the following steps:
  • Step S600 Obtain time information of each data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
  • Step S602 Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed.
  • Step S604 Determine the to-be-processed data whose time difference is greater than or equal to a preset time difference threshold as a result of the judgment as pending data.
  • Step S606 Obtain the cluster status of the container cluster corresponding to each pending data.
  • the undetermined data is a part of the data selected from the above-mentioned to-be-processed data based on the judgment in step S602. Generally, the amount of the undetermined data is not greater than the to-be-processed data.
  • a request can be sent to the container cluster management platform to query the cluster status corresponding to the cluster status of the container cluster corresponding to each pending data, and receive the information returned by the platform to obtain the cluster status of the container cluster corresponding to each pending data.
  • Step S608 For each pending data, it is determined whether the cluster state of the container cluster corresponding to the current pending data is deleted; if so, step 610 is executed; if not, the end is ended.
  • Step S610 Delete the data to be processed.
  • the cluster status of the corresponding container cluster is acquired, and the data to be processed is deleted after the cluster status is determined to be deleted; this method only acquires The cluster status of the container cluster corresponding to the pending data that is not updated in time is used to delete pending data that is not updated in time and the cluster status is deleted.
  • the number of queries to the training state is reduced. , It saves computing resources and is more efficient.
  • the embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the foregoing embodiment; the method is applied to the monitoring data reception and adaptation process of a multi-container cluster.
  • FIG. 7 A schematic diagram of the monitoring data receiving process of a multi-container cluster is shown in Figure 7.
  • the data aggregation forwarding agent device also called a collection agent
  • obtains the monitoring data of the container cluster from the data collector also called the collection object
  • the monitoring data is sent to the gateway server;
  • the gateway server receives the monitoring data, stores the monitoring data in the data queue, and then stores the data in the memory of the gateway server; after receiving the monitoring data belonging to a container cluster, the gateway server
  • the historical monitoring data of the container cluster in the memory is updated to the current monitoring data;
  • the central control platform also referred to as the monitoring platform
  • the data processing method provided in the above embodiment Mainly applied to the gateway server, the data cleaning function of the gateway server memory is realized between the data storage and data retrieval steps.
  • the method includes the following steps:
  • Step 1 Obtain the time stamp of each stored monitoring data; where the time stamp is the time when the gateway server receives the data.
  • Step 2 Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
  • Step 3 Compare the time difference with the preset threshold.
  • Step 4 Determine whether each piece of monitoring data is invalid data based on the comparison result, that is, if the time difference is greater than a preset threshold, then determine that the monitoring data is invalid data.
  • Step 5 Clear invalid data.
  • the above method can be used to process some invalid data regularly.
  • some monitoring data may be deleted as invalid data by mistake; for example, when the connection between the container cluster and the gateway server is interrupted due to some problems, the data stored in the gateway and the cluster The corresponding monitoring data cannot be updated.
  • the monitoring data will be determined as invalid data and deleted.
  • the container cluster corresponding to the monitoring data is not deleted.
  • the monitoring data corresponding to the container cluster can continue to be updated. Therefore, after considering the cluster status of the container cluster corresponding to the monitoring data, after improving the method, the method can be implemented through the following steps:
  • Step 1 Obtain the timestamp of each monitored data.
  • Step 2 Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
  • Step 3 Compare the time difference with the preset threshold to obtain the comparison result.
  • Step 4 Obtain the cluster status of the container cluster corresponding to each monitoring data.
  • Step 5 Determine whether each piece of data is invalid data based on the comparison result and the cluster status of the container cluster corresponding to each monitoring data; specifically, if the comparison result time difference is greater than a preset threshold, and the monitoring data corresponds to the container If the cluster status of the cluster is deleted, the data is determined to be invalid data.
  • Step 6 Clear invalid data.
  • the amount of data for obtaining the status of the cluster can be reduced through the judgment of the timestamp, and then the corresponding cluster status can be queried; at this time, the method can be implemented through the following steps:
  • Step 1 Obtain the timestamp of each monitored data.
  • Step 2 Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
  • Step 3 The time difference is compared with the preset threshold to obtain the comparison result; if the comparison result is that the time difference is greater than the preset threshold, the monitoring data is determined to be invalid data to be determined.
  • Step 4 Obtain the cluster status of the container cluster corresponding to each pending invalid data.
  • Step 5 Determine whether the pending invalid data is invalid data based on the cluster status of the container cluster corresponding to each piece of invalid data to be determined; if the cluster status is a deleted state, determine whether the invalid data to be determined is invalid data.
  • Step 6 Clear invalid data.
  • the gateway server According to the time difference between the time stamp of the monitoring data and the current time, and the cluster status of the container cluster corresponding to the monitoring data, determine whether the monitoring data is invalid data, delete the invalid data, and release the storage space of the gateway server. Improved the utilization of gateway storage space.
  • an embodiment of the present application provides a data processing device. As shown in FIG. 8, the device includes:
  • the information acquisition module 800 is configured to acquire time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device;
  • the condition determining module 802 is configured to determine whether the data to be processed meets a preset condition according to the time information
  • the data deletion module 804 is configured to delete the data to be processed if the preset conditions are met.
  • the above-mentioned data processing device first obtains time information of the data to be processed; and then determines whether the data to be processed meets a preset condition according to the time information; if the preset conditions are met, the data to be processed is deleted. This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
  • the above-mentioned condition determination module includes: a time difference calculation unit configured to calculate the time difference between the time information and the current time; a first threshold judgment unit configured to: if the time difference is greater than or equal to a preset time difference threshold, then It is determined that the preset condition is satisfied; if the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.
  • the above-mentioned condition determination module includes: a first condition determination unit configured to determine whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed.
  • the first condition determining unit is configured to: calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain each The judgment result of the processed data; obtain the cluster status of the container cluster corresponding to each data to be processed; for each data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the current data to be processed The cluster status of the corresponding container cluster is deleted, and it is determined that the current to-be-processed data meets the preset conditions.
  • the first condition determining unit is configured to: calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain each The judgment result of the processed data; the judgment result is that the time difference is greater than or equal to the preset time difference threshold to determine the pending data as pending data; the cluster status of the container cluster corresponding to each pending data is obtained; for each pending data, if the current pending If the cluster status of the container cluster corresponding to the data is deleted, it is determined that the current pending data meets the preset condition.
  • An embodiment of the present application also provides a gateway server.
  • the gateway server includes a processor 130 and a memory 131.
  • the memory 131 stores machine executable instructions that can be executed by the processor 130.
  • the machine executable instructions are executed to implement the above-mentioned data processing method.
  • the gateway server shown in FIG. 9 further includes a bus 132 and a communication interface 133, and the processor 130, the communication interface 133, and the memory 131 are connected through the bus 132.
  • the memory 131 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
  • RAM Random Access Memory
  • non-volatile memory such as at least one disk memory.
  • the communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. may be used.
  • the bus 132 may be an ISA bus, PCI bus, EISA bus, or the like. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one bidirectional arrow is used in FIG. 9, but it does not mean that there is only one bus or one type of bus.
  • the processor 130 may be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the foregoing method may be completed by an integrated logic circuit of hardware in the processor 130 or instructions in the form of software.
  • the aforementioned processor 130 may be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processing, DSP for short). ), Application Specific Integrated Circuit (ASIC for short), Field-Programmable Gate Array (FPGA for short) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • CPU central processing unit
  • NP Network Processor
  • DSP Digital Signal Processing
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131, and completes the steps of the method of the foregoing embodiment in combination with its hardware.
  • the embodiment of the present application also provides a machine-readable storage medium that stores machine-executable instructions.
  • the machine-executable instructions When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor.
  • the above-mentioned data processing method is implemented. For specific implementation, please refer to the method embodiment, which will not be repeated here.
  • the data processing method and device and the computer program product of the gateway server provided by the embodiments of the present application include a computer-readable storage medium storing program code.
  • the instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For the specific implementation of the method, please refer to the method embodiment, which will not be repeated here.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the computer software product is stored in a storage medium, including several
  • the instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Telephonic Communication Services (AREA)
  • Computer And Data Communications (AREA)

Abstract

L'invention concerne un procédé et un appareil de traitement de données, et un serveur passerelle. Le procédé consiste à: obtenir des informations temporelles de données à traiter, les informations temporelles étant utilisées pour indiquer le moment où lesdites données arrivent au niveau du dispositif courant; déterminer si lesdites données satisfont ou non une condition prédéfinie en fonction des informations temporelles; et si la condition prédéfinie est satisfaite, à supprimer lesdites données. Selon la présente invention, les données invalides sont effacées au moyen des informations temporelles, le facteur telps des données de surveillance est garantie, et le taux d'utilisation d'un espace de stockage de passerelle est augmenté en même temps.
PCT/CN2020/102520 2019-07-31 2020-07-17 Procédé et appareil de traitement de données, et serveur passerelle WO2021017884A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910706918.0 2019-07-31
CN201910706918.0A CN110417901B (zh) 2019-07-31 2019-07-31 数据处理方法、装置及网关服务器

Publications (1)

Publication Number Publication Date
WO2021017884A1 true WO2021017884A1 (fr) 2021-02-04

Family

ID=68365267

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/102520 WO2021017884A1 (fr) 2019-07-31 2020-07-17 Procédé et appareil de traitement de données, et serveur passerelle

Country Status (2)

Country Link
CN (1) CN110417901B (fr)
WO (1) WO2021017884A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113485896A (zh) * 2021-07-22 2021-10-08 京东方科技集团股份有限公司 容器状态的监控方法、装置、系统和介质
CN113504995A (zh) * 2021-07-27 2021-10-15 平安普惠企业管理有限公司 批量数据处理方法、装置、计算机设备及存储介质

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110417901B (zh) * 2019-07-31 2022-04-29 北京金山云网络技术有限公司 数据处理方法、装置及网关服务器
CN110856040B (zh) * 2019-11-07 2021-10-26 网易（杭州）网络有限公司 客户端中监控数据的处理方法和装置
CN111064781A (zh) * 2019-12-10 2020-04-24 北京金山云网络技术有限公司 多容器集群监控数据的采集方法、装置及电子设备
CN112511339B (zh) * 2020-11-09 2023-04-07 宝付网络科技（上海）有限公司 基于多集群的容器监控告警方法、系统、设备及存储介质
CN113010797B (zh) * 2021-04-15 2022-04-12 贵州华泰智远大数据服务有限公司 一种基于云平台的智慧城市数据分享方法及其系统
CN113792058B (zh) * 2021-08-09 2023-10-24 北京达佳互联信息技术有限公司 指标数据处理方法、装置、电子设备及存储介质
CN114844794B (zh) * 2022-03-25 2024-06-04 之江实验室 一种面向容器的资源监控方法、系统及存储介质
CN115080617B (zh) * 2022-06-14 2024-04-12 北京智慧星光信息技术有限公司 一种基于网关流量控制的Elasticsearch查询方法及系统
CN118041701A (zh) * 2024-04-12 2024-05-14 华信咨询设计研究院有限公司 一种基于容器云微服务访问流量定向采集预警方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107172171A (zh) * 2017-05-27 2017-09-15 腾讯科技（深圳）有限公司 一种服务请求处理方法、装置及计算机可读存储介质
CN108287890A (zh) * 2018-01-17 2018-07-17 中国科学院寒区旱区环境与工程研究所 数据管理方法及装置
US20180218058A1 (en) * 2015-09-24 2018-08-02 Alibaba Group Holding Limited Data synchronization method and system
CN109213604A (zh) * 2018-10-15 2019-01-15 北京奇艺世纪科技有限公司 一种数据源的管理方法和装置
CN109521957A (zh) * 2018-10-22 2019-03-26 郑州云海信息技术有限公司 一种数据处理方法和装置
CN110417901A (zh) * 2019-07-31 2019-11-05 北京金山云网络技术有限公司 数据处理方法、装置及网关服务器

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008038271A1 (fr) * 2006-09-28 2008-04-03 Xeround Systems Ltd. Appareil et procédé pour une base de données globale de stockage distribué
CN101442443B (zh) * 2009-01-13 2011-01-12 武汉烽火网络有限责任公司 Snmp穿越nat的网络管理系统及实现方法
CN102790809B (zh) * 2011-05-16 2016-09-28 北京奇虎科技有限公司 域名系统解析方法、装置及客户端
CN102752288B (zh) * 2012-06-06 2015-07-08 华为技术有限公司 网络访问行为识别方法和装置
CN105991451B (zh) * 2015-03-09 2018-12-25 杭州迪普科技股份有限公司 基于ospf的lsa删除方法及装置
CN106657249B (zh) * 2016-10-25 2019-07-09 杭州迪普科技股份有限公司 更新缓存资源的方法及装置
CN109343965A (zh) * 2018-10-31 2019-02-15 北京金山云网络技术有限公司 资源调整方法、装置、云平台和服务器
CN110011909A (zh) * 2019-02-28 2019-07-12 北京龙腾融智信息技术有限公司 存储网关及存储网关数据发送、接收方法和装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180218058A1 (en) * 2015-09-24 2018-08-02 Alibaba Group Holding Limited Data synchronization method and system
CN107172171A (zh) * 2017-05-27 2017-09-15 腾讯科技（深圳）有限公司 一种服务请求处理方法、装置及计算机可读存储介质
CN108287890A (zh) * 2018-01-17 2018-07-17 中国科学院寒区旱区环境与工程研究所 数据管理方法及装置
CN109213604A (zh) * 2018-10-15 2019-01-15 北京奇艺世纪科技有限公司 一种数据源的管理方法和装置
CN109521957A (zh) * 2018-10-22 2019-03-26 郑州云海信息技术有限公司 一种数据处理方法和装置
CN110417901A (zh) * 2019-07-31 2019-11-05 北京金山云网络技术有限公司 数据处理方法、装置及网关服务器

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113485896A (zh) * 2021-07-22 2021-10-08 京东方科技集团股份有限公司 容器状态的监控方法、装置、系统和介质
CN113504995A (zh) * 2021-07-27 2021-10-15 平安普惠企业管理有限公司 批量数据处理方法、装置、计算机设备及存储介质

Also Published As

Publication number Publication date
CN110417901B (zh) 2022-04-29
CN110417901A (zh) 2019-11-05

Similar Documents

Publication Publication Date Title
WO2021017884A1 (fr) Procédé et appareil de traitement de données, et serveur passerelle
WO2018032936A1 (fr) Procédé et dispositif de vérification de nom de domaine généré par un algorithme de génération de domaine
WO2015135355A1 (fr) Procédé, appareil et dispositif pour prédire un déplacement de bus
US11778055B2 (en) Systems and methods for storage-efficient sensors
US20150106342A1 (en) System and method of detecting cache inconsistencies
CN109558065B (zh) 数据删除方法及分布式存储系统
CN109271363B (zh) 一种文件存储的方法及设备
CN106921665B (zh) 一种报文处理方法及网络设备
WO2020143181A1 (fr) Procédé de mémorisation de données, appareil, dispositif informatique et support d'informations
CN111585837B (zh) 物联网数据链路监控方法、装置、计算机设备和存储介质
WO2019127895A1 (fr) Serveur, procédé de guidage de flux de données et système
CN110147470B (zh) 一种跨机房数据比对系统及方法
WO2017152765A1 (fr) Procédé, dispositif et système d'envoi de message basé sur une fusion automatique
CN113225339B (zh) 网络安全监测方法、装置、计算机设备及存储介质
US11347646B2 (en) Method, device and computer program product for managing cache based on matching API
CN111078651A (zh) 统计对象存储的使用量的方法及装置
US9641600B2 (en) System, method, computer-readable medium and apparatus
CN113656178B (zh) 数据处理方法、装置、设备及可读存储介质
WO2020083023A1 (fr) Procédé de traitement de flux d'événements, dispositif électronique et support de stockage lisible
CN114363091A (zh) 一种基于apisix实现平台应用统一登录的方法及系统
WO2020000742A1 (fr) Procédé, dispositif, serveur et support de stockage d'enregistrement de trafic de déduplication
US20240022507A1 (en) Information flow recognition method, network chip, and network device
CN105893150B (zh) 接口调用频度控制、接口调用请求处理方法及装置
CN106599005B (zh) 一种数据归档方法及装置
US20200097341A1 (en) Adaptive event aggregation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20847033

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20847033

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20847033

Country of ref document: EP

Kind code of ref document: A1