WO2021017884A1

WO2021017884A1 - Data processing method and apparatus, and gateway server

Info

Publication number: WO2021017884A1
Application number: PCT/CN2020/102520
Authority: WO
Inventors: 王向前
Original assignee: 北京金山云网络技术有限公司; 北京金山云科技有限公司
Priority date: 2019-07-31
Filing date: 2020-07-17
Publication date: 2021-02-04
Also published as: CN110417901B; CN110417901A

Abstract

The present application provides a data processing method and apparatus, and a gateway server. The method comprises: obtaining time information of data to be processed, the time information being used for indicating the time when said data arrives at the current device; determining whether said data satisfies a preset condition according to the time information; and if the preset condition is satisfied, deleting said data. According to the present application, the invalid data is cleared by means of the time information, the timeliness of monitoring data is guaranteed, and the utilization rate of a gateway storage space is increased at the same time.

Description

Data processing method, device and gateway server

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201910706918.0, and the application name is "data processing method, device and gateway server" on July 31, 2019, the entire content of which is incorporated into this application by reference in.

Technical field

This application relates to the field of data processing technology, and in particular to a data processing method, device and gateway server.

Background technique

In related technologies, in order to monitor the working status of the container clusters, monitoring data can be collected in each container cluster, and the monitoring data can be temporarily stored in the gateway server (also called the data receiving gateway), and the central control platform (also called For monitoring data storage processing platform) read monitoring data from the gateway server to analyze the working status of the container cluster. Limited by the storage space of the gateway server, historical monitoring data will be overwritten by new monitoring data; if a container cluster is deleted, the monitoring data about the container cluster will no longer be updated; and the gateway server still saves the historical monitoring of the container cluster Data not only wastes the storage space of the gateway server, but also is not conducive to subsequent analysis and judgment of the overall cluster status based on monitoring data.

Summary of the invention

The purpose of this application is to provide a data processing method, device, and gateway server to ensure the timeliness of monitoring data and at the same time improve the utilization of the gateway storage space.

In the first aspect, an embodiment of the present application provides a data processing method, which includes: obtaining time information of the data to be processed; time information is used to indicate the time when the data to be processed arrives at the current device; and determining the data to be processed according to the time information Whether the preset conditions are met; if the preset conditions are met, delete the pending data.

With reference to the first aspect, the embodiments of the present application provide the first possible implementation manner of the first aspect, wherein the step of determining whether the data to be processed meets a preset condition according to the time information includes: calculating the time information and the current time Time difference; if the time difference is greater than or equal to the preset time difference threshold, it is determined that the preset condition is satisfied; if the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.

With reference to the first aspect, the embodiments of the present application provide a second possible implementation manner of the first aspect, wherein the step of determining whether the data to be processed meets a preset condition according to the time information includes: according to the time information and the data to be processed The cluster status of the corresponding container cluster determines whether the data to be processed meets the preset conditions.

In combination with the second possible implementation manner of the first aspect, the embodiments of the present application provide a third possible implementation manner of the first aspect, wherein the above-mentioned to-be-processed data includes multiple; according to the time information and the corresponding to-be-processed data For the cluster status of the container cluster, the step of determining whether the data to be processed meets the preset conditions includes: calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to the preset time difference threshold, and obtaining each The judgment result of the data to be processed; obtain the cluster status of the container cluster corresponding to each data to be processed; for each data to be processed, if the judgment result of the data to be processed is greater than or equal to the preset time difference threshold, and the current to be processed The cluster status of the container cluster corresponding to the data is deleted, and it is determined that the current to-be-processed data meets the preset conditions.

In combination with the second possible implementation manner of the first aspect, the embodiments of the present application provide the fourth possible implementation manner of the first aspect, wherein the above-mentioned to-be-processed data includes multiple; according to the time information and the corresponding to-be-processed data For the cluster status of the container cluster, the step of determining whether the data to be processed meets the preset conditions includes: calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to the preset time difference threshold, and obtaining each Judgment result of the data to be processed; determine the data to be processed whose time difference is greater than or equal to the preset time difference threshold as pending data; obtain the cluster status of the container cluster corresponding to each pending data; for each pending data, if the current If the cluster state of the container cluster corresponding to the pending data is deleted, it is determined that the current pending data meets the preset condition.

In a second aspect, an embodiment of the present application also provides a data processing device, the device includes: an information acquisition module configured to acquire time information of the data to be processed; time information is used to indicate the time when the data to be processed arrives at the current device; a condition determination module , Is set to determine whether the data to be processed meets the preset conditions based on time information; the data deletion module is set to delete the data to be processed if the preset conditions are met.

With reference to the second aspect, the embodiments of the present application provide the first possible implementation manner of the second aspect, wherein the above-mentioned condition determination module includes: a time difference calculation unit configured to calculate the time difference between the time information and the current time; and a first threshold judgment The unit is configured to determine that the preset condition is satisfied if the time difference is greater than or equal to the preset time difference threshold; if the time difference is less than the preset time difference threshold, then it is determined that the preset condition is not satisfied.

With reference to the second aspect, the embodiments of the present application provide a second possible implementation manner of the second aspect, wherein the above-mentioned condition determining module includes: a first condition determining unit configured to correspond to the container cluster according to the time information and the data to be processed To determine whether the data to be processed meets the preset conditions.

In combination with the second possible implementation manner of the second aspect, an embodiment of the present application provides a third possible implementation manner of the second aspect, wherein the above-mentioned to-be-processed data includes multiple; the first condition determining unit is configured to: calculate The time difference between the time information of each data to be processed and the current time, to determine whether the time difference is greater than or equal to the preset time difference threshold, to obtain the judgment result of each data to be processed; to obtain the cluster status of the container cluster corresponding to each data to be processed; For each data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, it is determined that the current data to be processed meets the preset conditions .

In combination with the second possible implementation manner of the second aspect, an embodiment of the present application provides a fourth possible implementation manner of the second aspect, wherein the aforementioned data to be processed includes multiple; the aforementioned first condition determination unit is configured as: Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to the preset time difference threshold, and obtain the judgment result of each data to be processed; the judgment result is that the time difference is greater than or equal to the preset time difference threshold The pending data is determined as pending data; the cluster status of the container cluster corresponding to each pending data is obtained; for each pending data, if the cluster status of the container cluster corresponding to the current pending data is deleted, it is determined that the current pending data meets the pre-determined data Set conditions.

In a third aspect, an embodiment of the present application further provides a gateway server, including a processor and a memory. The memory stores machine executable instructions that can be executed by the processor. The processor executes the machine executable instructions to implement the above data processing method.

In a fourth aspect, the embodiments of the present application also provide a machine-readable storage medium. The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions cause the processor to implement the above Data processing method.

The above-mentioned data processing method, device and gateway server first obtain the time information of the data to be processed; then, according to the time information, determine whether the data to be processed meets a preset condition; if the preset conditions are met, delete the data to be processed. This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.

Description of the drawings

In order to more clearly illustrate the specific embodiments of the application or the technical solutions in related technologies, the following will briefly introduce the drawings that need to be used in the specific embodiments or related technical descriptions. Obviously, the drawings in the following description are For some of the embodiments of the present application, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic diagram of an application scenario of data processing provided by an embodiment of the application;

2 is a flowchart of a data processing method provided by an embodiment of the application;

FIG. 3 is a flowchart of another data processing method provided by an embodiment of the application;

FIG. 4 is a flowchart of another data processing method provided by an embodiment of the application;

FIG. 5 is a flowchart of another data processing method provided by an embodiment of the application;

FIG. 6 is a flowchart of another data processing method provided by an embodiment of the application;

FIG. 7 is a schematic diagram of a monitoring data receiving process of a multi-container cluster provided by an embodiment of this application;

FIG. 8 is a schematic structural diagram of a data processing device provided by an embodiment of this application;

Fig. 9 is a schematic structural diagram of a gateway server provided by an embodiment of the application.

Detailed ways

The technical solutions of the present application will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

In related technologies, multiple data collectors can be set up in each container cluster to collect various monitoring data of the cluster, and at the same time, a data summary forwarding agent device connected to each collector can be set to collect various data collectors. The collected data is pushed or forwarded to the gateway server that receives the data. For ease of understanding, Figure 1 shows the application scenario, which includes multiple container clusters, a gateway server that receives data, and a central control platform; among them, the gateway server that receives data can be installed with Pushgateway components, and the central control platform can also be called To monitor the data storage and processing platform, the Prometheus system can be run on the platform.

The Prometheus system is an open source monitoring and alarm system that can be used to monitor the running status of the container cluster; the Prometheus system usually obtains data from the target device by pulling, but some special monitoring data may not be able to be pulled In this case, the above-mentioned Pushgateway component can be used, and the Pushgateway component is used to receive the monitoring data pushed to it by the target device and to save and record the monitoring data. It can be seen that the target device can push and save the monitoring data to the Pushgateway component, and then the Prometheus system obtains the monitoring data from the Pushgateway component by pulling the data. For example, the data aggregation forwarding agent device in the container cluster can push and save the data collected by various data collectors in the container cluster to the Pushgateway component in the gateway server, and Prometheus in the central control platform will receive data from Pull data from Pushgateway component.

The data collector in Figure 1 takes the node monitoring data collector, container monitoring data collector, and Kubernetes (k8s) service monitoring data collector as examples. These three data collectors can collect the node processor utilization of the cluster where they are located. Monitoring data such as rate, Pod operating status, kube-apiserver request delay, etc. Among them, Kubernetes is an open source container orchestration engine, through which Kubernetes can automatically deploy, expand, and monitor container clusters. There are a variety of components in Kubernetes to monitor the running status of the container cluster, such as Probes, cAdvisor, etc. These components can be set in the container monitoring data collector or the Kubernetes service monitoring data collector to collect part of the container cluster Monitoring data.

Generally, all containers in a container cluster run in Pods. Pod is the basic unit of container cloud scheduling. A Pod can be understood as a set of containers, and a container cluster can be divided into multiple Pods; the above-mentioned container monitoring data collector or The Kubernetes service monitoring data collector can monitor the running status of the Pod through related components. The running status of the Pod can be running, waiting, normal termination, abnormal termination, unknown status, etc.

The above-mentioned kube-apiserver is a REST (Representational State Transfer) API (Application Programming Interface) interface used to provide container cluster management. The kube-apiserver can be used for authentication, authorization, and data verification of container clusters. It can also be used for storage operations of the container cluster status, etc., so the request delay of kube-apiserver greatly affects the running status of the container cluster. The above-mentioned Kubernetes service monitoring data collector can monitor kube through related components. -The request of apiserver is delayed.

The data summary forwarding agent device in each container cluster collects the data of each data collector in the current cluster, processes and pushes it to the gateway server that receives the data, and the gateway server receives and temporarily stores the data summary forwarding agent device of all container clusters and pushes it over The most recent monitoring data. The Prometheus system in the central control platform regularly obtains the collected monitoring data from the gateway server, and analyzes and judges the status of the overall cluster and each cluster based on the monitoring data; in addition, the central control platform also has both retrieval and display functions, which can be used For the retrieval and display of monitoring data.

In the specific work process, due to the limited storage space of the gateway server, every time the data aggregation forwarding agent device sends a new piece of data to the gateway server, it will overwrite the previous piece of historical data corresponding to the new data, that is, only each piece of historical data is stored in the gateway server. The latest piece of data in the cluster. Based on this feature, if a cluster is deleted, the gateway will retain the last piece of historical data that the cluster uploaded to the gateway. This will cause two problems: First, when the amount of historical data left is large, It will occupy more storage space of the gateway server; second, because the Prometheus system of the central control platform regularly obtains data from the gateway server, as long as there is historical data in the gateway server, it will be taken away. Therefore, if a cluster If it has been deleted, the Prometheus system will continuously obtain the same historical data. These repeated historical data are invalid samples, which will mislead the Prometheus system's subsequent judgment and analysis, which is not conducive to the analysis and judgment of the overall cluster status based on the monitoring data.

Based on this, the embodiments of the present application provide a data processing method, device, and gateway server. The technology can be applied to various scenarios that require data to be time-sensitive, such as monitoring data of container clusters, monitoring data of network systems, etc. Data processing scene.

First, refer to the flowchart of a data processing method shown in Figure 2; the method can be applied to a gateway server, and the method includes:

Step S200: Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.

The above-mentioned data to be processed can be various types of monitoring data, sensor data, etc.; for example, monitoring data collected by data collectors in each container cluster, such as network traffic, central processing unit (CPU) usage rate, and Memory usage, etc.; these monitoring data changes in real time according to the working status and system status of the container cluster. By analyzing and counting the monitoring data, the status of each container cluster and container cluster system can be analyzed and evaluated. The aforementioned data to be processed can be actively sent to the current device by other devices. For example, if the current device is a gateway server, the data aggregation forwarding agent device in each container cluster sends the data to be processed to the gateway server; the data to be processed can also be sent through the current device. The device obtains from other devices. For example, the gateway server obtains the data to be processed from the data aggregation forwarding agent device in each container cluster.

When the data to be processed reaches the current device, the current device usually adds time information to the data to be processed when storing the data to be processed, and the time information indicates the time when the data to be processed arrives at the current device. In another way, when other devices connected to the current device send data to be processed, the time information can be carried into the data to be processed and sent to the current device together. The time information can be a string of pre-appointed characters representing the time, which are added at the agreed position corresponding to the data to be processed, such as adding a character representing time at the end of the data to be processed; the time information can also be a more common time Stamp, the time stamp is a credential document formed after encryption. Generally, the timestamp is more reliable than the conventional characters mentioned above.

Step S202: Determine whether the data to be processed meets a preset condition according to the above-mentioned time information.

The foregoing preset condition is used to indicate whether to delete the to-be-processed data; the preset condition may be a condition associated with time information, for example, the preset condition may be a time difference threshold. Specifically, the preset condition can be related to the update cycle of the data to be processed; if the update cycle of the data to be processed is 24 hours, the preset condition can be set as follows: the storage time in the current device is greater than 24 hours; The storage time of the processed data in the current device is 25 hours, and the data to be processed meets the above preset conditions. Taking into account that during the process of updating the data to be processed, there may be delayed updates caused by network problems, the preset condition can also be set as: the storage time in the current device is greater than n times the update period of the data to be processed, n is greater than or equal to An integer of 1.

In the process of determining whether the data to be processed meets the preset condition, the storage time of the data to be processed in the current device can be compared with the time difference threshold, and it is determined whether the preset condition is met according to the comparison result. The storage time of the data to be processed in the current device can be obtained by calculating the time information of the data to be processed and the current time.

In addition, on the basis of time information, the state of the container cluster corresponding to the data to be processed can also be considered to set preset conditions. For example, if the storage time of the data to be processed in the current device is relatively long, and at the same time, the container cluster corresponding to the data to be processed is deleted, it can be determined that the data to be processed meets the preset condition.

Step S204, if the above-mentioned preset conditions are met, delete the data to be processed.

When the data to be processed meets the preset condition, the data to be processed can be deleted from the current device to release the storage space for storing the data to be processed.

In the above data processing method, firstly, the time information of the data to be processed is obtained; then, according to the time information, it is determined whether the data to be processed meets a preset condition; if the preset conditions are met, the data to be processed is deleted. This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.

The embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific implementation process of determining whether the data to be processed meets preset conditions based on time information, such as As shown in Figure 3, the method includes the following steps:

Step S300: Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.

Step S302: Calculate the time difference between the above-mentioned time information and the current time.

Generally speaking, the current time can be obtained directly from the current device or the Internet; for example, if the current device is a gateway server, the gateway server is usually provided with an accurate timing device, such as a radio clock or an atomic clock. The time difference is obtained by subtracting the above-mentioned time information from the current time, that is, the time when the data to be processed arrives at the current device; the time difference can also be understood as the storage time of the data to be processed in the current device.

Step S304: It is judged whether the above-mentioned time difference is greater than or equal to the preset time difference threshold; if so, step S306 is executed; if not, the end is ended.

The above-mentioned time difference threshold may be related to the cycle of reading the data to be processed by the central control platform, or may be related to the cycle of updating the data to be processed. As an example, the central control platform reads the monitoring data in the gateway server (that is, the data to be processed above) in a preset first time period; the data aggregation forwarding agent device of each container cluster uses the preset second time period (that is, the data Update period) The monitoring data collected by the data collector is sent to the gateway server, and the second time period corresponding to each container cluster may be different. In order that the read monitoring data are all updated monitoring data, the above-mentioned first time period may be greater than the longest second time period in the container cluster. In this case, between the last time the monitoring data is read and the current time when the monitoring data is read, the data aggregation forwarding agent device of each container cluster has sent the monitoring data to the gateway server at least once, that is, the data to be processed is at least Updated once. The foregoing first time period may be set according to specific requirements, such as being set to several times the foregoing longest data update period.

The above-mentioned time difference threshold may be the same as the time period for the central control platform to read the monitoring data. Within the time difference threshold, the monitoring data of each container cluster stored in the gateway server, that is, the data to be processed has been updated at least once; when it is determined that the storage time of the pending data of a certain container cluster in the current device is greater than the preset time difference threshold If the data to be processed is not updated, the data to be processed needs to be deleted.

The aforementioned time difference threshold may also be the same as the aforementioned longest second time period. If the process of forwarding the monitoring data of each container cluster to the gateway server is normal, within the time difference threshold, the monitoring data of each container cluster stored in the gateway server, that is, the data to be processed, has been updated at least once; When the storage time of the data to be processed in the current device is greater than the preset time difference threshold, it indicates that the data to be processed has not been updated and the data to be processed needs to be deleted.

When the above time difference threshold is set to any length of time between the longest second time period and the first time period, if it can be determined that the data to be processed does not meet the preset conditions, it can be guaranteed that the central control platform is adjacent to each other twice The read monitoring data are all updated monitoring data. If there are other requirements for the time interval for obtaining updated monitoring data, such as the central control platform cannot read the same monitoring data three times in a row, the time difference threshold can be set correspondingly according to the requirements.

Step S306: Delete the data to be processed.

In the above data processing method, the time difference between the time information of the data to be processed and the current time is calculated, and the time difference is compared with a preset time difference threshold. When the time difference is greater than the time difference threshold, the data to be processed is deleted; The timely updated data to be processed ensures the timeliness of the monitoring data and releases the storage space of the gateway server.

The embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on determining whether to delete based on the time information of the data to be processed and the cluster status of the corresponding container cluster The specific implementation process of the data is shown in Figure 4. The method includes the following steps:

Step S400: Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.

Step S402: According to the above-mentioned time information and the cluster status of the container cluster corresponding to the data to be processed, it is determined whether the data to be processed meets a preset condition; if it is, step S404 is executed; if not, it ends.

The aforementioned data to be processed may be monitoring data of the corresponding container cluster; the cluster status of the aforementioned container cluster includes creating (Initial), running (Running), upgrading (Updating), deleted (Deleted), etc. The aforementioned preset conditions can be set according to the time information and the cluster status of the corresponding container cluster. For example, the preset condition may be divided into a first preset sub-condition and a second preset sub-condition; wherein, the first preset sub-condition may be determining the storage time of the data to be processed according to the above time information, and the storage time is greater than the preset Set the time difference threshold; the above second preset sub-condition can be set as the cluster status corresponding to the data to be processed is deleted.

For the data to be processed, it is possible to determine whether the storage time of the data to be processed meets the first sub-preset condition, and obtain the first judgment result; to determine whether the cluster state of the container cluster corresponding to the data to be processed meets the second sub-preset condition, Obtain the second judgment result; then according to the first judgment result and the second judgment result, judge whether the data to be processed meets the first preset sub-condition and the second preset sub-condition at the same time, if yes, confirm that the data to be processed meets the preset condition.

In another way, it is also possible to first determine whether the storage time of the data to be processed meets the first preset sub-condition, and if the first preset sub-condition is satisfied, then determine whether the cluster state of the container cluster corresponding to the data to be processed is The second preset sub-condition is satisfied, and if the second preset sub-condition is also satisfied, it is confirmed that the data to be processed meets the preset condition. Similarly, it is also possible to first determine whether the second preset sub-condition is met, and if the condition is met, then the first preset sub-condition is judged; if the first preset sub-condition is also met, the data to be processed is confirmed Meet the preset conditions.

Step S404: Delete the data to be processed.

In the above data processing method, the time information of the data to be processed is obtained, and the data to be processed is cleared when the time information and the state of the container cluster corresponding to the data to be processed meet the preset conditions; this method is in the process of deleting the data to be processed , While considering whether the data to be processed is updated in time, consider whether the container cluster corresponding to the data to be processed has been deleted, ensuring the timeliness of the data to be processed.

This embodiment of the application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; this method focuses on determining the data to be processed based on time information and cluster status when there are multiple data to be processed The specific implementation process of processing whether the data meets the preset conditions is shown in Figure 5. The method includes the following steps:

Step S500: Obtain time information of each data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.

Step S502: Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed. The judgment result may include two kinds, one is that the time difference is greater than or equal to the preset time difference threshold, and the other is that the time difference is less than the preset time difference threshold.

Step S504: Obtain the cluster status of the container cluster corresponding to each data to be processed.

Normally, the relational database of the container cluster management platform stores basic information of each cluster, including the cluster status; you can send a request to the container cluster management platform to inquire about the cluster status of the container cluster corresponding to the data to be processed; The request usually includes the identity of the container cluster in question. After obtaining the cluster state of the container cluster corresponding to the data to be processed, the cluster state can be saved in the storage space set by the current device.

Step S506: For each data to be processed, determine whether the judgment result of the current data to be processed is greater than or equal to a preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted; if so, perform step S508; If not, end.

Specifically, the following processing can be performed on each data to be processed in a preset order: first, obtain the judgment result of the current data to be processed from the judgment result above; then obtain the result of the cluster state obtained above The cluster status of the container cluster corresponding to the current data to be processed; if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, you can determine the current The data to be processed meets preset conditions, and the data to be processed needs to be deleted.

Step S508: Delete the data to be processed.

In the above data processing method, the time information of the data to be processed and the cluster status of the corresponding container cluster are obtained, and the data to be processed is cleared when the time information and the container cluster status meet the preset conditions; this method is deleting the data to be processed In the process, while considering whether the data to be processed is updated in time, it is also considered whether the container cluster corresponding to the data to be processed has been deleted, which ensures the timeliness of the data to be processed.

This embodiment of the application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; this method focuses on determining the data to be processed based on time information and cluster status when there are multiple data to be processed The specific implementation process of processing whether the data meets the preset conditions is shown in Figure 6. The method includes the following steps:

Step S600: Obtain time information of each data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.

Step S602: Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed.

Step S604: Determine the to-be-processed data whose time difference is greater than or equal to a preset time difference threshold as a result of the judgment as pending data.

Step S606: Obtain the cluster status of the container cluster corresponding to each pending data.

The undetermined data is a part of the data selected from the above-mentioned to-be-processed data based on the judgment in step S602. Generally, the amount of the undetermined data is not greater than the to-be-processed data. After the pending data is determined, a request can be sent to the container cluster management platform to query the cluster status corresponding to the cluster status of the container cluster corresponding to each pending data, and receive the information returned by the platform to obtain the cluster status of the container cluster corresponding to each pending data.

Step S608: For each pending data, it is determined whether the cluster state of the container cluster corresponding to the current pending data is deleted; if so, step 610 is executed; if not, the end is ended.

Step S610: Delete the data to be processed.

In the above data processing method, after the data to be processed is determined to be data that has not been updated for a long time, the cluster status of the corresponding container cluster is acquired, and the data to be processed is deleted after the cluster status is determined to be deleted; this method only acquires The cluster status of the container cluster corresponding to the pending data that is not updated in time is used to delete pending data that is not updated in time and the cluster status is deleted. On the basis of ensuring the timeliness of the monitoring data, the number of queries to the training state is reduced. , It saves computing resources and is more efficient.

The embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the foregoing embodiment; the method is applied to the monitoring data reception and adaptation process of a multi-container cluster.

A schematic diagram of the monitoring data receiving process of a multi-container cluster is shown in Figure 7. The data aggregation forwarding agent device (also called a collection agent) obtains the monitoring data of the container cluster from the data collector (also called the collection object), and The monitoring data is sent to the gateway server; the gateway server receives the monitoring data, stores the monitoring data in the data queue, and then stores the data in the memory of the gateway server; after receiving the monitoring data belonging to a container cluster, the gateway server The historical monitoring data of the container cluster in the memory is updated to the current monitoring data; the central control platform (also referred to as the monitoring platform) reads the monitoring data in the gateway server memory after data retrieval; the data processing method provided in the above embodiment Mainly applied to the gateway server, the data cleaning function of the gateway server memory is realized between the data storage and data retrieval steps.

When only considering the storage time of the monitoring data in the gateway server, the method includes the following steps:

Step 1. Obtain the time stamp of each stored monitoring data; where the time stamp is the time when the gateway server receives the data.

Step 2: Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.

Step 3. Compare the time difference with the preset threshold.

Step 4. Determine whether each piece of monitoring data is invalid data based on the comparison result, that is, if the time difference is greater than a preset threshold, then determine that the monitoring data is invalid data.

Step 5. Clear invalid data.

The above method can be used to process some invalid data regularly. However, some monitoring data may be deleted as invalid data by mistake; for example, when the connection between the container cluster and the gateway server is interrupted due to some problems, the data stored in the gateway and the cluster The corresponding monitoring data cannot be updated. In the process of determining invalid data, the monitoring data will be determined as invalid data and deleted. However, the container cluster corresponding to the monitoring data is not deleted. When the network connection between the container cluster and the gateway server is restored, the monitoring data corresponding to the container cluster can continue to be updated. Therefore, after considering the cluster status of the container cluster corresponding to the monitoring data, after improving the method, the method can be implemented through the following steps:

Step 1. Obtain the timestamp of each monitored data.

Step 3. Compare the time difference with the preset threshold to obtain the comparison result.

Step 4. Obtain the cluster status of the container cluster corresponding to each monitoring data.

Step 5. Determine whether each piece of data is invalid data based on the comparison result and the cluster status of the container cluster corresponding to each monitoring data; specifically, if the comparison result time difference is greater than a preset threshold, and the monitoring data corresponds to the container If the cluster status of the cluster is deleted, the data is determined to be invalid data.

Step 6. Clear invalid data.

In a possible implementation manner, in order to save computing resources, the amount of data for obtaining the status of the cluster can be reduced through the judgment of the timestamp, and then the corresponding cluster status can be queried; at this time, the method can be implemented through the following steps:

Step 1. Obtain the timestamp of each monitored data.

Step 3: The time difference is compared with the preset threshold to obtain the comparison result; if the comparison result is that the time difference is greater than the preset threshold, the monitoring data is determined to be invalid data to be determined.

Step 4. Obtain the cluster status of the container cluster corresponding to each pending invalid data.

Step 5: Determine whether the pending invalid data is invalid data based on the cluster status of the container cluster corresponding to each piece of invalid data to be determined; if the cluster status is a deleted state, determine whether the invalid data to be determined is invalid data.

Step 6. Clear invalid data.

In the above method, according to the time difference between the time stamp of the monitoring data and the current time, and the cluster status of the container cluster corresponding to the monitoring data, determine whether the monitoring data is invalid data, delete the invalid data, and release the storage space of the gateway server. Improved the utilization of gateway storage space.

Corresponding to the foregoing data processing method embodiment, an embodiment of the present application provides a data processing device. As shown in FIG. 8, the device includes:

The information acquisition module 800 is configured to acquire time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device;

The condition determining module 802 is configured to determine whether the data to be processed meets a preset condition according to the time information;

The data deletion module 804 is configured to delete the data to be processed if the preset conditions are met.

The above-mentioned data processing device first obtains time information of the data to be processed; and then determines whether the data to be processed meets a preset condition according to the time information; if the preset conditions are met, the data to be processed is deleted. This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.

In a possible implementation manner, the above-mentioned condition determination module includes: a time difference calculation unit configured to calculate the time difference between the time information and the current time; a first threshold judgment unit configured to: if the time difference is greater than or equal to a preset time difference threshold, then It is determined that the preset condition is satisfied; if the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.

In a possible implementation manner, the above-mentioned condition determination module includes: a first condition determination unit configured to determine whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed.

When the data to be processed includes multiple data, the first condition determining unit is configured to: calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain each The judgment result of the processed data; obtain the cluster status of the container cluster corresponding to each data to be processed; for each data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the current data to be processed The cluster status of the corresponding container cluster is deleted, and it is determined that the current to-be-processed data meets the preset conditions.

When the data to be processed includes multiple data, the first condition determining unit is configured to: calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain each The judgment result of the processed data; the judgment result is that the time difference is greater than or equal to the preset time difference threshold to determine the pending data as pending data; the cluster status of the container cluster corresponding to each pending data is obtained; for each pending data, if the current pending If the cluster status of the container cluster corresponding to the data is deleted, it is determined that the current pending data meets the preset condition.

The implementation principles and technical effects of the data processing device provided in the embodiments of this application are the same as those of the foregoing data processing method embodiments. For a brief description, the data processing device embodiments that are not mentioned in the data processing device embodiments may refer to the foregoing data processing method. Corresponding content in the embodiment.

An embodiment of the present application also provides a gateway server. As shown in FIG. 9, the gateway server includes a processor 130 and a memory 131. The memory 131 stores machine executable instructions that can be executed by the processor 130. The machine executable instructions are executed to implement the above-mentioned data processing method.

In a possible implementation manner, the gateway server shown in FIG. 9 further includes a bus 132 and a communication interface 133, and the processor 130, the communication interface 133, and the memory 131 are connected through the bus 132.

The memory 131 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, PCI bus, EISA bus, or the like. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one bidirectional arrow is used in FIG. 9, but it does not mean that there is only one bus or one type of bus.

The processor 130 may be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the foregoing method may be completed by an integrated logic circuit of hardware in the processor 130 or instructions in the form of software. The aforementioned processor 130 may be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processing, DSP for short). ), Application Specific Integrated Circuit (ASIC for short), Field-Programmable Gate Array (FPGA for short) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131, and completes the steps of the method of the foregoing embodiment in combination with its hardware.

The embodiment of the present application also provides a machine-readable storage medium that stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor The above-mentioned data processing method is implemented. For specific implementation, please refer to the method embodiment, which will not be repeated here.

The data processing method and device and the computer program product of the gateway server provided by the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For the specific implementation of the method, please refer to the method embodiment, which will not be repeated here.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of this application essentially or the part that contributes to the related technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several The instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code .

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the application, which are used to illustrate the technical solutions of the application, rather than limit it. The scope of protection of the application is not limited thereto, although referring to the foregoing The examples describe the application in detail, and those of ordinary skill in the art should understand that any person skilled in the art can still modify the technical solutions described in the foregoing examples within the technical scope disclosed in this application. Or it is easy to think of changes, or equivalent replacements of some of the technical features; and these modifications, changes or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be covered in this application Within the scope of protection. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A data processing method, the method includes:

Acquiring time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device;

According to the time information, determine whether the data to be processed meets a preset condition;

If the preset condition is met, delete the to-be-processed data.
The method according to claim 1, wherein the step of determining whether the data to be processed meets a preset condition according to the time information comprises:

Calculating the time difference between the time information and the current time;

If the time difference is greater than or equal to the preset time difference threshold, it is determined that the preset condition is met;

If the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.
The method according to claim 1, wherein the step of determining whether the data to be processed meets a preset condition according to the time information comprises:

According to the time information and the cluster state of the container cluster corresponding to the data to be processed, it is determined whether the data to be processed meets a preset condition.
The method according to claim 3, wherein the data to be processed includes a plurality of;

The step of determining whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed includes:

Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed;

Acquiring the cluster status of the container cluster corresponding to each of the data to be processed;

For each of the data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, determine the current data to be processed The processed data meets preset conditions.
The method according to claim 3, wherein the data to be processed includes a plurality of;

The step of determining whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed includes:

Calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to a preset time difference threshold, and obtaining a judgment result of each data to be processed;

Determine the data to be processed whose time difference is greater than or equal to the preset time difference threshold as the undetermined data;

Acquiring the cluster state of the container cluster corresponding to each of the pending data;

For each of the pending data, if it is determined that the cluster state of the container cluster corresponding to the current pending data is deleted, it is determined that the current pending data meets a preset condition.
A data processing device, the device comprising:

The information acquisition module is configured to acquire time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device;

The condition determining module is configured to determine whether the data to be processed meets preset conditions according to the time information;

The data deletion module is configured to delete the to-be-processed data if the preset conditions are met.
The apparatus according to claim 6, wherein the condition determination module comprises:

A time difference calculation unit, configured to calculate the time difference between the time information and the current time;

The first threshold determination unit is configured to determine that the preset condition is satisfied if the time difference is greater than or equal to the preset time difference threshold; if the time difference is less than the preset time difference threshold, then it is determined that the preset condition is not satisfied.
The device according to claim 6, wherein the condition determining module comprises:

The first condition determining unit is configured to determine whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed.
The device according to claim 8, wherein the data to be processed includes a plurality of;

The first condition determining unit is set to:

Calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to a preset time difference threshold, and obtaining a judgment result of each data to be processed;

Acquiring the cluster status of the container cluster corresponding to each of the data to be processed;

For each of the data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, determine the current data to be processed The processed data meets preset conditions.
The device according to claim 8, wherein the data to be processed includes a plurality of;

The first condition determining unit is set to:

Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed;

Determine the data to be processed whose time difference is greater than or equal to the preset time difference threshold as the undetermined data;

Acquiring the cluster state of the container cluster corresponding to each of the pending data;

For each of the pending data, if the cluster state of the container cluster corresponding to the current pending data is deleted, it is determined that the current pending data meets a preset condition.
A gateway server, comprising a processor and a memory, the memory storing machine executable instructions that can be executed by the processor, and the processor executing the machine executable instructions to implement any one of claims 1 to 5 The data processing method.
A machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement claims 1 to 5 Any of the data processing methods.