WO2021017884A1 - Data processing method and apparatus, and gateway server - Google Patents
Data processing method and apparatus, and gateway server Download PDFInfo
- Publication number
- WO2021017884A1 WO2021017884A1 PCT/CN2020/102520 CN2020102520W WO2021017884A1 WO 2021017884 A1 WO2021017884 A1 WO 2021017884A1 CN 2020102520 W CN2020102520 W CN 2020102520W WO 2021017884 A1 WO2021017884 A1 WO 2021017884A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- processed
- time difference
- time
- preset
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
Definitions
- This application relates to the field of data processing technology, and in particular to a data processing method, device and gateway server.
- monitoring data can be collected in each container cluster, and the monitoring data can be temporarily stored in the gateway server (also called the data receiving gateway), and the central control platform (also called For monitoring data storage processing platform) read monitoring data from the gateway server to analyze the working status of the container cluster.
- the gateway server also called the data receiving gateway
- the central control platform also called For monitoring data storage processing platform
- the purpose of this application is to provide a data processing method, device, and gateway server to ensure the timeliness of monitoring data and at the same time improve the utilization of the gateway storage space.
- an embodiment of the present application provides a data processing method, which includes: obtaining time information of the data to be processed; time information is used to indicate the time when the data to be processed arrives at the current device; and determining the data to be processed according to the time information Whether the preset conditions are met; if the preset conditions are met, delete the pending data.
- the embodiments of the present application provide the first possible implementation manner of the first aspect, wherein the step of determining whether the data to be processed meets a preset condition according to the time information includes: calculating the time information and the current time Time difference; if the time difference is greater than or equal to the preset time difference threshold, it is determined that the preset condition is satisfied; if the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.
- the embodiments of the present application provide a second possible implementation manner of the first aspect, wherein the step of determining whether the data to be processed meets a preset condition according to the time information includes: according to the time information and the data to be processed The cluster status of the corresponding container cluster determines whether the data to be processed meets the preset conditions.
- the embodiments of the present application provide a third possible implementation manner of the first aspect, wherein the above-mentioned to-be-processed data includes multiple; according to the time information and the corresponding to-be-processed data
- the step of determining whether the data to be processed meets the preset conditions includes: calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to the preset time difference threshold, and obtaining each The judgment result of the data to be processed; obtain the cluster status of the container cluster corresponding to each data to be processed; for each data to be processed, if the judgment result of the data to be processed is greater than or equal to the preset time difference threshold, and the current to be processed
- the cluster status of the container cluster corresponding to the data is deleted, and it is determined that the current to-be-processed data meets the preset conditions.
- the embodiments of the present application provide the fourth possible implementation manner of the first aspect, wherein the above-mentioned to-be-processed data includes multiple; according to the time information and the corresponding to-be-processed data
- the step of determining whether the data to be processed meets the preset conditions includes: calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to the preset time difference threshold, and obtaining each Judgment result of the data to be processed; determine the data to be processed whose time difference is greater than or equal to the preset time difference threshold as pending data; obtain the cluster status of the container cluster corresponding to each pending data; for each pending data, if the current If the cluster state of the container cluster corresponding to the pending data is deleted, it is determined that the current pending data meets the preset condition.
- an embodiment of the present application also provides a data processing device, the device includes: an information acquisition module configured to acquire time information of the data to be processed; time information is used to indicate the time when the data to be processed arrives at the current device; a condition determination module , Is set to determine whether the data to be processed meets the preset conditions based on time information; the data deletion module is set to delete the data to be processed if the preset conditions are met.
- the embodiments of the present application provide the first possible implementation manner of the second aspect, wherein the above-mentioned condition determination module includes: a time difference calculation unit configured to calculate the time difference between the time information and the current time; and a first threshold judgment The unit is configured to determine that the preset condition is satisfied if the time difference is greater than or equal to the preset time difference threshold; if the time difference is less than the preset time difference threshold, then it is determined that the preset condition is not satisfied.
- the embodiments of the present application provide a second possible implementation manner of the second aspect, wherein the above-mentioned condition determining module includes: a first condition determining unit configured to correspond to the container cluster according to the time information and the data to be processed To determine whether the data to be processed meets the preset conditions.
- an embodiment of the present application provides a third possible implementation manner of the second aspect, wherein the above-mentioned to-be-processed data includes multiple; the first condition determining unit is configured to: calculate The time difference between the time information of each data to be processed and the current time, to determine whether the time difference is greater than or equal to the preset time difference threshold, to obtain the judgment result of each data to be processed; to obtain the cluster status of the container cluster corresponding to each data to be processed; For each data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, it is determined that the current data to be processed meets the preset conditions .
- an embodiment of the present application provides a fourth possible implementation manner of the second aspect, wherein the aforementioned data to be processed includes multiple; the aforementioned first condition determination unit is configured as: Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to the preset time difference threshold, and obtain the judgment result of each data to be processed; the judgment result is that the time difference is greater than or equal to the preset time difference threshold
- the pending data is determined as pending data; the cluster status of the container cluster corresponding to each pending data is obtained; for each pending data, if the cluster status of the container cluster corresponding to the current pending data is deleted, it is determined that the current pending data meets the pre-determined data Set conditions.
- an embodiment of the present application further provides a gateway server, including a processor and a memory.
- the memory stores machine executable instructions that can be executed by the processor.
- the processor executes the machine executable instructions to implement the above data processing method.
- the embodiments of the present application also provide a machine-readable storage medium.
- the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions cause the processor to implement the above Data processing method.
- the above-mentioned data processing method, device and gateway server first obtain the time information of the data to be processed; then, according to the time information, determine whether the data to be processed meets a preset condition; if the preset conditions are met, delete the data to be processed.
- This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
- FIG. 1 is a schematic diagram of an application scenario of data processing provided by an embodiment of the application
- FIG. 3 is a flowchart of another data processing method provided by an embodiment of the application.
- FIG. 4 is a flowchart of another data processing method provided by an embodiment of the application.
- FIG. 5 is a flowchart of another data processing method provided by an embodiment of the application.
- FIG. 6 is a flowchart of another data processing method provided by an embodiment of the application.
- FIG. 7 is a schematic diagram of a monitoring data receiving process of a multi-container cluster provided by an embodiment of this application.
- FIG. 8 is a schematic structural diagram of a data processing device provided by an embodiment of this application.
- Fig. 9 is a schematic structural diagram of a gateway server provided by an embodiment of the application.
- FIG. 1 shows the application scenario, which includes multiple container clusters, a gateway server that receives data, and a central control platform; among them, the gateway server that receives data can be installed with Pushgateway components, and the central control platform can also be called To monitor the data storage and processing platform, the Prometheus system can be run on the platform.
- the Prometheus system is an open source monitoring and alarm system that can be used to monitor the running status of the container cluster; the Prometheus system usually obtains data from the target device by pulling, but some special monitoring data may not be able to be pulled
- the above-mentioned Pushgateway component can be used, and the Pushgateway component is used to receive the monitoring data pushed to it by the target device and to save and record the monitoring data. It can be seen that the target device can push and save the monitoring data to the Pushgateway component, and then the Prometheus system obtains the monitoring data from the Pushgateway component by pulling the data.
- the data aggregation forwarding agent device in the container cluster can push and save the data collected by various data collectors in the container cluster to the Pushgateway component in the gateway server, and Prometheus in the central control platform will receive data from Pull data from Pushgateway component.
- the data collector in Figure 1 takes the node monitoring data collector, container monitoring data collector, and Kubernetes (k8s) service monitoring data collector as examples. These three data collectors can collect the node processor utilization of the cluster where they are located. Monitoring data such as rate, Pod operating status, kube-apiserver request delay, etc.
- Kubernetes is an open source container orchestration engine, through which Kubernetes can automatically deploy, expand, and monitor container clusters.
- Pod is the basic unit of container cloud scheduling.
- a Pod can be understood as a set of containers, and a container cluster can be divided into multiple Pods; the above-mentioned container monitoring data collector or The Kubernetes service monitoring data collector can monitor the running status of the Pod through related components.
- the running status of the Pod can be running, waiting, normal termination, abnormal termination, unknown status, etc.
- the above-mentioned kube-apiserver is a REST (Representational State Transfer) API (Application Programming Interface) interface used to provide container cluster management.
- the kube-apiserver can be used for authentication, authorization, and data verification of container clusters. It can also be used for storage operations of the container cluster status, etc., so the request delay of kube-apiserver greatly affects the running status of the container cluster.
- the above-mentioned Kubernetes service monitoring data collector can monitor kube through related components. -The request of apiserver is delayed.
- the data summary forwarding agent device in each container cluster collects the data of each data collector in the current cluster, processes and pushes it to the gateway server that receives the data, and the gateway server receives and temporarily stores the data summary forwarding agent device of all container clusters and pushes it over The most recent monitoring data.
- the Prometheus system in the central control platform regularly obtains the collected monitoring data from the gateway server, and analyzes and judges the status of the overall cluster and each cluster based on the monitoring data; in addition, the central control platform also has both retrieval and display functions, which can be used For the retrieval and display of monitoring data.
- the gateway server due to the limited storage space of the gateway server, every time the data aggregation forwarding agent device sends a new piece of data to the gateway server, it will overwrite the previous piece of historical data corresponding to the new data, that is, only each piece of historical data is stored in the gateway server. The latest piece of data in the cluster. Based on this feature, if a cluster is deleted, the gateway will retain the last piece of historical data that the cluster uploaded to the gateway. This will cause two problems: First, when the amount of historical data left is large, It will occupy more storage space of the gateway server; second, because the Prometheus system of the central control platform regularly obtains data from the gateway server, as long as there is historical data in the gateway server, it will be taken away.
- the embodiments of the present application provide a data processing method, device, and gateway server.
- the technology can be applied to various scenarios that require data to be time-sensitive, such as monitoring data of container clusters, monitoring data of network systems, etc. Data processing scene.
- the method can be applied to a gateway server, and the method includes:
- Step S200 Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
- the above-mentioned data to be processed can be various types of monitoring data, sensor data, etc.; for example, monitoring data collected by data collectors in each container cluster, such as network traffic, central processing unit (CPU) usage rate, and Memory usage, etc.; these monitoring data changes in real time according to the working status and system status of the container cluster.
- monitoring data collected by data collectors in each container cluster such as network traffic, central processing unit (CPU) usage rate, and Memory usage, etc.
- CPU central processing unit
- Memory usage etc.
- the aforementioned data to be processed can be actively sent to the current device by other devices.
- the current device is a gateway server
- the data aggregation forwarding agent device in each container cluster sends the data to be processed to the gateway server; the data to be processed can also be sent through the current device.
- the device obtains from other devices.
- the gateway server obtains the data to be processed from the data aggregation forwarding agent device in each container cluster.
- the current device When the data to be processed reaches the current device, the current device usually adds time information to the data to be processed when storing the data to be processed, and the time information indicates the time when the data to be processed arrives at the current device.
- the time information can be carried into the data to be processed and sent to the current device together.
- the time information can be a string of pre-appointed characters representing the time, which are added at the agreed position corresponding to the data to be processed, such as adding a character representing time at the end of the data to be processed; the time information can also be a more common time Stamp, the time stamp is a credential document formed after encryption. Generally, the timestamp is more reliable than the conventional characters mentioned above.
- Step S202 Determine whether the data to be processed meets a preset condition according to the above-mentioned time information.
- the foregoing preset condition is used to indicate whether to delete the to-be-processed data;
- the preset condition may be a condition associated with time information, for example, the preset condition may be a time difference threshold.
- the preset condition can be related to the update cycle of the data to be processed; if the update cycle of the data to be processed is 24 hours, the preset condition can be set as follows: the storage time in the current device is greater than 24 hours; The storage time of the processed data in the current device is 25 hours, and the data to be processed meets the above preset conditions.
- the preset condition can also be set as: the storage time in the current device is greater than n times the update period of the data to be processed, n is greater than or equal to An integer of 1.
- the storage time of the data to be processed in the current device can be compared with the time difference threshold, and it is determined whether the preset condition is met according to the comparison result.
- the storage time of the data to be processed in the current device can be obtained by calculating the time information of the data to be processed and the current time.
- the state of the container cluster corresponding to the data to be processed can also be considered to set preset conditions. For example, if the storage time of the data to be processed in the current device is relatively long, and at the same time, the container cluster corresponding to the data to be processed is deleted, it can be determined that the data to be processed meets the preset condition.
- Step S204 if the above-mentioned preset conditions are met, delete the data to be processed.
- the data to be processed can be deleted from the current device to release the storage space for storing the data to be processed.
- the time information of the data to be processed is obtained; then, according to the time information, it is determined whether the data to be processed meets a preset condition; if the preset conditions are met, the data to be processed is deleted.
- This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
- the embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific implementation process of determining whether the data to be processed meets preset conditions based on time information, such as As shown in Figure 3, the method includes the following steps:
- Step S300 Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
- Step S302 Calculate the time difference between the above-mentioned time information and the current time.
- the current time can be obtained directly from the current device or the Internet; for example, if the current device is a gateway server, the gateway server is usually provided with an accurate timing device, such as a radio clock or an atomic clock.
- the time difference is obtained by subtracting the above-mentioned time information from the current time, that is, the time when the data to be processed arrives at the current device; the time difference can also be understood as the storage time of the data to be processed in the current device.
- Step S304 It is judged whether the above-mentioned time difference is greater than or equal to the preset time difference threshold; if so, step S306 is executed; if not, the end is ended.
- the above-mentioned time difference threshold may be related to the cycle of reading the data to be processed by the central control platform, or may be related to the cycle of updating the data to be processed.
- the central control platform reads the monitoring data in the gateway server (that is, the data to be processed above) in a preset first time period; the data aggregation forwarding agent device of each container cluster uses the preset second time period (that is, the data Update period)
- the monitoring data collected by the data collector is sent to the gateway server, and the second time period corresponding to each container cluster may be different.
- the above-mentioned first time period may be greater than the longest second time period in the container cluster.
- the data aggregation forwarding agent device of each container cluster has sent the monitoring data to the gateway server at least once, that is, the data to be processed is at least Updated once.
- the foregoing first time period may be set according to specific requirements, such as being set to several times the foregoing longest data update period.
- the above-mentioned time difference threshold may be the same as the time period for the central control platform to read the monitoring data.
- the monitoring data of each container cluster stored in the gateway server that is, the data to be processed has been updated at least once; when it is determined that the storage time of the pending data of a certain container cluster in the current device is greater than the preset time difference threshold If the data to be processed is not updated, the data to be processed needs to be deleted.
- the aforementioned time difference threshold may also be the same as the aforementioned longest second time period. If the process of forwarding the monitoring data of each container cluster to the gateway server is normal, within the time difference threshold, the monitoring data of each container cluster stored in the gateway server, that is, the data to be processed, has been updated at least once; When the storage time of the data to be processed in the current device is greater than the preset time difference threshold, it indicates that the data to be processed has not been updated and the data to be processed needs to be deleted.
- the time difference threshold is set to any length of time between the longest second time period and the first time period, if it can be determined that the data to be processed does not meet the preset conditions, it can be guaranteed that the central control platform is adjacent to each other twice
- the read monitoring data are all updated monitoring data. If there are other requirements for the time interval for obtaining updated monitoring data, such as the central control platform cannot read the same monitoring data three times in a row, the time difference threshold can be set correspondingly according to the requirements.
- Step S306 Delete the data to be processed.
- the time difference between the time information of the data to be processed and the current time is calculated, and the time difference is compared with a preset time difference threshold.
- the time difference is greater than the time difference threshold, the data to be processed is deleted;
- the timely updated data to be processed ensures the timeliness of the monitoring data and releases the storage space of the gateway server.
- the embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on determining whether to delete based on the time information of the data to be processed and the cluster status of the corresponding container cluster
- the specific implementation process of the data is shown in Figure 4. The method includes the following steps:
- Step S400 Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
- Step S402 According to the above-mentioned time information and the cluster status of the container cluster corresponding to the data to be processed, it is determined whether the data to be processed meets a preset condition; if it is, step S404 is executed; if not, it ends.
- the aforementioned data to be processed may be monitoring data of the corresponding container cluster; the cluster status of the aforementioned container cluster includes creating (Initial), running (Running), upgrading (Updating), deleted (Deleted), etc.
- the aforementioned preset conditions can be set according to the time information and the cluster status of the corresponding container cluster.
- the preset condition may be divided into a first preset sub-condition and a second preset sub-condition; wherein, the first preset sub-condition may be determining the storage time of the data to be processed according to the above time information, and the storage time is greater than the preset Set the time difference threshold; the above second preset sub-condition can be set as the cluster status corresponding to the data to be processed is deleted.
- the data to be processed it is possible to determine whether the storage time of the data to be processed meets the first sub-preset condition, and obtain the first judgment result; to determine whether the cluster state of the container cluster corresponding to the data to be processed meets the second sub-preset condition, Obtain the second judgment result; then according to the first judgment result and the second judgment result, judge whether the data to be processed meets the first preset sub-condition and the second preset sub-condition at the same time, if yes, confirm that the data to be processed meets the preset condition.
- Step S404 Delete the data to be processed.
- the time information of the data to be processed is obtained, and the data to be processed is cleared when the time information and the state of the container cluster corresponding to the data to be processed meet the preset conditions; this method is in the process of deleting the data to be processed , While considering whether the data to be processed is updated in time, consider whether the container cluster corresponding to the data to be processed has been deleted, ensuring the timeliness of the data to be processed.
- This embodiment of the application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; this method focuses on determining the data to be processed based on time information and cluster status when there are multiple data to be processed
- the specific implementation process of processing whether the data meets the preset conditions is shown in Figure 5.
- the method includes the following steps:
- Step S500 Obtain time information of each data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
- Step S502 Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed.
- the judgment result may include two kinds, one is that the time difference is greater than or equal to the preset time difference threshold, and the other is that the time difference is less than the preset time difference threshold.
- Step S504 Obtain the cluster status of the container cluster corresponding to each data to be processed.
- the relational database of the container cluster management platform stores basic information of each cluster, including the cluster status; you can send a request to the container cluster management platform to inquire about the cluster status of the container cluster corresponding to the data to be processed; The request usually includes the identity of the container cluster in question. After obtaining the cluster state of the container cluster corresponding to the data to be processed, the cluster state can be saved in the storage space set by the current device.
- Step S506 For each data to be processed, determine whether the judgment result of the current data to be processed is greater than or equal to a preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted; if so, perform step S508; If not, end.
- the following processing can be performed on each data to be processed in a preset order: first, obtain the judgment result of the current data to be processed from the judgment result above; then obtain the result of the cluster state obtained above The cluster status of the container cluster corresponding to the current data to be processed; if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, you can determine the current The data to be processed meets preset conditions, and the data to be processed needs to be deleted.
- Step S508 Delete the data to be processed.
- the time information of the data to be processed and the cluster status of the corresponding container cluster are obtained, and the data to be processed is cleared when the time information and the container cluster status meet the preset conditions; this method is deleting the data to be processed In the process, while considering whether the data to be processed is updated in time, it is also considered whether the container cluster corresponding to the data to be processed has been deleted, which ensures the timeliness of the data to be processed.
- This embodiment of the application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; this method focuses on determining the data to be processed based on time information and cluster status when there are multiple data to be processed
- the specific implementation process of processing whether the data meets the preset conditions is shown in Figure 6.
- the method includes the following steps:
- Step S600 Obtain time information of each data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
- Step S602 Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed.
- Step S604 Determine the to-be-processed data whose time difference is greater than or equal to a preset time difference threshold as a result of the judgment as pending data.
- Step S606 Obtain the cluster status of the container cluster corresponding to each pending data.
- the undetermined data is a part of the data selected from the above-mentioned to-be-processed data based on the judgment in step S602. Generally, the amount of the undetermined data is not greater than the to-be-processed data.
- a request can be sent to the container cluster management platform to query the cluster status corresponding to the cluster status of the container cluster corresponding to each pending data, and receive the information returned by the platform to obtain the cluster status of the container cluster corresponding to each pending data.
- Step S608 For each pending data, it is determined whether the cluster state of the container cluster corresponding to the current pending data is deleted; if so, step 610 is executed; if not, the end is ended.
- Step S610 Delete the data to be processed.
- the cluster status of the corresponding container cluster is acquired, and the data to be processed is deleted after the cluster status is determined to be deleted; this method only acquires The cluster status of the container cluster corresponding to the pending data that is not updated in time is used to delete pending data that is not updated in time and the cluster status is deleted.
- the number of queries to the training state is reduced. , It saves computing resources and is more efficient.
- the embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the foregoing embodiment; the method is applied to the monitoring data reception and adaptation process of a multi-container cluster.
- FIG. 7 A schematic diagram of the monitoring data receiving process of a multi-container cluster is shown in Figure 7.
- the data aggregation forwarding agent device also called a collection agent
- obtains the monitoring data of the container cluster from the data collector also called the collection object
- the monitoring data is sent to the gateway server;
- the gateway server receives the monitoring data, stores the monitoring data in the data queue, and then stores the data in the memory of the gateway server; after receiving the monitoring data belonging to a container cluster, the gateway server
- the historical monitoring data of the container cluster in the memory is updated to the current monitoring data;
- the central control platform also referred to as the monitoring platform
- the data processing method provided in the above embodiment Mainly applied to the gateway server, the data cleaning function of the gateway server memory is realized between the data storage and data retrieval steps.
- the method includes the following steps:
- Step 1 Obtain the time stamp of each stored monitoring data; where the time stamp is the time when the gateway server receives the data.
- Step 2 Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
- Step 3 Compare the time difference with the preset threshold.
- Step 4 Determine whether each piece of monitoring data is invalid data based on the comparison result, that is, if the time difference is greater than a preset threshold, then determine that the monitoring data is invalid data.
- Step 5 Clear invalid data.
- the above method can be used to process some invalid data regularly.
- some monitoring data may be deleted as invalid data by mistake; for example, when the connection between the container cluster and the gateway server is interrupted due to some problems, the data stored in the gateway and the cluster The corresponding monitoring data cannot be updated.
- the monitoring data will be determined as invalid data and deleted.
- the container cluster corresponding to the monitoring data is not deleted.
- the monitoring data corresponding to the container cluster can continue to be updated. Therefore, after considering the cluster status of the container cluster corresponding to the monitoring data, after improving the method, the method can be implemented through the following steps:
- Step 1 Obtain the timestamp of each monitored data.
- Step 2 Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
- Step 3 Compare the time difference with the preset threshold to obtain the comparison result.
- Step 4 Obtain the cluster status of the container cluster corresponding to each monitoring data.
- Step 5 Determine whether each piece of data is invalid data based on the comparison result and the cluster status of the container cluster corresponding to each monitoring data; specifically, if the comparison result time difference is greater than a preset threshold, and the monitoring data corresponds to the container If the cluster status of the cluster is deleted, the data is determined to be invalid data.
- Step 6 Clear invalid data.
- the amount of data for obtaining the status of the cluster can be reduced through the judgment of the timestamp, and then the corresponding cluster status can be queried; at this time, the method can be implemented through the following steps:
- Step 1 Obtain the timestamp of each monitored data.
- Step 2 Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
- Step 3 The time difference is compared with the preset threshold to obtain the comparison result; if the comparison result is that the time difference is greater than the preset threshold, the monitoring data is determined to be invalid data to be determined.
- Step 4 Obtain the cluster status of the container cluster corresponding to each pending invalid data.
- Step 5 Determine whether the pending invalid data is invalid data based on the cluster status of the container cluster corresponding to each piece of invalid data to be determined; if the cluster status is a deleted state, determine whether the invalid data to be determined is invalid data.
- Step 6 Clear invalid data.
- the gateway server According to the time difference between the time stamp of the monitoring data and the current time, and the cluster status of the container cluster corresponding to the monitoring data, determine whether the monitoring data is invalid data, delete the invalid data, and release the storage space of the gateway server. Improved the utilization of gateway storage space.
- an embodiment of the present application provides a data processing device. As shown in FIG. 8, the device includes:
- the information acquisition module 800 is configured to acquire time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device;
- the condition determining module 802 is configured to determine whether the data to be processed meets a preset condition according to the time information
- the data deletion module 804 is configured to delete the data to be processed if the preset conditions are met.
- the above-mentioned data processing device first obtains time information of the data to be processed; and then determines whether the data to be processed meets a preset condition according to the time information; if the preset conditions are met, the data to be processed is deleted. This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
- the above-mentioned condition determination module includes: a time difference calculation unit configured to calculate the time difference between the time information and the current time; a first threshold judgment unit configured to: if the time difference is greater than or equal to a preset time difference threshold, then It is determined that the preset condition is satisfied; if the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.
- the above-mentioned condition determination module includes: a first condition determination unit configured to determine whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed.
- the first condition determining unit is configured to: calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain each The judgment result of the processed data; obtain the cluster status of the container cluster corresponding to each data to be processed; for each data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the current data to be processed The cluster status of the corresponding container cluster is deleted, and it is determined that the current to-be-processed data meets the preset conditions.
- the first condition determining unit is configured to: calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain each The judgment result of the processed data; the judgment result is that the time difference is greater than or equal to the preset time difference threshold to determine the pending data as pending data; the cluster status of the container cluster corresponding to each pending data is obtained; for each pending data, if the current pending If the cluster status of the container cluster corresponding to the data is deleted, it is determined that the current pending data meets the preset condition.
- An embodiment of the present application also provides a gateway server.
- the gateway server includes a processor 130 and a memory 131.
- the memory 131 stores machine executable instructions that can be executed by the processor 130.
- the machine executable instructions are executed to implement the above-mentioned data processing method.
- the gateway server shown in FIG. 9 further includes a bus 132 and a communication interface 133, and the processor 130, the communication interface 133, and the memory 131 are connected through the bus 132.
- the memory 131 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
- RAM Random Access Memory
- non-volatile memory such as at least one disk memory.
- the communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. may be used.
- the bus 132 may be an ISA bus, PCI bus, EISA bus, or the like. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one bidirectional arrow is used in FIG. 9, but it does not mean that there is only one bus or one type of bus.
- the processor 130 may be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the foregoing method may be completed by an integrated logic circuit of hardware in the processor 130 or instructions in the form of software.
- the aforementioned processor 130 may be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processing, DSP for short). ), Application Specific Integrated Circuit (ASIC for short), Field-Programmable Gate Array (FPGA for short) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
- CPU central processing unit
- NP Network Processor
- DSP Digital Signal Processing
- ASIC Application Specific Integrated Circuit
- FPGA Field-Programmable Gate Array
- the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
- the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
- the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
- the storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131, and completes the steps of the method of the foregoing embodiment in combination with its hardware.
- the embodiment of the present application also provides a machine-readable storage medium that stores machine-executable instructions.
- the machine-executable instructions When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor.
- the above-mentioned data processing method is implemented. For specific implementation, please refer to the method embodiment, which will not be repeated here.
- the data processing method and device and the computer program product of the gateway server provided by the embodiments of the present application include a computer-readable storage medium storing program code.
- the instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For the specific implementation of the method, please refer to the method embodiment, which will not be repeated here.
- the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
- the computer software product is stored in a storage medium, including several
- the instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
- the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code .
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Telephonic Communication Services (AREA)
- Computer And Data Communications (AREA)
Abstract
The present application provides a data processing method and apparatus, and a gateway server. The method comprises: obtaining time information of data to be processed, the time information being used for indicating the time when said data arrives at the current device; determining whether said data satisfies a preset condition according to the time information; and if the preset condition is satisfied, deleting said data. According to the present application, the invalid data is cleared by means of the time information, the timeliness of monitoring data is guaranteed, and the utilization rate of a gateway storage space is increased at the same time.
Description
本申请要求于2019年07月31日提交中国专利局、申请号为201910706918.0、申请名称为“数据处理方法、装置及网关服务器”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201910706918.0, and the application name is "data processing method, device and gateway server" on July 31, 2019, the entire content of which is incorporated into this application by reference in.
本申请涉及数据处理技术领域,尤其是涉及一种数据处理方法、装置及网关服务器。This application relates to the field of data processing technology, and in particular to a data processing method, device and gateway server.
相关技术中,为了监控容器集群的工作状态,可以在各个容器集群中进行监控数据采集,并将监控数据暂存于网关服务器(也可以称为数据接收网关)中,中控平台(也可以称为监控数据存储处理平台)从网关服务器中读取监控数据以分析容器集群的工作状态。受网关服务器存储空间的限制,历史监控数据会被新的监控数据覆盖;如果某个容器集群被删除,关于该容器集群的监控数据不再更新;而网关服务器中依然保存该容器集群的历史监控数据,不仅浪费网关服务器的存储空间,而且不利于后续基于监控数据对整体集群的状态分析和判断。In related technologies, in order to monitor the working status of the container clusters, monitoring data can be collected in each container cluster, and the monitoring data can be temporarily stored in the gateway server (also called the data receiving gateway), and the central control platform (also called For monitoring data storage processing platform) read monitoring data from the gateway server to analyze the working status of the container cluster. Limited by the storage space of the gateway server, historical monitoring data will be overwritten by new monitoring data; if a container cluster is deleted, the monitoring data about the container cluster will no longer be updated; and the gateway server still saves the historical monitoring of the container cluster Data not only wastes the storage space of the gateway server, but also is not conducive to subsequent analysis and judgment of the overall cluster status based on monitoring data.
发明内容Summary of the invention
本申请的目的在于提供一种数据处理方法、装置及网关服务器,以保证监控数据的时效性,同时提高网关存储空间的利用率。The purpose of this application is to provide a data processing method, device, and gateway server to ensure the timeliness of monitoring data and at the same time improve the utilization of the gateway storage space.
第一方面,本申请实施例提供了一种数据处理方法,该方法包括:获取待处理数据的时间信息;时间信息用于指示待处理数据到达当前设备的时间;根据时间信息,确定待处理数据是否满足预设条件;如果满足预设条件,删除待处理数据。In the first aspect, an embodiment of the present application provides a data processing method, which includes: obtaining time information of the data to be processed; time information is used to indicate the time when the data to be processed arrives at the current device; and determining the data to be processed according to the time information Whether the preset conditions are met; if the preset conditions are met, delete the pending data.
结合第一方面,本申请实施例提供了第一方面的第一种可能的实施方式,其中,根据时间信息,确定待处理数据是否满足预设条件的步骤,包括:计算时间信息与当前时间的时间差;如果时间差大于或等于预设的时间差阈值,则确定满足预设条件;如果所述时间差小于预设的时间差阈值,则确定不满足预设条件。With reference to the first aspect, the embodiments of the present application provide the first possible implementation manner of the first aspect, wherein the step of determining whether the data to be processed meets a preset condition according to the time information includes: calculating the time information and the current time Time difference; if the time difference is greater than or equal to the preset time difference threshold, it is determined that the preset condition is satisfied; if the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.
结合第一方面,本申请实施例提供了第一方面的第二种可能的实施方式,其中,根据时间信息,确定待处理数据是否满足预设条件的步骤,包括:根据时间信息和待处理数据对应的容器集群的集群状态,确定待处理数据是否满足预设条件。With reference to the first aspect, the embodiments of the present application provide a second possible implementation manner of the first aspect, wherein the step of determining whether the data to be processed meets a preset condition according to the time information includes: according to the time information and the data to be processed The cluster status of the corresponding container cluster determines whether the data to be processed meets the preset conditions.
结合第一方面的第二种可能的实施方式,本申请实施例提供了第一方面的第三种可能的实施方式,其中,上述待处理数据包括多个;根据时间信息和待处理数据对应的容器集群的集群状态,确定待处理数据是否满足预设条件的步骤,包括:计算每个待处理数据的时间信息与当前时间的时间差,判断时间差是否大于或等于预设的时间差阈值,得到每个待处理数据的判断结果;获取每个待处理数据对应的容器集群的集群状态;针对每个待处理数据,如果当前待处理数据的判断结果为大于或等于预设的时间差阈值,且当前待处理数据对应的容器集群的集群状态为已删除,确定当前待处理数据满足预设条件。In combination with the second possible implementation manner of the first aspect, the embodiments of the present application provide a third possible implementation manner of the first aspect, wherein the above-mentioned to-be-processed data includes multiple; according to the time information and the corresponding to-be-processed data For the cluster status of the container cluster, the step of determining whether the data to be processed meets the preset conditions includes: calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to the preset time difference threshold, and obtaining each The judgment result of the data to be processed; obtain the cluster status of the container cluster corresponding to each data to be processed; for each data to be processed, if the judgment result of the data to be processed is greater than or equal to the preset time difference threshold, and the current to be processed The cluster status of the container cluster corresponding to the data is deleted, and it is determined that the current to-be-processed data meets the preset conditions.
结合第一方面的第二种可能的实施方式,本申请实施例提供了第一方面的第四种可能的实施方式,其中,上述待处理数据包括多个;根据时间信息和待处理数据对应的容器集 群的集群状态,确定待处理数据是否满足预设条件的步骤,包括:计算每个待处理数据的时间信息与当前时间的时间差,判断时间差是否大于或等于预设的时间差阈值,得到每个待处理数据的判断结果;将判断结果为时间差大于或等于预设的时间差阈值的待处理数据确定为待定数据;获取每个待定数据对应的容器集群的集群状态;针对每个待定数据,如果当前待定数据对应的容器集群的集群状态是已删除,则确定当前待定数据满足预设条件。In combination with the second possible implementation manner of the first aspect, the embodiments of the present application provide the fourth possible implementation manner of the first aspect, wherein the above-mentioned to-be-processed data includes multiple; according to the time information and the corresponding to-be-processed data For the cluster status of the container cluster, the step of determining whether the data to be processed meets the preset conditions includes: calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to the preset time difference threshold, and obtaining each Judgment result of the data to be processed; determine the data to be processed whose time difference is greater than or equal to the preset time difference threshold as pending data; obtain the cluster status of the container cluster corresponding to each pending data; for each pending data, if the current If the cluster state of the container cluster corresponding to the pending data is deleted, it is determined that the current pending data meets the preset condition.
第二方面,本申请实施例还提供一种数据处理装置,装置包括:信息获取模块,设置为获取待处理数据的时间信息;时间信息用于指示待处理数据到达当前设备的时间;条件确定模块,设置为根据时间信息,确定待处理数据是否满足预设条件;数据删除模块,设置为如果满足预设条件,删除待处理数据。In a second aspect, an embodiment of the present application also provides a data processing device, the device includes: an information acquisition module configured to acquire time information of the data to be processed; time information is used to indicate the time when the data to be processed arrives at the current device; a condition determination module , Is set to determine whether the data to be processed meets the preset conditions based on time information; the data deletion module is set to delete the data to be processed if the preset conditions are met.
结合第二方面,本申请实施例提供了第二方面的第一种可能的实施方式,其中,上述条件确定模块包括:时间差计算单元,设置为计算时间信息与当前时间的时间差;第一阈值判断单元,设置为如果时间差大于或等于预设的时间差阈值,则确定满足预设条件;如果所述时间差小于预设的时间差阈值,则确定不满足预设条件。With reference to the second aspect, the embodiments of the present application provide the first possible implementation manner of the second aspect, wherein the above-mentioned condition determination module includes: a time difference calculation unit configured to calculate the time difference between the time information and the current time; and a first threshold judgment The unit is configured to determine that the preset condition is satisfied if the time difference is greater than or equal to the preset time difference threshold; if the time difference is less than the preset time difference threshold, then it is determined that the preset condition is not satisfied.
结合第二方面,本申请实施例提供了第二方面的第二种可能的实施方式,其中,上述条件确定模块包括:第一条件确定单元,设置为根据时间信息和待处理数据对应的容器集群的集群状态,确定待处理数据是否满足预设条件。With reference to the second aspect, the embodiments of the present application provide a second possible implementation manner of the second aspect, wherein the above-mentioned condition determining module includes: a first condition determining unit configured to correspond to the container cluster according to the time information and the data to be processed To determine whether the data to be processed meets the preset conditions.
结合第二方面的第二种可能的实施方式,本申请实施例提供了第二方面的第三种可能的实施方式,其中,上述待处理数据包括多个;第一条件确定单元设置为:计算每个待处理数据的时间信息与当前时间的时间差,判断时间差是否大于或等于预设的时间差阈值,得到每个待处理数据的判断结果;获取每个待处理数据对应的容器集群的集群状态;针对每个待处理数据,如果当前待处理数据的判断结果为大于或等于预设的时间差阈值,且当前待处理数据对应的容器集群的集群状态为已删除,确定当前待处理数据满足预设条件。In combination with the second possible implementation manner of the second aspect, an embodiment of the present application provides a third possible implementation manner of the second aspect, wherein the above-mentioned to-be-processed data includes multiple; the first condition determining unit is configured to: calculate The time difference between the time information of each data to be processed and the current time, to determine whether the time difference is greater than or equal to the preset time difference threshold, to obtain the judgment result of each data to be processed; to obtain the cluster status of the container cluster corresponding to each data to be processed; For each data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, it is determined that the current data to be processed meets the preset conditions .
结合第二方面的第二种可能的实施方式,本申请实施例提供了第二方面的第四种可能的实施方式,其中,上述待处理数据包括多个;上述第一条件确定单元设置为:计算每个待处理数据的时间信息与当前时间的时间差,判断时间差是否大于或等于预设的时间差阈值,得到每个待处理数据的判断结果;将判断结果为时间差大于或等于预设的时间差阈值的待处理数据确定为待定数据;获取每个待定数据对应的容器集群的集群状态;针对每个待定数据,如果当前待定数据对应的容器集群的集群状态是已删除,则确定当前待定数据满足预设条件。In combination with the second possible implementation manner of the second aspect, an embodiment of the present application provides a fourth possible implementation manner of the second aspect, wherein the aforementioned data to be processed includes multiple; the aforementioned first condition determination unit is configured as: Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to the preset time difference threshold, and obtain the judgment result of each data to be processed; the judgment result is that the time difference is greater than or equal to the preset time difference threshold The pending data is determined as pending data; the cluster status of the container cluster corresponding to each pending data is obtained; for each pending data, if the cluster status of the container cluster corresponding to the current pending data is deleted, it is determined that the current pending data meets the pre-determined data Set conditions.
第三方面,本申请实施例还提供一种网关服务器,包括处理器和存储器,存储器存储有能够被处理器执行的机器可执行指令,处理器执行机器可执行指令以实现上述数据处理方法。In a third aspect, an embodiment of the present application further provides a gateway server, including a processor and a memory. The memory stores machine executable instructions that can be executed by the processor. The processor executes the machine executable instructions to implement the above data processing method.
第四方面,本申请实施例还提供机器可读存储介质,机器可读存储介质存储有机器可执行指令,机器可执行指令在被处理器调用和执行时,机器可执行指令促使处理器实现上述数据处理方法。In a fourth aspect, the embodiments of the present application also provide a machine-readable storage medium. The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions cause the processor to implement the above Data processing method.
上述一种数据处理方法、装置及网关服务器,首先获取待处理数据的时间信息;进而根据该时间信息,确定待处理数据是否满足预设条件;如果满足预设条件,则删除待处理数据。该方式通过时间信息清理无效数据,保证了监控数据的时效性,同时提高了网关存 储空间的利用率。The above-mentioned data processing method, device and gateway server first obtain the time information of the data to be processed; then, according to the time information, determine whether the data to be processed meets a preset condition; if the preset conditions are met, delete the data to be processed. This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
为了更清楚地说明本申请具体实施方式或相关技术中的技术方案,下面将对具体实施方式或相关技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图是本申请的一些实施方式,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the specific embodiments of the application or the technical solutions in related technologies, the following will briefly introduce the drawings that need to be used in the specific embodiments or related technical descriptions. Obviously, the drawings in the following description are For some of the embodiments of the present application, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.
图1为本申请实施例提供的一种数据处理的应用场景的示意图;FIG. 1 is a schematic diagram of an application scenario of data processing provided by an embodiment of the application;
图2为本申请实施例提供的一种数据处理方法的流程图;2 is a flowchart of a data processing method provided by an embodiment of the application;
图3为本申请实施例提供的另一种数据处理方法的流程图;FIG. 3 is a flowchart of another data processing method provided by an embodiment of the application;
图4为本申请实施例提供的另一种数据处理方法的流程图;FIG. 4 is a flowchart of another data processing method provided by an embodiment of the application;
图5为本申请实施例提供的另一种数据处理方法的流程图;FIG. 5 is a flowchart of another data processing method provided by an embodiment of the application;
图6为本申请实施例提供的另一种数据处理方法的流程图;FIG. 6 is a flowchart of another data processing method provided by an embodiment of the application;
图7为本申请实施例提供的一种多容器集群的监控数据接收过程示意图;FIG. 7 is a schematic diagram of a monitoring data receiving process of a multi-container cluster provided by an embodiment of this application;
图8为本申请实施例提供的一种数据处理装置的结构示意图;FIG. 8 is a schematic structural diagram of a data processing device provided by an embodiment of this application;
图9为本申请实施例提供的一种网关服务器的结构示意图。Fig. 9 is a schematic structural diagram of a gateway server provided by an embodiment of the application.
下面将结合实施例对本申请的技术方案进行清楚、完整地描述,显然,所描述的实施例是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。The technical solutions of the present application will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.
相关技术中,可以在每个容器集群中设置多个数据采集器,以采集所在集群的各种监控数据,同时设置与各个采集器连接的数据汇总转发代理设备,用于收集各种数据采集器采集的数据,并推送或转发到接收数据的网关服务器。为了便于理解,图1示出该应用场景,该场景包括多个容器集群、接收数据的网关服务器以及中控平台;其中,接收数据的网关服务器上可以安装有Pushgateway组件,中控平台也可以称为监控数据存储处理平台,该平台中可以运行有Prometheus系统。In related technologies, multiple data collectors can be set up in each container cluster to collect various monitoring data of the cluster, and at the same time, a data summary forwarding agent device connected to each collector can be set to collect various data collectors. The collected data is pushed or forwarded to the gateway server that receives the data. For ease of understanding, Figure 1 shows the application scenario, which includes multiple container clusters, a gateway server that receives data, and a central control platform; among them, the gateway server that receives data can be installed with Pushgateway components, and the central control platform can also be called To monitor the data storage and processing platform, the Prometheus system can be run on the platform.
该Prometheus系统是一种开源的监控报警系统,可以用于监控容器集群的运行状态;Prometheus系统通常采用拉取的方式从目标设备中获取数据,但是对于一些特殊的监控数据可能不能通过拉取的方式获得,此时,可以采用上述Pushgateway组件,该Pushgateway组件用于接收目标设备向其推送的监控数据并对该监控数据进行保存记录。可见,目标设备可以将监控数据推送并保存至Pushgateway组件中,再由Prometheus系统以拉取数据的方式从Pushgateway组件中获取监控数据。例如,上述容器集群中的数据汇总转发代理设备,可以将该容器集群中各种数据采集器采集的数据推送并保存至网关服务器中的Pushgateway组件,中控平台中的Prometheus以一定的时间频率从Pushgateway组件中拉取数据。The Prometheus system is an open source monitoring and alarm system that can be used to monitor the running status of the container cluster; the Prometheus system usually obtains data from the target device by pulling, but some special monitoring data may not be able to be pulled In this case, the above-mentioned Pushgateway component can be used, and the Pushgateway component is used to receive the monitoring data pushed to it by the target device and to save and record the monitoring data. It can be seen that the target device can push and save the monitoring data to the Pushgateway component, and then the Prometheus system obtains the monitoring data from the Pushgateway component by pulling the data. For example, the data aggregation forwarding agent device in the container cluster can push and save the data collected by various data collectors in the container cluster to the Pushgateway component in the gateway server, and Prometheus in the central control platform will receive data from Pull data from Pushgateway component.
图1中的数据采集器以节点监控数据采集器、容器监控数据采集器及Kubernetes(简称k8s)服务监控数据采集器为例进行说明,这三种数据采集器可以采集所在集群的节点处理器利用率、Pod运行状态、kube-apiserver请求延迟等监控数据。其中,Kubernetes是一种开源的容器编排引擎,通过Kubernetes可以自动化地部署、扩展、监控容器集群。Kubernetes内设置有多种组件用于监控容器集群的运行状态,如Probes组件、cAdvisor组件等,这些组件可以设置在上述容器监控数据采集器或者上述Kubernetes服务监控数据采集器中,以采集部分容器集群的监控数据。The data collector in Figure 1 takes the node monitoring data collector, container monitoring data collector, and Kubernetes (k8s) service monitoring data collector as examples. These three data collectors can collect the node processor utilization of the cluster where they are located. Monitoring data such as rate, Pod operating status, kube-apiserver request delay, etc. Among them, Kubernetes is an open source container orchestration engine, through which Kubernetes can automatically deploy, expand, and monitor container clusters. There are a variety of components in Kubernetes to monitor the running status of the container cluster, such as Probes, cAdvisor, etc. These components can be set in the container monitoring data collector or the Kubernetes service monitoring data collector to collect part of the container cluster Monitoring data.
通常,容器集群中的所有容器均运行在Pod中,Pod是容器云调度的基础单位,一个Pod可以理解为一组容器集合,一个容器集群可以划分为多个Pod;上述容器监控数据采集器或者Kubernetes服务监控数据采集器可以通过相关组件监控Pod的运行状态,该Pod的运行状态可以为运行中、等待中、正常终止、异常终止、未知状态等。Generally, all containers in a container cluster run in Pods. Pod is the basic unit of container cloud scheduling. A Pod can be understood as a set of containers, and a container cluster can be divided into multiple Pods; the above-mentioned container monitoring data collector or The Kubernetes service monitoring data collector can monitor the running status of the Pod through related components. The running status of the Pod can be running, waiting, normal termination, abnormal termination, unknown status, etc.
上述kube-apiserver是用于提供容器集群管理的REST(Representational State Transfer,表现层状态转移)API(Application Programming Interface,应用程序接口)接口,该kube-apiserver可以用于容器集群的认证授权、数据校验以及集群状态变更等,还可以用于容器集群状态的存储操作等,因此kube-apiserver的请求延迟很大程度上影响容器集群的运行状态,上述Kubernetes服务监控数据采集器可以通过相关组件监控kube-apiserver的请求延迟。The above-mentioned kube-apiserver is a REST (Representational State Transfer) API (Application Programming Interface) interface used to provide container cluster management. The kube-apiserver can be used for authentication, authorization, and data verification of container clusters. It can also be used for storage operations of the container cluster status, etc., so the request delay of kube-apiserver greatly affects the running status of the container cluster. The above-mentioned Kubernetes service monitoring data collector can monitor kube through related components. -The request of apiserver is delayed.
每个容器集群中的数据汇总转发代理设备收集当前集群每个数据采集器的数据,处理并推送到接收数据的网关服务器,该网关服务器接收并暂存所有容器集群的数据汇总转发代理设备推送过来的最近一次监控数据。中控平台中的Prometheus系统定时从网关服务器获取采集的监控数据,并根据该监控数据对整体集群及各个集群的状态分析和判断;此外,该中控平台还兼具检索和展示功能,可以用于监控数据的检索和展示。The data summary forwarding agent device in each container cluster collects the data of each data collector in the current cluster, processes and pushes it to the gateway server that receives the data, and the gateway server receives and temporarily stores the data summary forwarding agent device of all container clusters and pushes it over The most recent monitoring data. The Prometheus system in the central control platform regularly obtains the collected monitoring data from the gateway server, and analyzes and judges the status of the overall cluster and each cluster based on the monitoring data; in addition, the central control platform also has both retrieval and display functions, which can be used For the retrieval and display of monitoring data.
在具体工作过程中,由于网关服务器的存储空间有限,因此数据汇总转发代理设备每向网关服务器发送一条新数据时,都会覆盖该新数据对应的上一条历史数据,即网关服务器中只存储每个集群的最新的一条数据。基于该特点,若某个集群被删除后,网关中还会保留该集群最后上传至网关的一条历史数据,这样就会产生两个问题:其一,遗留的历史数据的数据量大的时候,会占据网关服务器较多的存储空间;其二,由于中控平台的Prometheus系统在定时从网关服务器获取数据时,只要网关服务器中存在历史数据,就会将其取走,因此,若某个集群已经被删除,Prometheus系统会不停地获取同一条历史数据,这些重复的历史数据是无效采样,会误导Prometheus系统后续判断分析,不利于基于监控数据对整体集群的状态分析和判断。In the specific work process, due to the limited storage space of the gateway server, every time the data aggregation forwarding agent device sends a new piece of data to the gateway server, it will overwrite the previous piece of historical data corresponding to the new data, that is, only each piece of historical data is stored in the gateway server. The latest piece of data in the cluster. Based on this feature, if a cluster is deleted, the gateway will retain the last piece of historical data that the cluster uploaded to the gateway. This will cause two problems: First, when the amount of historical data left is large, It will occupy more storage space of the gateway server; second, because the Prometheus system of the central control platform regularly obtains data from the gateway server, as long as there is historical data in the gateway server, it will be taken away. Therefore, if a cluster If it has been deleted, the Prometheus system will continuously obtain the same historical data. These repeated historical data are invalid samples, which will mislead the Prometheus system's subsequent judgment and analysis, which is not conducive to the analysis and judgment of the overall cluster status based on the monitoring data.
基于此,本申请实施例提供一种数据处理方法、装置及网关服务器,该技术可以应用于各种要求数据具有时效性的场景中,如容器集群的监控数据、网络系统的状态监控数据等的数据处理场景中。Based on this, the embodiments of the present application provide a data processing method, device, and gateway server. The technology can be applied to various scenarios that require data to be time-sensitive, such as monitoring data of container clusters, monitoring data of network systems, etc. Data processing scene.
首先参见图2所示的一种数据处理方法的流程图;该方法可以应用于网关服务器,该方法包括:First, refer to the flowchart of a data processing method shown in Figure 2; the method can be applied to a gateway server, and the method includes:
步骤S200,获取待处理数据的时间信息;该时间信息用于指示待处理数据到达当前设备的时间。Step S200: Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
上述待处理数据可以为各类监控数据、传感数据等;例如,通过各个容器集群中的数据采集器采集到的监控数据,如网络流量、中央处理器(Central Processing Unit,CPU)使用率及内存使用情况等;这些监控数据都是根据容器集群的工作状态及系统状态实时变化的,通过对监控数据进行分析及统计,可以分析及评价各个容器集群及容器集群系统的状态。上述待处理数据可以由其他设备主动发送至当前设备,例如,如果当前设备是网关服务器时,各个容器集群中的数据汇总转发代理设备向网关服务器发送待处理数据;上述待处理数据也可以通过当前设备从其他设备获取,如网关服务器从每个容器集群中的数据汇总转发代理设备获取待处理数据。The above-mentioned data to be processed can be various types of monitoring data, sensor data, etc.; for example, monitoring data collected by data collectors in each container cluster, such as network traffic, central processing unit (CPU) usage rate, and Memory usage, etc.; these monitoring data changes in real time according to the working status and system status of the container cluster. By analyzing and counting the monitoring data, the status of each container cluster and container cluster system can be analyzed and evaluated. The aforementioned data to be processed can be actively sent to the current device by other devices. For example, if the current device is a gateway server, the data aggregation forwarding agent device in each container cluster sends the data to be processed to the gateway server; the data to be processed can also be sent through the current device. The device obtains from other devices. For example, the gateway server obtains the data to be processed from the data aggregation forwarding agent device in each container cluster.
当待处理数据到达当前设备时,当前设备在存储该待处理数据的时候通常会为该待处理数据添加时间信息,该时间信息指示待处理数据到达当前设备的时间。另一种方式中,与当前设备连接的其他设备发送待处理数据时,可以将时间信息携带至待处理数据中,一同发送至当前设备。该时间信息可以为一串预先约定的表示时间的字符,该字符添加在待处理数据对应的约定位置,如在待处理数据结尾处添加表示时间的字符;该时间信息还可以为较为常见的时间戳,时间戳是一个经加密后形成的凭证文档。通常,相比于上述约定字符,时间戳更为可信。When the data to be processed reaches the current device, the current device usually adds time information to the data to be processed when storing the data to be processed, and the time information indicates the time when the data to be processed arrives at the current device. In another way, when other devices connected to the current device send data to be processed, the time information can be carried into the data to be processed and sent to the current device together. The time information can be a string of pre-appointed characters representing the time, which are added at the agreed position corresponding to the data to be processed, such as adding a character representing time at the end of the data to be processed; the time information can also be a more common time Stamp, the time stamp is a credential document formed after encryption. Generally, the timestamp is more reliable than the conventional characters mentioned above.
步骤S202,根据上述时间信息,确定待处理数据是否满足预设条件。Step S202: Determine whether the data to be processed meets a preset condition according to the above-mentioned time information.
上述预设条件用于指示是否删除该待处理数据;该预设条件可以为与时间信息相关联的条件,如该预设条件可以是一个时间差阈值。具体而言,该预设条件可以与待处理数据的更新周期相关;如待处理数据的更新周期为24小时,预设条件可以设定为:在当前设备中的存储时间大于24小时;如果待处理数据在当前设备的存储时间为25小时,待处理数据满足上述预设条件。考虑到更新待处理数据过程中,可能出现网络问题导致的延迟更新,也可以将预设条件设置为:在当前设备中的存储时间大于n倍的待处理数据的更新周期,n为大于或等于1的整数。The foregoing preset condition is used to indicate whether to delete the to-be-processed data; the preset condition may be a condition associated with time information, for example, the preset condition may be a time difference threshold. Specifically, the preset condition can be related to the update cycle of the data to be processed; if the update cycle of the data to be processed is 24 hours, the preset condition can be set as follows: the storage time in the current device is greater than 24 hours; The storage time of the processed data in the current device is 25 hours, and the data to be processed meets the above preset conditions. Taking into account that during the process of updating the data to be processed, there may be delayed updates caused by network problems, the preset condition can also be set as: the storage time in the current device is greater than n times the update period of the data to be processed, n is greater than or equal to An integer of 1.
在确定待处理数据是否满足预设条件的过程中,可以将待处理数据在当前设备中的存储时间与时间差阈值比较,根据比较结果确定是否满足预设条件。上述待处理数据在当前设备中的存储时间,可以通过将待处理数据的时间信息与当前时间进行运算得到。In the process of determining whether the data to be processed meets the preset condition, the storage time of the data to be processed in the current device can be compared with the time difference threshold, and it is determined whether the preset condition is met according to the comparison result. The storage time of the data to be processed in the current device can be obtained by calculating the time information of the data to be processed and the current time.
此外,在时间信息的基础上,同时还可以考虑待处理数据对应的容器集群的状态设置预设条件。例如,如果待处理数据在当前设备中的存储时间较长,同时,该待处理数据对应的容器集群被删除,则可以确定待处理数据满足预设条件。In addition, on the basis of time information, the state of the container cluster corresponding to the data to be processed can also be considered to set preset conditions. For example, if the storage time of the data to be processed in the current device is relatively long, and at the same time, the container cluster corresponding to the data to be processed is deleted, it can be determined that the data to be processed meets the preset condition.
步骤S204,如果满足上述预设条件,删除待处理数据。Step S204, if the above-mentioned preset conditions are met, delete the data to be processed.
当待处理数据满足预设条件时,可以将待处理数据从当前设备中删除,以释放保存该待处理数据的存储空间。When the data to be processed meets the preset condition, the data to be processed can be deleted from the current device to release the storage space for storing the data to be processed.
上述数据处理方法,首先获取待处理数据的时间信息;进而根据该时间信息,确定待处理数据是否满足预设条件;如果满足预设条件,则删除待处理数据。该方式通过时间信息清理无效数据,保证了监控数据的时效性,同时提高了网关存储空间的利用率。In the above data processing method, firstly, the time information of the data to be processed is obtained; then, according to the time information, it is determined whether the data to be processed meets a preset condition; if the preset conditions are met, the data to be processed is deleted. This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
本申请实施例还提供另一种数据处理方法,该方法在上述实施例所述方法的基础上实现;该方法重点描述根据时间信息,确定待处理数据是否满足预设条件的具体实现过程, 如图3所示,该方法包括如下步骤:The embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific implementation process of determining whether the data to be processed meets preset conditions based on time information, such as As shown in Figure 3, the method includes the following steps:
步骤S300,获取待处理数据的时间信息;该时间信息用于指示待处理数据到达当前设备的时间。Step S300: Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
步骤S302,计算上述时间信息与当前时间的时间差。Step S302: Calculate the time difference between the above-mentioned time information and the current time.
通常来说,可以从当前设备中或互联网中直接获取到当前时间;例如,如果当前设备为网关服务器,网关服务器通常设置有精确的计时装置,如无线电时钟或原子钟。将当前时间减去上述时间信息,即待处理数据到达当前设备的时间,即可得到上述时间差;该时间差也可以理解为待处理数据在当前设备中存储时间。Generally speaking, the current time can be obtained directly from the current device or the Internet; for example, if the current device is a gateway server, the gateway server is usually provided with an accurate timing device, such as a radio clock or an atomic clock. The time difference is obtained by subtracting the above-mentioned time information from the current time, that is, the time when the data to be processed arrives at the current device; the time difference can also be understood as the storage time of the data to be processed in the current device.
步骤S304,判断上述时间差是否大于或等于预设的时间差阈值;如果是,执行步骤S306;如果否,结束。Step S304: It is judged whether the above-mentioned time difference is greater than or equal to the preset time difference threshold; if so, step S306 is executed; if not, the end is ended.
上述时间差阈值可以与中控平台读取待处理数据的周期相关,也可以与待处理数据更新的周期相关。作为示例,中控平台以预设的第一时间周期读取网关服务器中的监控数据(即上述待处理数据);各个容器集群的数据汇总转发代理设备以预设的第二时间周期(即数据更新周期)将数据采集器采集到的监控数据发送至网关服务器,每个容器集群对应的第二时间周期可能不同。为了使读取到的监控数据均为已经更新过的监控数据,上述第一时间周期可以大于容器集群中最长的第二时间周期。在这种情况下,在上次监控数据读取时刻至本次监控数据读取时刻之间,每个容器集群的数据汇总转发代理设备至少已经发送一次监控数据至网关服务器,即待处理数据至少更新过一次。上述第一时间周期可以根据具体需求进行设置,如设置为上述最长的数据更新周期的若干倍等。The above-mentioned time difference threshold may be related to the cycle of reading the data to be processed by the central control platform, or may be related to the cycle of updating the data to be processed. As an example, the central control platform reads the monitoring data in the gateway server (that is, the data to be processed above) in a preset first time period; the data aggregation forwarding agent device of each container cluster uses the preset second time period (that is, the data Update period) The monitoring data collected by the data collector is sent to the gateway server, and the second time period corresponding to each container cluster may be different. In order that the read monitoring data are all updated monitoring data, the above-mentioned first time period may be greater than the longest second time period in the container cluster. In this case, between the last time the monitoring data is read and the current time when the monitoring data is read, the data aggregation forwarding agent device of each container cluster has sent the monitoring data to the gateway server at least once, that is, the data to be processed is at least Updated once. The foregoing first time period may be set according to specific requirements, such as being set to several times the foregoing longest data update period.
上述时间差阈值可以与中控平台读取监控数据的时间周期相同。在该时间差阈值内,网关服务器中存储的各个容器集群的监控数据,即待处理数据至少更新过一次;当确定某个容器集群的待处理数据在当前设备中的存储时间大于预设的时间差阈值时,说明该待处理数据没有被更新,该待处理数据需要被删除。The above-mentioned time difference threshold may be the same as the time period for the central control platform to read the monitoring data. Within the time difference threshold, the monitoring data of each container cluster stored in the gateway server, that is, the data to be processed has been updated at least once; when it is determined that the storage time of the pending data of a certain container cluster in the current device is greater than the preset time difference threshold If the data to be processed is not updated, the data to be processed needs to be deleted.
上述时间差阈值还可以与上述最长的第二时间周期相同。如果各个容器集群的监控数据转发至网关服务器的过程均正常,在该时间差阈值内,网关服务器中存储的各个容器集群的监控数据,即待处理数据至少更新过一次;当确定某个容器集群的待处理数据在当前设备中的存储时间大于预设的时间差阈值时,说明该待处理数据没有被更新,该待处理数据需要被删除。The aforementioned time difference threshold may also be the same as the aforementioned longest second time period. If the process of forwarding the monitoring data of each container cluster to the gateway server is normal, within the time difference threshold, the monitoring data of each container cluster stored in the gateway server, that is, the data to be processed, has been updated at least once; When the storage time of the data to be processed in the current device is greater than the preset time difference threshold, it indicates that the data to be processed has not been updated and the data to be processed needs to be deleted.
当上述时间差阈值设置为上述最长的第二时间周期与第一时间周期之间的任一时间长度时,如果可以确定待处理数据均不满足预设条件,可以保证中控平台相邻两次读取的监控数据均为更新过的监控数据。如果对获取更新监控数据的时间间隔还有其他要求,如中控平台不能连续三次读取到同一监控数据,可以依据该要求对该时间差阈值进行对应的设置。When the above time difference threshold is set to any length of time between the longest second time period and the first time period, if it can be determined that the data to be processed does not meet the preset conditions, it can be guaranteed that the central control platform is adjacent to each other twice The read monitoring data are all updated monitoring data. If there are other requirements for the time interval for obtaining updated monitoring data, such as the central control platform cannot read the same monitoring data three times in a row, the time difference threshold can be set correspondingly according to the requirements.
步骤S306,删除待处理数据。Step S306: Delete the data to be processed.
上述数据处理方法中,计算待处理数据的时间信息与当前时间的时间差,并将该时间差与预设的时间差阈值进行比较,当该时间差大于时间差阈值时,删除待处理数据;该方式通过删除没有及时更新的待处理数据,保证了监控数据的时效性,同时释放了网关服务器的存储空间。In the above data processing method, the time difference between the time information of the data to be processed and the current time is calculated, and the time difference is compared with a preset time difference threshold. When the time difference is greater than the time difference threshold, the data to be processed is deleted; The timely updated data to be processed ensures the timeliness of the monitoring data and releases the storage space of the gateway server.
本申请实施例还提供另一种数据处理方法,该方法在上述实施例所述方法的基础上实现;该方法重点描述根据待处理数据的时间信息和对应的容器集群的集群状态,判断是否删除该数据的具体实现过程,如图4所示,该方法包括如下步骤:The embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on determining whether to delete based on the time information of the data to be processed and the cluster status of the corresponding container cluster The specific implementation process of the data is shown in Figure 4. The method includes the following steps:
步骤S400,获取待处理数据的时间信息;该时间信息用于指示待处理数据到达当前设备的时间。Step S400: Obtain time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
步骤S402,根据上述时间信息和待处理数据对应的容器集群的集群状态,判断待处理数据是否满足预设条件;如果是,执行步骤S404;如果否,结束。Step S402: According to the above-mentioned time information and the cluster status of the container cluster corresponding to the data to be processed, it is determined whether the data to be processed meets a preset condition; if it is, step S404 is executed; if not, it ends.
上述待处理数据可以为对应的容器集群的监控数据;上述容器集群的集群状态包括创建中(Initial)、运行中(Running)、升级中(Updating)、已删除(Deleted)等。上述预设条件可以根据时间信息及对应的容器集群的集群状态进行设置。例如,预设条件可以划分为第一预设子条件和第二预设子条件;其中,第一预设子条件可以为,根据上述时间信息确定待处理数据的存储时间,该存储时间大于预设的时间差阈值;上述第二预设子条件可以设置为待处理数据对应的集群状态为已删除。The aforementioned data to be processed may be monitoring data of the corresponding container cluster; the cluster status of the aforementioned container cluster includes creating (Initial), running (Running), upgrading (Updating), deleted (Deleted), etc. The aforementioned preset conditions can be set according to the time information and the cluster status of the corresponding container cluster. For example, the preset condition may be divided into a first preset sub-condition and a second preset sub-condition; wherein, the first preset sub-condition may be determining the storage time of the data to be processed according to the above time information, and the storage time is greater than the preset Set the time difference threshold; the above second preset sub-condition can be set as the cluster status corresponding to the data to be processed is deleted.
针对待处理数据,可以分别判断该待处理数据的存储时间是否满足第一子预设条件,得到第一判断结果;判断该待处理数据对应容器集群的集群状态是否满足第二子预设条件,得到第二判断结果;然后根据第一判断结果及第二判断结果,判断待处理数据是否同时满足第一预设子条件及第二预设子条件,如果是,则确认待处理数据满足预设条件。For the data to be processed, it is possible to determine whether the storage time of the data to be processed meets the first sub-preset condition, and obtain the first judgment result; to determine whether the cluster state of the container cluster corresponding to the data to be processed meets the second sub-preset condition, Obtain the second judgment result; then according to the first judgment result and the second judgment result, judge whether the data to be processed meets the first preset sub-condition and the second preset sub-condition at the same time, if yes, confirm that the data to be processed meets the preset condition.
另一种方式中,还可以首先判断该待处理数据的存储时间是否满足第一预设子条件,在满足第一预设子条件的情况下,再判断待处理数据对应容器集群的集群状态是否满足第二预设子条件,如果也满足第二预设子条件,确认该待处理数据满足预设条件。同样地,也可以先判断是否满足第二预设子条件,在满足该条件的情况下,再进行第一预设子条件的判断;如果也满足第一预设子条件,确认该待处理数据满足预设条件。In another way, it is also possible to first determine whether the storage time of the data to be processed meets the first preset sub-condition, and if the first preset sub-condition is satisfied, then determine whether the cluster state of the container cluster corresponding to the data to be processed is The second preset sub-condition is satisfied, and if the second preset sub-condition is also satisfied, it is confirmed that the data to be processed meets the preset condition. Similarly, it is also possible to first determine whether the second preset sub-condition is met, and if the condition is met, then the first preset sub-condition is judged; if the first preset sub-condition is also met, the data to be processed is confirmed Meet the preset conditions.
步骤S404,删除待处理数据。Step S404: Delete the data to be processed.
上述数据处理方法中,获取待处理数据的时间信息,并在时间信息及待处理数据对应的容器集群状态满足预设条件的情况下,清除待处理数据;该方式在删除待处理数据的过程中,考虑待处理数据是否及时更新的同时,考虑了该待处理数据对应的容器集群是否已被删除,保证了待处理数据的时效性。In the above data processing method, the time information of the data to be processed is obtained, and the data to be processed is cleared when the time information and the state of the container cluster corresponding to the data to be processed meet the preset conditions; this method is in the process of deleting the data to be processed , While considering whether the data to be processed is updated in time, consider whether the container cluster corresponding to the data to be processed has been deleted, ensuring the timeliness of the data to be processed.
本申请实施例还提供另一种数据处理方法,该方法在上述实施例所述方法的基础上实现;该方法重点描述了在待处理数据为多个时,根据时间信息和集群状态,判断待处理数据是否满足预设条件的具体实现过程,如图5所示,该方法包括如下步骤:This embodiment of the application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; this method focuses on determining the data to be processed based on time information and cluster status when there are multiple data to be processed The specific implementation process of processing whether the data meets the preset conditions is shown in Figure 5. The method includes the following steps:
步骤S500,获取每个待处理数据的时间信息;该时间信息用于指示待处理数据到达当前设备的时间。Step S500: Obtain time information of each data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
步骤S502,计算每个待处理数据的时间信息与当前时间的时间差,判断时间差是否大于或等于预设的时间差阈值,得到每个待处理数据的判断结果。该判断结果可以包括两种,一种为时间差大于或等于预设的时间差阈值,另一种为时间差小于预设的时间差阈值。Step S502: Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed. The judgment result may include two kinds, one is that the time difference is greater than or equal to the preset time difference threshold, and the other is that the time difference is less than the preset time difference threshold.
步骤S504,获取每个待处理数据对应的容器集群的集群状态。Step S504: Obtain the cluster status of the container cluster corresponding to each data to be processed.
通常情况下,容器集群管理平台的关系型数据库中,存储有每个集群的基本信息,其中包括集群状态;可以向容器集群管理平台发送请求,询问待处理数据对应的容器集群的集群状态;该请求中通常包括所查询的容器集群的标识。在获取到待处理数据对应的容器集群的集群状态后,可以把集群状态保存在当前设备设定的存储空间中。Normally, the relational database of the container cluster management platform stores basic information of each cluster, including the cluster status; you can send a request to the container cluster management platform to inquire about the cluster status of the container cluster corresponding to the data to be processed; The request usually includes the identity of the container cluster in question. After obtaining the cluster state of the container cluster corresponding to the data to be processed, the cluster state can be saved in the storage space set by the current device.
步骤S506,针对每个待处理数据,判断当前待处理数据的判断结果是否为大于或等于预设的时间差阈值,且当前待处理数据对应的容器集群的集群状态为已删除;如果是,执行步骤S508;如果否,结束。Step S506: For each data to be processed, determine whether the judgment result of the current data to be processed is greater than or equal to a preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted; if so, perform step S508; If not, end.
具体而言,可以按照预设的顺序,逐一对每个待处理数据进行下述处理:首先,从上述判断结果中获取当前待处理数据的判断结果;然后再从上述获取到的集群状态中获取当前待处理数据对应的容器集群的集群状态;如果当前待处理数据的判断结果为大于或等于预设的时间差阈值,且当前待处理数据对应的容器集群的集群状态为已删除,则可以确定当前待处理数据满足预设条件,该待处理数据需要删除。Specifically, the following processing can be performed on each data to be processed in a preset order: first, obtain the judgment result of the current data to be processed from the judgment result above; then obtain the result of the cluster state obtained above The cluster status of the container cluster corresponding to the current data to be processed; if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, you can determine the current The data to be processed meets preset conditions, and the data to be processed needs to be deleted.
步骤S508,删除待处理数据。Step S508: Delete the data to be processed.
上述数据处理方法中,获取待处理数据的时间信息及对应的容器集群的集群状态,并在时间信息及容器集群状态满足预设条件的情况下,清除待处理数据;该方式在删除待处理数据的过程中,考虑待处理数据是否及时更新的同时,考虑了该待处理数据对应的容器集群是否已被删除,保证了待处理数据的时效性。In the above data processing method, the time information of the data to be processed and the cluster status of the corresponding container cluster are obtained, and the data to be processed is cleared when the time information and the container cluster status meet the preset conditions; this method is deleting the data to be processed In the process, while considering whether the data to be processed is updated in time, it is also considered whether the container cluster corresponding to the data to be processed has been deleted, which ensures the timeliness of the data to be processed.
本申请实施例还提供另一种数据处理方法,该方法在上述实施例所述方法的基础上实现;该方法重点描述了在待处理数据为多个时,根据时间信息和集群状态,判断待处理数据是否满足预设条件的具体实现过程,如图6所示,该方法包括如下步骤:This embodiment of the application also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; this method focuses on determining the data to be processed based on time information and cluster status when there are multiple data to be processed The specific implementation process of processing whether the data meets the preset conditions is shown in Figure 6. The method includes the following steps:
步骤S600,获取每个待处理数据的时间信息;该时间信息用于指示待处理数据到达当前设备的时间。Step S600: Obtain time information of each data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device.
步骤S602,计算每个待处理数据的时间信息与当前时间的时间差,判断时间差是否大于或等于预设的时间差阈值,得到每个待处理数据的判断结果。Step S602: Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed.
步骤S604,将判断结果为时间差大于或等于预设的时间差阈值的待处理数据确定为待定数据。Step S604: Determine the to-be-processed data whose time difference is greater than or equal to a preset time difference threshold as a result of the judgment as pending data.
步骤S606,获取每个待定数据对应的容器集群的集群状态。Step S606: Obtain the cluster status of the container cluster corresponding to each pending data.
该待定数据是基于上述步骤S602的判断,从上述待处理数据中筛选出的一部分数据,通常,该待定数据的数据量不大于待处理数据。待定数据确定后,即可向容器集群管理平台发送请求,查询各个待定数据对应的容器集群的集群状态对应的集群状态,接收该平台返回的信息,得到各个待定数据对应的容器集群的集群状态。The undetermined data is a part of the data selected from the above-mentioned to-be-processed data based on the judgment in step S602. Generally, the amount of the undetermined data is not greater than the to-be-processed data. After the pending data is determined, a request can be sent to the container cluster management platform to query the cluster status corresponding to the cluster status of the container cluster corresponding to each pending data, and receive the information returned by the platform to obtain the cluster status of the container cluster corresponding to each pending data.
步骤S608,针对每个待定数据,判断当前待定数据对应的容器集群的集群状态是否是已删除;如果是,执行步骤610;如果否,结束。Step S608: For each pending data, it is determined whether the cluster state of the container cluster corresponding to the current pending data is deleted; if so, step 610 is executed; if not, the end is ended.
步骤S610,删除待处理数据。Step S610: Delete the data to be processed.
上述数据处理方式中,在确定待处理数据为长时间未更新的数据后,获取其对应的容器集群的集群状态,在确定该集群状态为已删除后,删除该待处理数据;该方式仅获取未及时更新的待处理数据对应的容器集群的集群状态,以删除未及时更新且集群状态为已删 除的待处理数据,在保证监控数据的时效性的基础上,减少了对集训状态的查询次数,节约计算资源,效率较高。In the above data processing method, after the data to be processed is determined to be data that has not been updated for a long time, the cluster status of the corresponding container cluster is acquired, and the data to be processed is deleted after the cluster status is determined to be deleted; this method only acquires The cluster status of the container cluster corresponding to the pending data that is not updated in time is used to delete pending data that is not updated in time and the cluster status is deleted. On the basis of ensuring the timeliness of the monitoring data, the number of queries to the training state is reduced. , It saves computing resources and is more efficient.
本申请实施例还提供了另一种数据处理方法,该方法在上述实施例所述方法的基础上实现;该方法应用于多容器集群的监控数据接收适配过程。The embodiment of the present application also provides another data processing method, which is implemented on the basis of the method described in the foregoing embodiment; the method is applied to the monitoring data reception and adaptation process of a multi-container cluster.
多容器集群的监控数据接收过程示意图如图7所示,数据汇总转发代理设备(也可以称为采集代理)从数据采集器(也可以称为采集对象)中获取容器集群的监控数据,将该监控数据发送至网关服务器;网关服务器接收该监控数据,并将该监控数据存入数据队列,继而将数据存储至网关服务器的内存中;在接收到属于某个容器集群的监控数据后,网关服务器将内存中该容器集群的历史监控数据更新为当前的监控数据;中控平台(也可称为监控平台)经过数据检索后读取网关服务器内存中的监控数据;上述实施例提供的数据处理方法主要应用于网关服务器,在数据存储和数据检索步骤之间实现了对网关服务器内存的数据清理功能。A schematic diagram of the monitoring data receiving process of a multi-container cluster is shown in Figure 7. The data aggregation forwarding agent device (also called a collection agent) obtains the monitoring data of the container cluster from the data collector (also called the collection object), and The monitoring data is sent to the gateway server; the gateway server receives the monitoring data, stores the monitoring data in the data queue, and then stores the data in the memory of the gateway server; after receiving the monitoring data belonging to a container cluster, the gateway server The historical monitoring data of the container cluster in the memory is updated to the current monitoring data; the central control platform (also referred to as the monitoring platform) reads the monitoring data in the gateway server memory after data retrieval; the data processing method provided in the above embodiment Mainly applied to the gateway server, the data cleaning function of the gateway server memory is realized between the data storage and data retrieval steps.
在仅考虑到监控数据在网关服务器中的存储时间时,该方法包括以下步骤:When only considering the storage time of the monitoring data in the gateway server, the method includes the following steps:
步骤1,获取已存储的每条监控数据的时间戳;其中,时间戳是网关服务器接收到数据的时间。Step 1. Obtain the time stamp of each stored monitoring data; where the time stamp is the time when the gateway server receives the data.
步骤2,获取当前时间,并将当前时间和监控数据的时间戳做差值计算,得到时间差。Step 2: Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
步骤3,将时间差和预设阈值进行比对。Step 3. Compare the time difference with the preset threshold.
步骤4,基于比对结果判定每条监控数据是否为无效数据,即如果时间差大于预设阈值,则判定该监控数据为无效数据。Step 4. Determine whether each piece of monitoring data is invalid data based on the comparison result, that is, if the time difference is greater than a preset threshold, then determine that the monitoring data is invalid data.
步骤5,清除无效数据。Step 5. Clear invalid data.
采用上述方法可以定期处理掉一些无效数据,然而,也可能会误将一些监控数据作为无效数据删掉;例如,当容器集群和网关服务器由于一些问题连接中断时,存储在网关中的与该集群对应的监控数据无法更新,在进行无效数据的判定过程中,会将该监控数据判定为无效数据,并删除。然而该监控数据对应的容器集群没有被删除,当该容器集群和网关服务器的网络连接恢复之后,该容器集群对应的监控数据可以继续更新。因此,在考虑到该监控数据对应的容器集群的集群状态,对该方法进行改进后,该方法可以通过以下步骤实现:The above method can be used to process some invalid data regularly. However, some monitoring data may be deleted as invalid data by mistake; for example, when the connection between the container cluster and the gateway server is interrupted due to some problems, the data stored in the gateway and the cluster The corresponding monitoring data cannot be updated. In the process of determining invalid data, the monitoring data will be determined as invalid data and deleted. However, the container cluster corresponding to the monitoring data is not deleted. When the network connection between the container cluster and the gateway server is restored, the monitoring data corresponding to the container cluster can continue to be updated. Therefore, after considering the cluster status of the container cluster corresponding to the monitoring data, after improving the method, the method can be implemented through the following steps:
步骤1,获取每条监控数据的时间戳。Step 1. Obtain the timestamp of each monitored data.
步骤2,获取当前时间,并将当前时间和该监控数据的时间戳做差值计算,得到时间差。Step 2: Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
步骤3,将时间差和预设阈值进行比对,得到比对结果。Step 3. Compare the time difference with the preset threshold to obtain the comparison result.
步骤4,获取每个监控数据对应的容器集群的集群状态。Step 4. Obtain the cluster status of the container cluster corresponding to each monitoring data.
步骤5,基于比对结果和每个监控数据对应的容器集群的集群状态,判定每条数据是否为无效数据;具体地,如果比对结果时间差为大于预设阈值,且该监控数据对应的容器集群的集群状态为已删除状态,则判定该数据为无效数据。Step 5. Determine whether each piece of data is invalid data based on the comparison result and the cluster status of the container cluster corresponding to each monitoring data; specifically, if the comparison result time difference is greater than a preset threshold, and the monitoring data corresponds to the container If the cluster status of the cluster is deleted, the data is determined to be invalid data.
步骤6,清除无效数据。Step 6. Clear invalid data.
在一种可能的实施方式中,为了节约计算资源,可以先通过时间戳的判断缩小获取集 群的状态的数据量,再查询对应的集群状态;此时,该方法可以通过以下步骤实现:In a possible implementation manner, in order to save computing resources, the amount of data for obtaining the status of the cluster can be reduced through the judgment of the timestamp, and then the corresponding cluster status can be queried; at this time, the method can be implemented through the following steps:
步骤1,获取每条监控数据的时间戳。Step 1. Obtain the timestamp of each monitored data.
步骤2,获取当前时间,并将当前时间和该监控数据的时间戳做差值计算,得到时间差。Step 2: Obtain the current time, and calculate the difference between the current time and the time stamp of the monitoring data to obtain the time difference.
步骤3,将时间差和预设阈值进行比对,得到比对结果;若比对结果为时间差大于预设阈值,则判定该监控数据为待定无效数据。Step 3: The time difference is compared with the preset threshold to obtain the comparison result; if the comparison result is that the time difference is greater than the preset threshold, the monitoring data is determined to be invalid data to be determined.
步骤4,获取每个待定无效数据对应的容器集群的集群状态。Step 4. Obtain the cluster status of the container cluster corresponding to each pending invalid data.
步骤5,基于每条待定无效数据对应的容器集群的集群状态判定该待定无效数据是否为无效数据;若集群状态为已删除状态,则判定该待定无效数据为无效数据。Step 5: Determine whether the pending invalid data is invalid data based on the cluster status of the container cluster corresponding to each piece of invalid data to be determined; if the cluster status is a deleted state, determine whether the invalid data to be determined is invalid data.
步骤6,清除无效数据。Step 6. Clear invalid data.
上述方法,根据监控数据的时间戳与当前时间的时间差,以及该监控数据对应的容器集群的集群状态,判定该监控数据是否为无效数据,并删除了无效数据,释放了网关服务器的存储空间,提高了网关存储空间的利用率。In the above method, according to the time difference between the time stamp of the monitoring data and the current time, and the cluster status of the container cluster corresponding to the monitoring data, determine whether the monitoring data is invalid data, delete the invalid data, and release the storage space of the gateway server. Improved the utilization of gateway storage space.
对应于上述数据处理方法实施例,本申请实施例提供了一种数据处理装置,如图8所述,该装置包括:Corresponding to the foregoing data processing method embodiment, an embodiment of the present application provides a data processing device. As shown in FIG. 8, the device includes:
信息获取模块800,设置为获取待处理数据的时间信息;时间信息用于指示待处理数据到达当前设备的时间;The information acquisition module 800 is configured to acquire time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device;
条件确定模块802,设置为根据时间信息,确定待处理数据是否满足预设条件;The condition determining module 802 is configured to determine whether the data to be processed meets a preset condition according to the time information;
数据删除模块804,设置为如果满足预设条件,删除待处理数据。The data deletion module 804 is configured to delete the data to be processed if the preset conditions are met.
上述一种数据处理装置,首先获取待处理数据的时间信息;进而根据该时间信息,确定待处理数据是否满足预设条件;如果满足预设条件,则删除待处理数据。该方式通过时间信息清理无效数据,保证了监控数据的时效性,同时提高了网关存储空间的利用率。The above-mentioned data processing device first obtains time information of the data to be processed; and then determines whether the data to be processed meets a preset condition according to the time information; if the preset conditions are met, the data to be processed is deleted. This method cleans up invalid data through time information, ensures the timeliness of monitoring data, and improves the utilization of gateway storage space.
在一种可能的实施方式中,上述条件确定模块包括:时间差计算单元,设置为计算时间信息与当前时间的时间差;第一阈值判断单元,设置为如果时间差大于或等于预设的时间差阈值,则确定满足预设条件;如果所述时间差小于预设的时间差阈值,则确定不满足预设条件。In a possible implementation manner, the above-mentioned condition determination module includes: a time difference calculation unit configured to calculate the time difference between the time information and the current time; a first threshold judgment unit configured to: if the time difference is greater than or equal to a preset time difference threshold, then It is determined that the preset condition is satisfied; if the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.
在一种可能的实施方式中,上述条件确定模块包括:第一条件确定单元,设置为根据时间信息和待处理数据对应的容器集群的集群状态,确定待处理数据是否满足预设条件。In a possible implementation manner, the above-mentioned condition determination module includes: a first condition determination unit configured to determine whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed.
当上述待处理数据包括多个时,上述第一条件确定单元设置为:计算每个待处理数据的时间信息与当前时间的时间差,判断时间差是否大于或等于预设的时间差阈值,得到每个待处理数据的判断结果;获取每个待处理数据对应的容器集群的集群状态;针对每个待处理数据,如果当前待处理数据的判断结果为大于或等于预设的时间差阈值,且当前待处理数据对应的容器集群的集群状态为已删除,确定当前待处理数据满足预设条件。When the data to be processed includes multiple data, the first condition determining unit is configured to: calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain each The judgment result of the processed data; obtain the cluster status of the container cluster corresponding to each data to be processed; for each data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the current data to be processed The cluster status of the corresponding container cluster is deleted, and it is determined that the current to-be-processed data meets the preset conditions.
当上述待处理数据包括多个时,上述第一条件确定单元设置为:计算每个待处理数据的时间信息与当前时间的时间差,判断时间差是否大于或等于预设的时间差阈值,得到每个待处理数据的判断结果;将判断结果为时间差大于或等于预设的时间差阈值的待处理数据确定为待定数据;获取每个待定数据对应的容器集群的集群状态;针对每个待定数据, 如果当前待定数据对应的容器集群的集群状态是已删除,则确定当前待定数据满足预设条件。When the data to be processed includes multiple data, the first condition determining unit is configured to: calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain each The judgment result of the processed data; the judgment result is that the time difference is greater than or equal to the preset time difference threshold to determine the pending data as pending data; the cluster status of the container cluster corresponding to each pending data is obtained; for each pending data, if the current pending If the cluster status of the container cluster corresponding to the data is deleted, it is determined that the current pending data meets the preset condition.
本申请实施例所提供的数据处理装置,其实现原理及产生的技术效果和前述数据处理方法实施例相同,为简要描述,数据处理装置实施例部分未提及之处,可参考前述数据处理方法实施例中相应内容。The implementation principles and technical effects of the data processing device provided in the embodiments of this application are the same as those of the foregoing data processing method embodiments. For a brief description, the data processing device embodiments that are not mentioned in the data processing device embodiments may refer to the foregoing data processing method. Corresponding content in the embodiment.
本申请实施例还提供了一种网关服务器,参见图9所示,该网关服务器包括处理器130和存储器131,该存储器131存储有能够被处理器130执行的机器可执行指令,该处理器130执行机器可执行指令以实现上述数据处理方法。An embodiment of the present application also provides a gateway server. As shown in FIG. 9, the gateway server includes a processor 130 and a memory 131. The memory 131 stores machine executable instructions that can be executed by the processor 130. The machine executable instructions are executed to implement the above-mentioned data processing method.
在一种可能的实施方式中,图9所示的网关服务器还包括总线132和通信接口133,处理器130、通信接口133和存储器131通过总线132连接。In a possible implementation manner, the gateway server shown in FIG. 9 further includes a bus 132 and a communication interface 133, and the processor 130, the communication interface 133, and the memory 131 are connected through the bus 132.
其中,存储器131可能包含高速随机存取存储器(RAM,Random Access Memory),也可能还包括非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。通过至少一个通信接口133(可以是有线或者无线)实现该系统网元与至少一个其他网元之间的通信连接,可以使用互联网,广域网,本地网,城域网等。总线132可以是ISA总线、PCI总线或EISA总线等。所述总线可以分为地址总线、数据总线、控制总线等。为便于表示,图9中仅用一个双向箭头表示,但并不表示仅有一根总线或一种类型的总线。The memory 131 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, PCI bus, EISA bus, or the like. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one bidirectional arrow is used in FIG. 9, but it does not mean that there is only one bus or one type of bus.
处理器130可能是一种集成电路芯片,具有信号的处理能力。在实现过程中,上述方法的各步骤可以通过处理器130中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器130可以是通用处理器,包括中央处理器(Central Processing Unit,简称CPU)、网络处理器(Network Processor,简称NP)等;还可以是数字信号处理器(Digital Signal Processing,简称DSP)、专用集成电路(Application Specific Integrated Circuit,简称ASIC)、现成可编程门阵列(Field-Programmable Gate Array,简称FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成,或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器,闪存、只读存储器,可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器131,处理器130读取存储器131中的信息,结合其硬件完成前述实施例的方法的步骤。The processor 130 may be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the foregoing method may be completed by an integrated logic circuit of hardware in the processor 130 or instructions in the form of software. The aforementioned processor 130 may be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processing, DSP for short). ), Application Specific Integrated Circuit (ASIC for short), Field-Programmable Gate Array (FPGA for short) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131, and completes the steps of the method of the foregoing embodiment in combination with its hardware.
本申请实施例还提供了一种机器可读存储介质,该机器可读存储介质存储有机器可执行指令,该机器可执行指令在被处理器调用和执行时,该机器可执行指令促使处理器实现上述数据处理方法,具体实现可参见方法实施例,在此不再赘述。The embodiment of the present application also provides a machine-readable storage medium that stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor The above-mentioned data processing method is implemented. For specific implementation, please refer to the method embodiment, which will not be repeated here.
本申请实施例所提供的数据处理方法及装置和网关服务器的计算机程序产品,包括存储了程序代码的计算机可读存储介质,所述程序代码包括的指令可用于执行前面方法实施例中所述的方法,具体实现可参见方法实施例,在此不再赘述。The data processing method and device and the computer program product of the gateway server provided by the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For the specific implementation of the method, please refer to the method embodiment, which will not be repeated here.
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个计算机可读取存储介质中。基于这样的理解,本申请的技术方案本质上或者说对相 关技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可以是个人计算机,服务器,或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括:U盘、移动硬盘、只读存储器(ROM,Read-Only Memory)、随机存取存储器(RAM,Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of this application essentially or the part that contributes to the related technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several The instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code .
最后应说明的是:以上所述实施例,仅为本申请的具体实施方式,用以说明本申请的技术方案,而非对其限制,本申请的保护范围并不局限于此,尽管参照前述实施例对本申请进行了详细的说明,本领域的普通技术人员应当理解:任何熟悉本技术领域的技术人员在本申请揭露的技术范围内,其依然可以对前述实施例所记载的技术方案进行修改或可轻易想到变化,或者对其中部分技术特征进行等同替换;而这些修改、变化或者替换,并不使相应技术方案的本质脱离本申请实施例技术方案的精神和范围,都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应所述以权利要求的保护范围为准。Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the application, which are used to illustrate the technical solutions of the application, rather than limit it. The scope of protection of the application is not limited thereto, although referring to the foregoing The examples describe the application in detail, and those of ordinary skill in the art should understand that any person skilled in the art can still modify the technical solutions described in the foregoing examples within the technical scope disclosed in this application. Or it is easy to think of changes, or equivalent replacements of some of the technical features; and these modifications, changes or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be covered in this application Within the scope of protection. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.
Claims (12)
- 一种数据处理方法,所述方法包括:A data processing method, the method includes:获取待处理数据的时间信息;所述时间信息用于指示所述待处理数据到达当前设备的时间;Acquiring time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device;根据所述时间信息,确定所述待处理数据是否满足预设条件;According to the time information, determine whether the data to be processed meets a preset condition;如果满足预设条件,删除所述待处理数据。If the preset condition is met, delete the to-be-processed data.
- 根据权利要求1所述的方法,其中,根据所述时间信息,确定所述待处理数据是否满足预设条件的步骤,包括:The method according to claim 1, wherein the step of determining whether the data to be processed meets a preset condition according to the time information comprises:计算所述时间信息与当前时间的时间差;Calculating the time difference between the time information and the current time;如果所述时间差大于或等于预设的时间差阈值,则确定满足预设条件;If the time difference is greater than or equal to the preset time difference threshold, it is determined that the preset condition is met;如果所述时间差小于预设的时间差阈值,则确定不满足预设条件。If the time difference is less than the preset time difference threshold, it is determined that the preset condition is not satisfied.
- 根据权利要求1所述的方法,其中,根据所述时间信息,确定所述待处理数据是否满足预设条件的步骤,包括:The method according to claim 1, wherein the step of determining whether the data to be processed meets a preset condition according to the time information comprises:根据所述时间信息和所述待处理数据对应的容器集群的集群状态,确定所述待处理数据是否满足预设条件。According to the time information and the cluster state of the container cluster corresponding to the data to be processed, it is determined whether the data to be processed meets a preset condition.
- 根据权利要求3所述的方法,其中,所述待处理数据包括多个;The method according to claim 3, wherein the data to be processed includes a plurality of;根据所述时间信息和所述待处理数据对应的容器集群的集群状态,确定所述待处理数据是否满足预设条件的步骤,包括:The step of determining whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed includes:计算每个所述待处理数据的时间信息与当前时间的时间差,判断所述时间差是否大于或等于预设的时间差阈值,得到每个所述待处理数据的判断结果;Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed;获取每个所述待处理数据对应的容器集群的集群状态;Acquiring the cluster status of the container cluster corresponding to each of the data to be processed;针对每个所述待处理数据,如果当前待处理数据的判断结果为大于或等于预设的时间差阈值,且所述当前待处理数据对应的容器集群的集群状态为已删除,确定所述当前待处理数据满足预设条件。For each of the data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, determine the current data to be processed The processed data meets preset conditions.
- 根据权利要求3所述的方法,其中,所述待处理数据包括多个;The method according to claim 3, wherein the data to be processed includes a plurality of;根据所述时间信息和所述待处理数据对应的容器集群的集群状态,确定所述待处理数据是否满足预设条件的步骤,包括:The step of determining whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed includes:计算每个所述待处理数据的时间信息与当前时间的时间差,判断所述时间差是否大于或等于预设的时间差阈值,得到每个所述待处理数据的判断结果;Calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to a preset time difference threshold, and obtaining a judgment result of each data to be processed;将判断结果为时间差大于或等于预设的时间差阈值的待处理数据确定为待定数据;Determine the data to be processed whose time difference is greater than or equal to the preset time difference threshold as the undetermined data;获取每个所述待定数据对应的容器集群的集群状态;Acquiring the cluster state of the container cluster corresponding to each of the pending data;针对每个所述待定数据,如果确定当前待定数据对应的容器集群的集群状态是已删除,则确定所述当前待定数据满足预设条件。For each of the pending data, if it is determined that the cluster state of the container cluster corresponding to the current pending data is deleted, it is determined that the current pending data meets a preset condition.
- 一种数据处理装置,所述装置包括:A data processing device, the device comprising:信息获取模块,设置为获取待处理数据的时间信息;所述时间信息用于指示所述待处理数据到达当前设备的时间;The information acquisition module is configured to acquire time information of the data to be processed; the time information is used to indicate the time when the data to be processed arrives at the current device;条件确定模块,设置为根据所述时间信息,确定所述待处理数据是否满足预设条 件;The condition determining module is configured to determine whether the data to be processed meets preset conditions according to the time information;数据删除模块,设置为如果满足预设条件,删除所述待处理数据。The data deletion module is configured to delete the to-be-processed data if the preset conditions are met.
- 根据权利要求6所述的装置,其中,所述条件确定模块包括:The apparatus according to claim 6, wherein the condition determination module comprises:时间差计算单元,设置为计算所述时间信息与当前时间的时间差;A time difference calculation unit, configured to calculate the time difference between the time information and the current time;第一阈值判断单元,设置为如果所述时间差大于或等于预设的时间差阈值,则确定满足预设条件;如果所述时间差小于预设的时间差阈值,则确定不满足预设条件。The first threshold determination unit is configured to determine that the preset condition is satisfied if the time difference is greater than or equal to the preset time difference threshold; if the time difference is less than the preset time difference threshold, then it is determined that the preset condition is not satisfied.
- 根据权利要求6所述的装置,其特征在于,所述条件确定模块包括:The device according to claim 6, wherein the condition determining module comprises:第一条件确定单元,设置为根据所述时间信息和所述待处理数据对应的容器集群的集群状态,确定所述待处理数据是否满足预设条件。The first condition determining unit is configured to determine whether the data to be processed meets a preset condition according to the time information and the cluster state of the container cluster corresponding to the data to be processed.
- 根据权利要求8所述的装置,其中,所述待处理数据包括多个;The device according to claim 8, wherein the data to be processed includes a plurality of;所述第一条件确定单元设置为:The first condition determining unit is set to:计算每个所述待处理数据的时间信息与当前时间的时间差,判断所述时间差是否大于或等于预设的时间差阈值,得到每个所述待处理数据的判断结果;Calculating the time difference between the time information of each data to be processed and the current time, judging whether the time difference is greater than or equal to a preset time difference threshold, and obtaining a judgment result of each data to be processed;获取每个所述待处理数据对应的容器集群的集群状态;Acquiring the cluster status of the container cluster corresponding to each of the data to be processed;针对每个所述待处理数据,如果当前待处理数据的判断结果为大于或等于预设的时间差阈值,且所述当前待处理数据对应的容器集群的集群状态为已删除,确定所述当前待处理数据满足预设条件。For each of the data to be processed, if the judgment result of the current data to be processed is greater than or equal to the preset time difference threshold, and the cluster status of the container cluster corresponding to the current data to be processed is deleted, determine the current data to be processed The processed data meets preset conditions.
- 根据权利要求8所述的装置,其中,所述待处理数据包括多个;The device according to claim 8, wherein the data to be processed includes a plurality of;所述第一条件确定单元设置为:The first condition determining unit is set to:计算每个所述待处理数据的时间信息与当前时间的时间差,判断所述时间差是否大于或等于预设的时间差阈值,得到每个所述待处理数据的判断结果;Calculate the time difference between the time information of each data to be processed and the current time, determine whether the time difference is greater than or equal to a preset time difference threshold, and obtain a judgment result of each data to be processed;将判断结果为时间差大于或等于预设的时间差阈值的待处理数据确定为待定数据;Determine the data to be processed whose time difference is greater than or equal to the preset time difference threshold as the undetermined data;获取每个所述待定数据对应的容器集群的集群状态;Acquiring the cluster state of the container cluster corresponding to each of the pending data;针对每个所述待定数据,如果当前待定数据对应的容器集群的集群状态是已删除,则确定所述当前待定数据满足预设条件。For each of the pending data, if the cluster state of the container cluster corresponding to the current pending data is deleted, it is determined that the current pending data meets a preset condition.
- 一种网关服务器,包括处理器和存储器,所述存储器存储有能够被所述处理器执行的机器可执行指令,所述处理器执行所述机器可执行指令以实现权利要求1至5任一项所述的数据处理方法。A gateway server, comprising a processor and a memory, the memory storing machine executable instructions that can be executed by the processor, and the processor executing the machine executable instructions to implement any one of claims 1 to 5 The data processing method.
- 一种机器可读存储介质,所述机器可读存储介质存储有机器可执行指令,所述机器可执行指令在被处理器调用和执行时,机器可执行指令促使处理器实现权利要求1至5任一项所述的数据处理方法。A machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement claims 1 to 5 Any of the data processing methods.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910706918.0 | 2019-07-31 | ||
CN201910706918.0A CN110417901B (en) | 2019-07-31 | 2019-07-31 | Data processing method and device and gateway server |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021017884A1 true WO2021017884A1 (en) | 2021-02-04 |
Family
ID=68365267
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/102520 WO2021017884A1 (en) | 2019-07-31 | 2020-07-17 | Data processing method and apparatus, and gateway server |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN110417901B (en) |
WO (1) | WO2021017884A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113051131A (en) * | 2021-03-23 | 2021-06-29 | 北京沃东天骏信息技术有限公司 | Acquisition terminal, management control platform, and Prometheus service adjusting method and system |
CN113485896A (en) * | 2021-07-22 | 2021-10-08 | 京东方科技集团股份有限公司 | Container state monitoring method, device, system and medium |
CN113504995A (en) * | 2021-07-27 | 2021-10-15 | 平安普惠企业管理有限公司 | Batch data processing method and device, computer equipment and storage medium |
CN115982587A (en) * | 2023-01-09 | 2023-04-18 | 重庆邮电大学 | Gas concentration prediction method and device based on self-adaptive nearest neighbor model |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110417901B (en) * | 2019-07-31 | 2022-04-29 | 北京金山云网络技术有限公司 | Data processing method and device and gateway server |
CN110856040B (en) * | 2019-11-07 | 2021-10-26 | 网易(杭州)网络有限公司 | Method and device for processing monitoring data in client |
CN111064781A (en) * | 2019-12-10 | 2020-04-24 | 北京金山云网络技术有限公司 | Multi-container cluster monitoring data acquisition method and device and electronic equipment |
CN112511339B (en) * | 2020-11-09 | 2023-04-07 | 宝付网络科技(上海)有限公司 | Container monitoring alarm method, system, equipment and storage medium based on multiple clusters |
CN113010797B (en) * | 2021-04-15 | 2022-04-12 | 贵州华泰智远大数据服务有限公司 | Smart city data sharing method and system based on cloud platform |
CN113792058B (en) * | 2021-08-09 | 2023-10-24 | 北京达佳互联信息技术有限公司 | Index data processing method and device, electronic equipment and storage medium |
CN114844794B (en) * | 2022-03-25 | 2024-06-04 | 之江实验室 | Container-oriented resource monitoring method, system and storage medium |
CN115080617B (en) * | 2022-06-14 | 2024-04-12 | 北京智慧星光信息技术有限公司 | Gateway flow control-based elastic search query method and system |
CN118041701B (en) * | 2024-04-12 | 2024-08-13 | 华信咨询设计研究院有限公司 | Directional acquisition and early warning method and system based on container cloud micro-service access flow |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107172171A (en) * | 2017-05-27 | 2017-09-15 | 腾讯科技(深圳)有限公司 | A kind of service request processing method, device and computer-readable recording medium |
CN108287890A (en) * | 2018-01-17 | 2018-07-17 | 中国科学院寒区旱区环境与工程研究所 | Data managing method and device |
US20180218058A1 (en) * | 2015-09-24 | 2018-08-02 | Alibaba Group Holding Limited | Data synchronization method and system |
CN109213604A (en) * | 2018-10-15 | 2019-01-15 | 北京奇艺世纪科技有限公司 | A kind of management method and device of data source |
CN109521957A (en) * | 2018-10-22 | 2019-03-26 | 郑州云海信息技术有限公司 | A kind of data processing method and device |
CN110417901A (en) * | 2019-07-31 | 2019-11-05 | 北京金山云网络技术有限公司 | Data processing method, device and gateway server |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2067104A1 (en) * | 2006-09-28 | 2009-06-10 | Xeround Systems Ltd. | Apparatus and method for a distributed storage global database |
CN101442443B (en) * | 2009-01-13 | 2011-01-12 | 武汉烽火网络有限责任公司 | Network management system and implementing method for SNMP traversing NAT |
CN102790809B (en) * | 2011-05-16 | 2016-09-28 | 北京奇虎科技有限公司 | Domain name system resolution, device and client |
CN102752288B (en) * | 2012-06-06 | 2015-07-08 | 华为技术有限公司 | Method and device for identifying network access action |
CN105991451B (en) * | 2015-03-09 | 2018-12-25 | 杭州迪普科技股份有限公司 | LSA delet method and device based on OSPF |
CN106657249B (en) * | 2016-10-25 | 2019-07-09 | 杭州迪普科技股份有限公司 | Update the method and device of cache resources |
CN109343965A (en) * | 2018-10-31 | 2019-02-15 | 北京金山云网络技术有限公司 | Resource adjusting method, device, cloud platform and server |
CN110011909A (en) * | 2019-02-28 | 2019-07-12 | 北京龙腾融智信息技术有限公司 | Store gateway and storage gateway data sending, receiving method and device |
-
2019
- 2019-07-31 CN CN201910706918.0A patent/CN110417901B/en active Active
-
2020
- 2020-07-17 WO PCT/CN2020/102520 patent/WO2021017884A1/en active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180218058A1 (en) * | 2015-09-24 | 2018-08-02 | Alibaba Group Holding Limited | Data synchronization method and system |
CN107172171A (en) * | 2017-05-27 | 2017-09-15 | 腾讯科技(深圳)有限公司 | A kind of service request processing method, device and computer-readable recording medium |
CN108287890A (en) * | 2018-01-17 | 2018-07-17 | 中国科学院寒区旱区环境与工程研究所 | Data managing method and device |
CN109213604A (en) * | 2018-10-15 | 2019-01-15 | 北京奇艺世纪科技有限公司 | A kind of management method and device of data source |
CN109521957A (en) * | 2018-10-22 | 2019-03-26 | 郑州云海信息技术有限公司 | A kind of data processing method and device |
CN110417901A (en) * | 2019-07-31 | 2019-11-05 | 北京金山云网络技术有限公司 | Data processing method, device and gateway server |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113051131A (en) * | 2021-03-23 | 2021-06-29 | 北京沃东天骏信息技术有限公司 | Acquisition terminal, management control platform, and Prometheus service adjusting method and system |
CN113485896A (en) * | 2021-07-22 | 2021-10-08 | 京东方科技集团股份有限公司 | Container state monitoring method, device, system and medium |
CN113504995A (en) * | 2021-07-27 | 2021-10-15 | 平安普惠企业管理有限公司 | Batch data processing method and device, computer equipment and storage medium |
CN115982587A (en) * | 2023-01-09 | 2023-04-18 | 重庆邮电大学 | Gas concentration prediction method and device based on self-adaptive nearest neighbor model |
Also Published As
Publication number | Publication date |
---|---|
CN110417901B (en) | 2022-04-29 |
CN110417901A (en) | 2019-11-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021017884A1 (en) | Data processing method and apparatus, and gateway server | |
WO2018032936A1 (en) | Method and device for checking domain name generated by domain generation algorithm | |
WO2021068488A1 (en) | Blockchain-based log processing method and apparatus, computer device, and storage medium | |
WO2015135355A1 (en) | Method, apparatus and device for predicting bus running | |
US11778055B2 (en) | Systems and methods for storage-efficient sensors | |
US9122717B2 (en) | System and method of detecting cache inconsistencies | |
CN106921665B (en) | Message processing method and network equipment | |
CN109558065B (en) | Data deleting method and distributed storage system | |
CN111585837B (en) | Internet of things data link monitoring method and device, computer equipment and storage medium | |
CN109271363B (en) | File storage method and device | |
WO2020143181A1 (en) | Data storage method, apparatus, computer device and storage medium | |
WO2019127895A1 (en) | Data stream guiding method, server, and system | |
WO2017152765A1 (en) | Automatic fusing-based message sending method, device and system | |
US11347646B2 (en) | Method, device and computer program product for managing cache based on matching API | |
CN115499187A (en) | API safety monitoring model training method, monitoring method, device and equipment | |
US9641600B2 (en) | System, method, computer-readable medium and apparatus | |
CN113656178B (en) | Data processing method, device, equipment and readable storage medium | |
WO2020083023A1 (en) | Event flow processing method, electronic device, and readable storage medium | |
WO2020000742A1 (en) | Deduplication traffic recording method, device, server and storage medium | |
CN113641676A (en) | Time sequence processing method and device | |
US20240022507A1 (en) | Information flow recognition method, network chip, and network device | |
CN105893150B (en) | Interface calling frequency control method and device and interface calling request processing method and device | |
CN106599005B (en) | Data archiving method and device | |
CN111291127B (en) | Data synchronization method, device, server and storage medium | |
JP6432377B2 (en) | Message log removing apparatus, message log removing method, and message log removing program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20847033 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20847033 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20847033 Country of ref document: EP Kind code of ref document: A1 |