CN114090644B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN114090644B CN114090644B CN202210064845.1A CN202210064845A CN114090644B CN 114090644 B CN114090644 B CN 114090644B CN 202210064845 A CN202210064845 A CN 202210064845A CN 114090644 B CN114090644 B CN 114090644B
- Authority
- CN
- China
- Prior art keywords
- operation data
- redis
- client device
- terminal operation
- management platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Telephonic Communication Services (AREA)
Abstract
The invention discloses a data processing method and a device, wherein a client device collects terminal operation data of a remote dictionary service Redis on the client device and sends the terminal operation data and a device identifier to a cloud management platform of the Redis; the cloud management platform obtains the server operation data and the cluster identification from the Redis cluster, counts index values of operation index items in the terminal operation data and the server operation data respectively, obtains a statistical result, and outputs processing information corresponding to the statistical result according to a predefined processing mode. The cloud management platform can obtain terminal operation data of the client device and cluster operation data of the Redis cluster, and realize functions of statistical analysis, monitoring alarm and the like of the Redis operation data of the client device and the Redis cluster, so that device monitoring of the client device and the Redis cluster is realized, timely positioning of fault events or abnormal events is realized, problem sources are determined, problem solving efficiency is improved, and operation efficiency and service quality of the Redis are improved.
Description
Technical Field
The invention relates to the field of computer science and technology, in particular to a data processing method and device.
Background
With the development of computer science and technology, the operation control technology of Remote Dictionary service (Redis) is continuously improved.
Currently, internet enterprises can apply Redis in various scenes such as high concurrency cache, application sharing and counters, the use scale of Redis can reach tens of thousands to hundreds of thousands of cases, and the large-scale deployment of Redis also becomes an inevitable trend. In the case of large-scale deployment, the operating state during the Redis operation needs to be effectively monitored.
However, the prior art cannot effectively monitor the running state of the redis.
Disclosure of Invention
In view of the above problems, the present invention provides a data processing method and apparatus for overcoming the above problems or at least partially solving the above problems, and the technical solution is as follows:
a method of data processing, comprising:
the method comprises the steps that a client device collects terminal operation data of remote dictionary service Redis on the client device, wherein the terminal operation data comprises index values of at least one Redis operation index item;
the client device sends the terminal operation data and the device identification to a cloud management platform of Redis;
the cloud management platform obtains service end operation data and a cluster identifier from a Redis cluster, wherein the service end operation data comprises index values of at least one Redis operation index item;
the cloud management platform respectively counts the index values of the operation index items in the terminal operation data and the server operation data to obtain a statistical result;
and the cloud management platform outputs processing information corresponding to the statistical result according to a predefined processing mode.
Optionally, the client device is provided with a collector and a target queue; the client device collects terminal operation data of remote dictionary service Redis on the client device, and the method comprises the following steps:
the client device collects the terminal operation data on the client device through a collector and sends the collected terminal operation data to the target queue for storage;
the client device periodically reads the terminal operation data from the target queue through a first thread, and accumulates the read terminal operation data to obtain accumulated data;
the client device sends the terminal operation data and the device identifier to a cloud management platform of Redis, and the method comprises the following steps:
the client device reports the current accumulated data and the device identifier to the cloud management platform periodically through a second thread; wherein the first thread and the second thread are independent of each other.
Optionally, the method further includes:
the client device collects the terminal operation data read in each period duration in the process of reading the terminal operation data;
and the client equipment monitors the memory consumed by the summarization each time, and outputs a memory consumption alarm when the memory consumed by the summarization is not less than a preset memory threshold value.
Optionally, the cloud management platform outputs processing information corresponding to the statistical result according to a predefined processing manner, where the processing information includes:
when at least one Redis operation index item with the index value meeting the preset alarm condition exists in the statistical result, the cloud management platform outputs corresponding alarm information, and the index item identifier, the index value, the index meaning and the importance degree of the at least one alarm index item are correspondingly stored in the alarm information.
Optionally, after the client device sends the terminal operation data and the device identifier to a cloud management platform of Redis, the method further includes:
and the client equipment deletes the terminal operation data which is stored by the client equipment and is sent to the cloud management platform.
A data processing apparatus comprising: client device and Redis's cloud management platform, client device includes first acquisition unit and first sending unit, the cloud management platform includes: the device comprises a first obtaining unit, a statistical unit, a second obtaining unit and a first output unit; wherein:
the first acquisition unit is used for acquiring terminal operation data of remote dictionary service Redis on the client device, wherein the terminal operation data comprises an index value of at least one Redis operation index item;
the first sending unit is used for sending the terminal operation data and the equipment identifier to the cloud management platform;
the first obtaining unit is configured to obtain server running data and a cluster identifier from a Redis cluster, where the server running data includes an index value of at least one Redis running index item;
the statistical unit is used for respectively performing statistics on the index values of the operation index items in the terminal operation data and the server operation data;
the second obtaining unit is used for obtaining a statistical result;
and the first output unit is used for outputting the processing information corresponding to the statistical result according to a predefined processing mode.
Optionally, the first collecting unit includes: the device comprises a collector, a second sending unit, a target queue, a reading unit, an accumulation unit and a third obtaining unit; wherein:
the collector is used for collecting the terminal operation data on the client device;
the second sending unit is used for sending the collected terminal operation data to the target queue for storage;
the reading unit is used for periodically reading the terminal running data from the target queue through a first thread;
the accumulation unit is used for accumulating the read terminal operation data;
the third obtaining unit is configured to obtain accumulated data;
the first sending unit is configured to periodically report the current accumulated data and the device identifier to the cloud management platform through a second thread; wherein the first thread and the second thread are independent of each other.
Optionally, the client device further includes: the system comprises a collecting unit, a monitoring unit and a second output unit; wherein:
the summarizing unit is used for summarizing the terminal operation data read in each period duration in the process of reading the terminal operation data by the client device;
the monitoring unit is used for monitoring the memory consumed by the summary performed by the client device each time;
and the second output unit is used for outputting a memory consumption alarm when the memory consumed by the summary is not less than a preset memory threshold value.
Optionally, the first output unit is configured to output corresponding warning information when at least one Redis operation index item whose index value meets a preset warning condition exists in the statistical result, where the warning information correspondingly stores an index item identifier, an index value, an index meaning, and an importance of the at least one warning index item.
Optionally, the client device further includes: a deletion unit;
the deleting unit is used for deleting the terminal operation data which are stored by the client device and are sent to the cloud management platform after the client device sends the terminal operation data and the device identification to the cloud management platform of Redis.
The invention provides a data processing method and a device.A client device collects terminal operation data of remote dictionary service Redis on the client device, wherein the terminal operation data comprises an index value of at least one Redis operation index item; the client device sends the terminal operation data and the device identification to a cloud management platform of Redis; the cloud management platform obtains service end operation data and a cluster identifier from the Redis cluster, wherein the service end operation data comprises index values of at least one Redis operation index item; the cloud management platform respectively counts the index values of the operation index items in the terminal operation data and the server operation data to obtain a statistical result; and the cloud management platform outputs processing information corresponding to the statistical result according to a predefined processing mode. The cloud management platform can obtain terminal operation data of the client device, obtain cluster operation data of the Redis cluster, realize functions of statistical analysis, monitoring alarm and the like of the Redis operation data of the client device and the Redis cluster, accordingly realize device monitoring of the client device and the Redis cluster, realize timely positioning of fault events or abnormal events, enable technicians to timely determine problem sources, improve problem solving efficiency, and improve operation efficiency and service quality of the Redis.
The foregoing description is only an overview of the technical solutions of the present invention, and the following detailed description of the present invention is provided to enable the technical means of the present invention to be more clearly understood, and to enable the above and other objects, features, and advantages of the present invention to be more clearly understood.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a signaling diagram illustrating a first data processing method according to an embodiment of the present invention;
fig. 2 is a signaling diagram illustrating a second data processing method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a first data processing apparatus according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
As shown in fig. 1, the present embodiment proposes a first data processing method, which may include the steps of:
s101, a client device collects terminal operation data of a Redis service on the client device, wherein the terminal operation data comprises index values of at least one Redis operation index item;
the client device may be a terminal electronic device provided with a Redis client. It should be noted that the terminal electronic device may be a mobile phone, a tablet computer, a desktop computer, or the like, and the device type of the terminal electronic device is not limited in the present invention.
The terminal operation data may be operation data of a Redis client on a client device.
Optionally, the client device may collect terminal operation data of a Redis client set by the client device in real time, and store the collected terminal operation data in a data storage space of the client device.
Specifically, the client device may monitor all operations of the Redis client for triggering the Redis, such as events of network connection, read-write and connection closing, timeout delay events, and Redis protocol specification (RESP) protocol interactions, and collect input data and output data and other indexes related to all operations.
S102, the client device sends the terminal operation data and the device identification to a cloud management platform of Redis;
the device identifier may be a device identifier of the client device.
Specifically, the device identifier may be one or more of characters such as chinese characters, numbers, letters, symbols, and the like.
The cloud management platform can be arranged at a cloud end and used for acquiring, summarizing and analyzing Redis running data on a Redis client and a Redis server, and achieving functions of monitoring and alarming the Redis running data, monitoring running states of client equipment and the Redis server, positioning fault equipment and the like, such as Cachecloud.
Optionally, the client device may send the terminal operation data and the device identifier to the cloud management platform in real time when acquiring the terminal operation data;
optionally, the client device may periodically send the acquired terminal operation data and the device identifier to the cloud management platform;
optionally, when the client device sends the terminal operation data and the device identifier to the cloud management platform, the client device may send the requested cluster identifier of the Redis cluster related to the terminal operation data to the cloud management platform.
Optionally, the client device may perform data sorting and summarization on the acquired terminal operation data according to a predefined data reporting format, and send the sorted and summarized terminal operation data to the cloud management platform. The terminal operation data collected and aggregated by the client device may include: general data, command time consumption data, command value statistical data, exception statistical data, and the like.
The general data can comprise data acquisition time and data reporting time; the command time consumption may include time consumption distribution information, command calling times, command names, instance addresses, 90% maximum consumed time data, 99% maximum consumed time data, longest consumed time, average consumed time, median consumed time, and the like; the command value statistic data may include a command name, an instance address, a call amount, a value size distribution interval, and the like; the exception statistics may include exception class name, exception message, exception occurrence time, instance address, number of exceptions, and exception type, among others.
Optionally, after the client device sends the terminal operation data and the device identifier to the cloud management platform of the Redis, the method further includes:
and the client equipment deletes the terminal operation data which is stored by the client equipment and is sent to the cloud management platform.
Specifically, after the client device sends the terminal operation data and the device identifier to the cloud management platform, the sent terminal data can be deleted, the data storage space is released, and the utilization rate of the data storage space is improved.
S103, the cloud management platform obtains service end operation data and a cluster identifier from a Redis cluster, wherein the service end operation data comprises index values of at least one Redis operation index item;
the Redis cluster may be a server cluster device of Redis.
The server running data may be Redis running data on a Redis server.
The cluster identifier may be a cluster identifier of a Redis cluster.
Specifically, the cloud management platform can actively obtain the server operating data and the cluster identifier from the Redis cluster.
Optionally, the cloud management platform may first obtain the terminal operation data from the client device, and then obtain the server operation data from the Redis cluster, as shown in fig. 1; optionally, the cloud management platform may also obtain the server operation data from the Redis cluster in advance, and then obtain the terminal operation data from the client device; of course, the cloud management platform may also obtain the terminal operation data from the client device and the server operation data from the Redis cluster at the same time.
S104, the cloud management platform respectively counts the index values of all operation index items in the terminal operation data and the server operation data;
specifically, the cloud management platform may perform statistics on index values of operation index items included in the terminal operation data and perform statistics on index values of operation index items included in the server operation data after obtaining the terminal operation data, the device identifier of the client device, the server operation data, and the cluster identifier.
Optionally, when the client device sends the terminal operation data, the device identifier, and the cluster identifier of the Redis cluster to the cloud management platform, the cloud management platform may perform statistics on the terminal operation data and the cluster operation data corresponding to the cluster identifier.
Optionally, the cloud management platform may perform data statistics on the terminal operation data and the cluster operation data according to a predefined data statistics manner.
Optionally, the cloud management platform may also perform statistics on the terminal operation data in advance when the terminal operation data and the device identifier of the client device are obtained and the server operation data and the cluster identifier are not obtained yet, and then perform statistics on the server operation data when the server operation data and the cluster identifier are obtained;
optionally, the cloud management platform may also perform statistics on the server operation data in advance when the server operation data and the cluster identifier are obtained but the terminal operation data and the device identifier of the client device are not obtained yet, and then perform statistics on the terminal operation data when the terminal operation data and the device identifier of the client device are obtained.
S105, the cloud management platform obtains a statistical result;
specifically, the invention can set a header comprising serial numbers, index items, alarm values, index meanings and importance degrees for distinguishing, and count the terminal operation data and the cluster operation data according to the header. In order to better illustrate the statistical manner of the cloud management platform for the terminal operation data and the cluster operation data, the following table 1 is provided in this embodiment for explanation.
TABLE 1 statistical tables
And S106, outputting the processing information corresponding to the statistical result by the cloud management platform according to a predefined processing mode.
Specifically, the cloud management platform may determine whether the index value of each operation index item is normal according to the index value of each operation index item in the statistical result, analyze the device states of the client device and the Redis cluster, determine an abnormal event and a fault event, locate an abnormal source, and output an alarm.
Optionally, step S106 may include:
and when at least one Redis operation index item with the index value meeting the preset alarm condition exists in the statistical result, the cloud management platform outputs corresponding alarm information, and the index item identifier, the index value, the index meaning and the importance degree of the at least one alarm index item are correspondingly stored in the alarm information.
Specifically, the cloud management platform may actively and timely trigger an alarm, such as a Redis emergency alarm, when an index value of a certain operation index item reaches an alarm threshold value.
Optionally, the cloud management platform may audit the health degrees of the Redis client and the Redis cluster based on the obtained terminal operation data and cluster operation data of the Redis, and perform visual display; the cloud management platform can audit the flow health degree of the Redis client and perform visual display; the cloud management platform can continuously iterate the monitoring process;
optionally, the cloud management platform may enable the operation and maintenance personnel to adjust the setting of the Redis cluster according to the alarm information by outputting the alarm information, for example, the memory expands and contracts each node, so as to implement the operation and maintenance auxiliary function; the cloud management platform can enable developers to optimize project codes according to the alarm information by outputting the alarm information, and avoid the occurrence of large objects, slow query and the like;
it should be noted that the cloud management platform may obtain terminal operation data of the client device, obtain cluster operation data of the plurality of Redis clusters, and implement functions such as statistical analysis and monitoring alarm of the Redis operation data of the client device and the Redis clusters, thereby implementing device monitoring of the client device and the Redis clusters, implementing timely positioning of a fault event or an abnormal event, so that a technician can determine a problem source in time, improve problem solving efficiency, and thereby improve operation efficiency and service quality of the Redis.
It can be understood that the cloud management platform may obtain terminal operation data of a plurality of client devices, may also obtain cluster operation data of a plurality of Redis clusters, and implement functions such as statistical analysis and monitoring alarm of the Redis operation data of the plurality of client devices and the plurality of Redis clusters, thereby implementing device monitoring of the plurality of client devices and the plurality of Redis clusters, and further improving operation efficiency and service quality of the Redis.
In the data processing method provided by this embodiment, a client device collects terminal operation data of a remote dictionary service Redis on the client device, where the terminal operation data includes an index value of at least one Redis operation index item; the client device sends the terminal operation data and the device identification to a cloud management platform of Redis; the cloud management platform obtains service end operation data and a cluster identifier from the Redis cluster, wherein the service end operation data comprises index values of at least one Redis operation index item; the cloud management platform respectively counts the index values of the operation index items in the terminal operation data and the server operation data to obtain a statistical result; and the cloud management platform outputs processing information corresponding to the statistical result according to a predefined processing mode. The cloud management platform can obtain terminal operation data of the client device, obtain cluster operation data of the Redis cluster, realize functions of statistical analysis, monitoring alarm and the like of the Redis operation data of the client device and the Redis cluster, accordingly realize device monitoring of the client device and the Redis cluster, realize timely positioning of fault events or abnormal events, enable technicians to timely determine problem sources, improve problem solving efficiency, and improve operation efficiency and service quality of the Redis.
Based on the steps shown in fig. 1, the present embodiment proposes a second data processing method, as shown in fig. 2. In the method, a collector and a target queue are arranged on client equipment; step S101 may include S1011, S1012, S1013, and S1014, and step S102 may be specifically S1021; wherein:
s1011, the client device collects terminal operation data on the client device through a collector;
it should be noted that, a technician may perform source code embedding on the client device in advance for all operations (such as a connection event, a RESP protocol execution data, an timeout exception event, and the like) that may be involved when the client device interacts with the Redis cluster, that is, modify a source code on the client device, and add a data acquisition logic to implement setting an acquirer on the client device.
Specifically, when an Application Programming Interface (API) of the Redis client on the client device is in a call state, the collector may actively execute a data collection logic to collect terminal operation data on the client device.
It should be noted that, when the client device starts the collector, it may start a single-thread worker to perform active collection of terminal operation data, and when collection is finished, the single thread may be closed. In particular, the single thread may be independent of other threads on the client device, such as threads of a Redis client.
S1012, the client device sends the collected terminal operation data to a target queue for storage;
optionally, the client device may send the collected terminal operation data to the target queue for temporary storage through the collector.
Wherein the target queue may be a bounded blocking queue supported by an array, such as arrayBlockingQueue. Optionally, the target queue may internally sort the received terminal operation data, and may manage the terminal operation data in a first-in first-out manner. Specifically, the head data of the queue may be the element with the longest dwell time in the queue, and the tail of the queue may be the element with the shortest dwell time in the queue. The target queue may place a new element at the tail of the queue as it receives it, and the queue retrieve operation may retrieve the head element in the queue as it retrieves the data.
Optionally, the queue size of the target queue may be limited to 1024, and the terminal running data exceeding the queue limit may be directly discarded without blocking the RESP interaction.
It should be noted that the target queue manages the received terminal operation data by using an elegant closing mechanism, which can prevent unnecessary data loss under normal conditions and ensure data reliability.
S1013, the client device periodically reads terminal running data from the target queue through the first thread;
the first thread may be a thread that is started each time the client reads the terminal operation data from the target queue. Specifically, the client device may read the terminal operation data stored in the target queue in a lock-free and single-threaded manner.
Optionally, the method may further include:
the method comprises the steps that in the process of reading terminal operation data, the client device collects the terminal operation data read in each period duration;
and the client equipment monitors the memory consumed by each time of summarizing, and outputs a memory consumption alarm when the memory consumed by summarizing is not less than a preset memory threshold value.
The period duration may be set by a technician according to actual conditions, and the present invention is not limited thereto. For example, the period duration may be 1 minute, at this time, the client device may summarize the terminal operation data read every minute in the process of reading the terminal operation data, for example, the client device may summarize the terminal operation data read in the 1 st minute, and summarize the terminal operation data read in the 2 nd minute.
Specifically, the client device may monitor the memory consumed or overhead during the summary of the data, and may present the memory to the user or the technician in an intuitive dashboard manner. Specifically, if the memory consumed in a certain summary is not less than the preset memory threshold, a memory consumption alarm may be output, so as to avoid an OOM error caused by excessive memory consumption.
The preset memory threshold may be set by a technician according to an actual situation, which is not limited in the present invention.
S1014, accumulating the read terminal operation data by the client equipment to obtain accumulated data;
specifically, the client device may accumulate the terminal operation data read in each period duration to obtain corresponding accumulated data.
It should be noted that the client device accumulates the read terminal operation data, and can realize preliminary statistics on the terminal operation data, so that the workload of the cloud management platform in the process of counting the terminal operation data can be reduced, the resource consumption of the cloud management platform is reduced, and the statistical efficiency of the cloud management platform on the terminal operation data is improved.
S1021, the client equipment periodically reports the current accumulated data and the equipment identification to the cloud management platform through a second thread; wherein the first thread and the second thread are independent of each other.
Specifically, the client device may actively and periodically send the accumulated data to the cloud management platform. When the client device sends the accumulated data, a second thread different from the first thread can be started to execute a task of sending the accumulated data to the cloud management platform.
It should be noted that, when the client device executes the steps of collecting, reading, accumulating, reporting and the like of the terminal operation data, each step can start an independent thread to execute, and does not need to occupy a thread pool used by the Redis client to execute normal service logic, so that the execution process of each step does not affect the normal execution of the Redis client to the service logic, the robustness of the client device to the logic of data collection, reading, accumulating, reporting and the like is ensured, and the intrusion to the client device under the condition of abnormal introduction is prevented.
The data processing method provided by the embodiment can effectively realize the collection, reading, accumulation and report of the terminal operation data by the client device, and ensure the realization of the management of the terminal operation data by the cloud management platform.
Corresponding to the steps shown in fig. 1, as shown in fig. 3, the present embodiment proposes a first data processing apparatus 100, which may include: the client device 101 and the cloud management platform 102 of the Redis, the client device 101 includes a first acquisition unit 1011 and a first transmission unit 1012, and the cloud management platform 102 includes: a first obtaining unit 1021, a statistical unit 1022, a second obtaining unit 1023, and a first output unit 1024; wherein:
a first collecting unit 1011, configured to collect terminal operation data of a remote dictionary service Redis on the client device 101, where the terminal operation data includes an index value of at least one Redis operation index item;
a first sending unit 1012, configured to send the terminal operation data and the device identifier to the cloud management platform 102;
a first obtaining unit 1021, configured to obtain server-side operation data and a cluster identifier from a Redis cluster, where the server-side operation data includes an index value of at least one Redis operation index item;
a statistics unit 1022, configured to perform statistics on index values of each operation index item in the terminal operation data and the server operation data respectively;
a second obtaining unit 1023 for obtaining statistical results;
the first output unit 1024 is configured to output processing information corresponding to the statistical result according to a predefined processing manner.
It should be noted that, specific processing procedures of the first acquiring unit 1011, the first sending unit 1012, the first obtaining unit 1021, the statistical unit 1022, the second obtaining unit 1023, and the first output unit 1024 and technical effects thereof may refer to steps S101, S102, S103, S104, S105, and S106 in fig. 1, respectively, and are not described herein again.
Optionally, the first acquisition unit 1011 includes: the device comprises a collector, a second sending unit, a target queue, a reading unit, an accumulation unit and a third obtaining unit; wherein:
the collector is used for collecting terminal operation data on the client device 101;
the second sending unit is used for sending the collected terminal operation data to the target queue for storage;
the reading unit is used for periodically reading terminal operation data from the target queue through a first thread;
the accumulation unit is used for accumulating the read terminal operation data;
a third obtaining unit configured to obtain accumulated data;
a first sending unit 1012, configured to periodically report the current accumulated data and the device identifier to the cloud management platform 102 through the second thread; wherein the first thread and the second thread are independent of each other.
Optionally, the client device 101 further includes: the system comprises a collecting unit, a monitoring unit and a second output unit; wherein:
the summarizing unit is used for summarizing the terminal operation data read in each period duration in the process of reading the terminal operation data by the client device 101;
a monitoring unit, configured to monitor a memory consumed by the aggregation performed by the client device 101 each time;
and the second output unit is used for outputting a memory consumption alarm when the memory consumed in the summary is not less than the preset memory threshold value.
Optionally, the first output unit 1024 is configured to output corresponding warning information when at least one Redis operation index item whose index value meets a preset warning condition exists in the statistical result, where the warning information correspondingly stores an index item identifier, an index value, an index meaning, and an importance of the at least one warning index item.
Optionally, the client device 101 further includes: a deletion unit;
and the deleting unit is used for deleting the terminal operation data which are stored by the client device 101 and are sent to the cloud management platform 102 after the client device 101 sends the terminal operation data and the device identifier to the cloud management platform 102 of Redis.
In the data processing apparatus 100 provided in this embodiment, the client device 101 acquires terminal operation data of the remote dictionary service Redis on the client device 101, where the terminal operation data includes an index value of at least one Redis operation index item; the client device 101 sends the terminal operation data and the device identifier to a cloud management platform 102 of Redis; the cloud management platform 102 obtains service end operation data and a cluster identifier from a Redis cluster, wherein the service end operation data comprises index values of at least one Redis operation index item; the cloud management platform 102 respectively counts the index values of the operation index items in the terminal operation data and the server operation data to obtain a statistical result; the cloud management platform 102 outputs processing information corresponding to the statistical result according to a predefined processing manner. The cloud management platform 102 can obtain terminal operation data of the client device 101, obtain cluster operation data of the Redis cluster, and realize functions of statistical analysis, monitoring alarm and the like on the Redis operation data of the client device 101 and the Redis cluster, so that device monitoring on the client device 101 and the Redis cluster is realized, timely positioning of a fault event or an abnormal event is realized, technical personnel can timely determine a problem source, problem solving efficiency is improved, and operation efficiency and service quality of the Redis are improved.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A data processing method, comprising:
the method comprises the steps that a client device collects terminal operation data of remote dictionary service Redis on the client device, wherein the terminal operation data comprises index values of at least one Redis operation index item;
the client device sends the terminal operation data and the device identification to a cloud management platform of Redis;
the cloud management platform obtains service end operation data and a cluster identifier from a Redis cluster, wherein the service end operation data comprises index values of at least one Redis operation index item;
the cloud management platform respectively counts the index values of the operation index items in the terminal operation data and the server operation data to obtain a statistical result;
the cloud management platform outputs processing information corresponding to the statistical result according to a predefined processing mode;
and auditing the flow health degree of the Redis client and the Redis cluster and visually displaying the flow health degree based on the acquired terminal operation data and cluster operation data of the Redis by the cloud management platform.
2. The data processing method of claim 1, wherein a collector and a target queue are arranged on the client device; the client device collects terminal operation data of remote dictionary service Redis on the client device, and the method comprises the following steps:
the client device collects the terminal operation data on the client device through a collector and sends the collected terminal operation data to the target queue for storage;
the client device periodically reads the terminal operation data from the target queue through a first thread, and accumulates the read terminal operation data to obtain accumulated data;
the client device sends the terminal operation data and the device identifier to a cloud management platform of Redis, and the method comprises the following steps:
the client device reports the current accumulated data and the device identifier to the cloud management platform periodically through a second thread; wherein the first thread and the second thread are independent of each other.
3. The data processing method of claim 2, wherein the method further comprises:
the client device collects the terminal operation data read in each period duration in the process of reading the terminal operation data;
and the client equipment monitors the memory consumed by the summarization each time, and outputs a memory consumption alarm when the memory consumed by the summarization is not less than a preset memory threshold value.
4. The data processing method according to claim 1, wherein the cloud management platform outputs the processing information corresponding to the statistical result according to a predefined processing manner, including:
when at least one Redis operation index item with the index value meeting the preset alarm condition exists in the statistical result, the cloud management platform outputs corresponding alarm information, and the index item identifier, the index value, the index meaning and the importance degree of the at least one alarm index item are correspondingly stored in the alarm information.
5. The data processing method according to claim 1, wherein after the client device sends the terminal operation data and the device identifier to a cloud management platform of Redis, the method further comprises:
and the client equipment deletes the terminal operation data which is stored by the client equipment and is sent to the cloud management platform.
6. A data processing apparatus, comprising: client device and Redis's cloud management platform, client device includes first acquisition unit and first sending unit, the cloud management platform includes: the device comprises a first obtaining unit, a statistical unit, a second obtaining unit and a first output unit; wherein:
the first acquisition unit is used for acquiring terminal operation data of remote dictionary service Redis on the client device, wherein the terminal operation data comprises an index value of at least one Redis operation index item;
the first sending unit is used for sending the terminal operation data and the equipment identifier to the cloud management platform;
the first obtaining unit is configured to obtain server running data and a cluster identifier from a Redis cluster, where the server running data includes an index value of at least one Redis running index item;
the statistical unit is used for respectively performing statistics on the index values of the operation index items in the terminal operation data and the server operation data;
the second obtaining unit is used for obtaining a statistical result;
the first output unit is used for outputting processing information corresponding to the statistical result according to a predefined processing mode;
and the first output unit is also used for auditing the flow health degree of the Redis client and the Redis cluster and visually displaying the flow health degree based on the obtained terminal operation data and the cluster operation data of the Redis.
7. The apparatus of claim 6, wherein the first acquisition unit comprises: the device comprises a collector, a second sending unit, a target queue, a reading unit, an accumulation unit and a third obtaining unit; wherein:
the collector is used for collecting the terminal operation data on the client device;
the second sending unit is used for sending the collected terminal operation data to the target queue for storage;
the reading unit is used for periodically reading the terminal running data from the target queue through a first thread;
the accumulation unit is used for accumulating the read terminal operation data;
the third obtaining unit is configured to obtain accumulated data;
the first sending unit is configured to periodically report the current accumulated data and the device identifier to the cloud management platform through a second thread; wherein the first thread and the second thread are independent of each other.
8. The apparatus of claim 7, wherein the client device further comprises: the system comprises a collecting unit, a monitoring unit and a second output unit; wherein:
the summarizing unit is used for summarizing the terminal operation data read in each period duration in the process of reading the terminal operation data by the client device;
the monitoring unit is used for monitoring the memory consumed by the summary performed by the client device each time;
and the second output unit is used for outputting a memory consumption alarm when the memory consumed by the summary is not less than a preset memory threshold value.
9. The apparatus according to claim 6, wherein the first output unit is configured to, when at least one Redis operation index item whose index value meets a preset alarm condition exists in the statistical result, output corresponding alarm information, where an index item identifier, an index value, an index meaning, and an importance of the at least one alarm index item are correspondingly stored in the alarm information.
10. The apparatus of claim 6, wherein the client device further comprises: a deletion unit;
the deleting unit is used for deleting the terminal operation data which are stored by the client device and are sent to the cloud management platform after the client device sends the terminal operation data and the device identification to the cloud management platform of Redis.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210064845.1A CN114090644B (en) | 2022-01-20 | 2022-01-20 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210064845.1A CN114090644B (en) | 2022-01-20 | 2022-01-20 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114090644A CN114090644A (en) | 2022-02-25 |
| CN114090644B true CN114090644B (en) | 2022-04-26 |
Family
ID=80308980
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210064845.1A Active CN114090644B (en) | 2022-01-20 | 2022-01-20 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114090644B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114637656B (en) * | 2022-05-13 | 2022-09-20 | 飞狐信息技术（天津）有限公司 | Redis-based monitoring method and device, storage medium and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106878064A (en) * | 2017-01-16 | 2017-06-20 | 腾讯科技（深圳）有限公司 | Data monitoring method and device |
| CN110213125A (en) * | 2019-05-23 | 2019-09-06 | 南京维拓科技股份有限公司 | Abnormality detection system based on time series data under a kind of cloud environment |
| CN111049705A (en) * | 2019-12-23 | 2020-04-21 | 深圳前海微众银行股份有限公司 | Method and device for monitoring distributed storage system |
| CN111740884A (en) * | 2020-08-25 | 2020-10-02 | 云盾智慧安全科技有限公司 | Log processing method, electronic equipment, server and storage medium |
| CN113934547A (en) * | 2021-08-30 | 2022-01-14 | 山东浪潮科学研究院有限公司 | Method, device, equipment and product for recording alarm log in cluster environment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108234170B (en) * | 2016-12-15 | 2021-06-22 | 北京神州泰岳软件股份有限公司 | Monitoring method and device for server cluster |
| CN109714192B (en) * | 2018-11-29 | 2022-03-04 | 深圳供电局有限公司 | Monitoring method and system for monitoring cloud platform |
| CN110445637B (en) * | 2019-07-05 | 2022-08-09 | 深圳壹账通智能科技有限公司 | Event monitoring method, system, computer device and storage medium |
| CN112636979B (en) * | 2020-12-24 | 2022-08-12 | 北京浪潮数据技术有限公司 | Cluster alarm method and related device |
-
2022
- 2022-01-20 CN CN202210064845.1A patent/CN114090644B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106878064A (en) * | 2017-01-16 | 2017-06-20 | 腾讯科技（深圳）有限公司 | Data monitoring method and device |
| CN110213125A (en) * | 2019-05-23 | 2019-09-06 | 南京维拓科技股份有限公司 | Abnormality detection system based on time series data under a kind of cloud environment |
| CN111049705A (en) * | 2019-12-23 | 2020-04-21 | 深圳前海微众银行股份有限公司 | Method and device for monitoring distributed storage system |
| CN111740884A (en) * | 2020-08-25 | 2020-10-02 | 云盾智慧安全科技有限公司 | Log processing method, electronic equipment, server and storage medium |
| CN113934547A (en) * | 2021-08-30 | 2022-01-14 | 山东浪潮科学研究院有限公司 | Method, device, equipment and product for recording alarm log in cluster environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114090644A (en) | 2022-02-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2791825B1 (en) | System and method for monitoring and managing data center resources in real time incorporating manageability subsystem | |
| US7275097B2 (en) | System and method for analyzing input/output activity on local attached storage | |
| CA2753019C (en) | Monitoring of distributed applications | |
| CN101997925A (en) | Server monitoring method with early warning function and system thereof | |
| CN106487574A (en) | Automatic operating safeguards monitoring system | |
| US20110160923A1 (en) | Method and apparatus for monitoring the performance of a power delivery control system | |
| US20080168044A1 (en) | System and method for providing performance statistics for application components | |
| CN109271243B (en) | Cluster task management system | |
| CN109885453B (en) | Big data platform monitoring system based on stream data processing | |
| CN109039817B (en) | Information processing method, device, equipment and medium for flow monitoring | |
| CN113746703B (en) | Abnormal link monitoring method, system and device | |
| CN114090644B (en) | Data processing method and device | |
| CN112865311B (en) | Method and device for monitoring message bus of power system | |
| CN108920322A (en) | A kind of method, apparatus and computer storage medium of middleware operational monitoring | |
| CN111339466A (en) | Interface management method and device, electronic equipment and readable storage medium | |
| CN110011845B (en) | Log collection method and system | |
| CN114328093A (en) | Hadoop-based monitoring method, system, storage medium and equipment | |
| CN109766238B (en) | Session number-based operation and maintenance platform performance monitoring method and device and related equipment | |
| CN116610532B (en) | Redis client monitoring method, system, electronic equipment and storage medium | |
| CN113434366A (en) | Event processing method and system | |
| CN111654410B (en) | Gateway request monitoring method, device, equipment and medium | |
| CN117891686A (en) | Monitoring and early warning method based on micro-service architecture | |
| CN114490248A (en) | Workstation performance monitoring and warning system and method thereof | |
| CN114138720A (en) | Log processing method, log processing device, electronic device and storage medium | |
| CN113609429A (en) | Data processing method and device based on message queue, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |