CN111506581A - Data aggregation method and server - Google Patents

Abstract

The invention discloses a data aggregation method, which is suitable for being executed in a server, wherein the server is respectively in communication connection with a message middleware and a data index database, and the method comprises the following steps: reading topic data of each computing node for different topic types from message middleware, wherein the topic data comprises a timestamp and at least one index value; temporarily storing the theme data in a preset data flow queue, and extracting the theme data from the data flow queue one by adopting a preset filtering flow queue; aggregating the same subject data of the same computing node according to at least one aggregation period, and computing the index statistic value of the computing node in each aggregation period; and summarizing the index statistics of a plurality of aggregation cycles in a preset index time period, taking the index statistics as a data index of the computing node in the preset index time period, and storing the data index into a data index database. The invention also discloses a server for executing the method.

Description

Data aggregation method and server

Technical Field

The invention relates to the field of log processing, in particular to a data aggregation method and a server.

Background

In recent years, high-performance computing technology is gradually developed, along with the increasing demand of computing power, the expansion of computing scale and the change of computing mode, and more users pay attention to not only monitoring and analysis of macroscopic performance data but also correlation analysis of microscopic performance data. The E-level system is large in scale, complex in supporting subject application, and needs to realize high-frequency and low-load acquisition, storage, analysis and mining of multi-dimensional management data and performance big data. Aiming at the collected mass performance data, it is important to design data aggregation so as to realize effective multi-dimensional and multi-index performance data visualization analysis, and higher requirements are provided for the real-time performance and the accuracy of data processing.

In one scheme, after the server side obtains data, the source data is directly stored in the non-relational database. When a data aggregation request occurs, reading a scene service dimension and an accounting service dimension corresponding to an aggregation requirement from a non-relational database according to different aggregation dimensions, processing real-time data in a single-instance multithreading mode on the read accounting service dimension based on the scene service dimension, and performing data aggregation of different dimensions until the data are returned to a request end. This approach is not suitable for use in large data volume scenarios, and the single point failure problem may produce inaccurate analysis results. Moreover, when a data aggregation request occurs, the server synchronously reads, calculates and aggregates data, which is unfriendly for some scenes with relatively high response speed requirements, and also has great pressure for a non-relational database, and a situation of an instant resource use peak value may occur.

In another scheme, different data processing flows are designed according to different needs of services, so that the efficiency and flexibility of data processing services are low; and the result of each service has strong service relevance, and cannot be provided for other services as input data, so that a great deal of computation power is wasted. Or each service may directly read the original data upon request, which may cause a large number of repeated I/O operations and may even degrade the I/O performance of the original service, thereby affecting the original service system.

Disclosure of Invention

In view of the above, the present invention proposes a data aggregation method and a server in an effort to solve or at least solve the above existing problems.

According to an aspect of the present invention, there is provided a data aggregation method adapted to be executed in a server, the server being communicatively connected to a message middleware and a data index repository, respectively, the message middleware being communicatively connected to at least one computing node, the method comprising the steps of: reading topic data of each computing node for different topic types from message middleware, wherein the topic data comprises a timestamp and at least one index value; temporarily storing the subject data in a preset data flow queue, and extracting the subject data from the data flow queue one by adopting a preset filtering flow queue; aggregating the same subject data of the same computing node according to at least one aggregation period, and computing the index statistic value of the computing node in each aggregation period; summarizing the index statistics values of a plurality of aggregation cycles in a preset index time period, using the summary values as data indexes of the computing nodes in the preset index time period, and storing the data indexes into the data index database.

Optionally, in the data aggregation method according to the present invention, the theme type includes at least one of a CPU index, a disk IO index, a database index, and a file system index.

Optionally, in the data aggregation method according to the present invention, the CPU index includes at least one of a total number of cores of the CPU, a total number of CPUs, a user mode duty ratio, a kernel mode duty ratio, an IO wait duty ratio, a hard interrupt duty ratio, a soft interrupt duty ratio, and a virtual machine stealing duty ratio.

Optionally, in the data aggregation method according to the present invention, the disk IO indicator includes at least one of a number of disks, a disk read rate, a disk write rate, an IO waiting time per second, an IO service time per second, a queue request number per second, IO read times per second, and IO write times per second.

Optionally, in the data aggregation method according to the present invention, the index statistics value in the aggregation period includes at least one of a maximum value, a minimum value, an average value, and a total number of the same index in the aggregation period.

Optionally, in the data aggregation method according to the present invention, the step of aggregating the same subject data of the same computing node according to at least one aggregation period, and calculating the index statistic of the computing node in each aggregation period includes: establishing an aggregation flow queue and an aggregation flow core corresponding to each aggregation period, and numbering the aggregation flow queue and the aggregation flow core respectively according to the total class number of the aggregation period; and aggregating the same subject data of the same computing node according to the corresponding aggregation period by adopting the aggregation flow queue corresponding to each aggregation period, and computing the index statistical value of the computing node in each aggregation period by adopting the corresponding core flow queue.

Optionally, in the data aggregation method according to the present invention, when the aggregation cycle is 5min, the preset index period is 1 week; when the polymerization cycle is 1h, the preset index time period is 1 month; when the polymerization cycle is 1day, the preset index period is 1 year.

Optionally, in the data aggregation method according to the present invention, the step of temporarily storing the theme data in a preset data flow queue includes: and according to the preset field corresponding relation, assembling the subject data read from the message middleware into a standard data set and storing the standard data set into a data stream queue.

Optionally, in the data aggregation method according to the present invention, the database into which the data index is stored includes a search database and a time series database, and the method further includes the steps of: respectively storing the subject data read from the message middleware as source data into a search database and a time sequence database; and monitoring whether the source data is abnormal or not in real time by adopting a monitoring queue, and sending an alarm indication when the source data is abnormal.

Optionally, in the data aggregation method according to the present invention, the method further includes the steps of: respectively calculating the input/output flow rate of the data flow in each queue; and monitoring whether the input/output flow rate or the index value in each aggregation period is abnormal or not in real time by using a monitoring queue, and sending an alarm indication when the abnormality occurs.

According to still another aspect of the present invention, there is provided a server including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs when executed by the processors implement the steps of the data aggregation method as described above.

According to a further aspect of the present invention, there is provided a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a server, implement the steps of the data aggregation method as described above.

According to the data aggregation scheme, the data flow queue RowFlow is established and closed in time, so that the abnormal condition caused by untimely application program processing in a large-batch data scene which may occur is avoided, and meanwhile, the resource waste is avoided. The invention carries out the aggregation processing of the corresponding business requirements on the collected original data and stores the processing result in the corresponding non-relational database. The aggregation operation greatly reduces the I/O pressure of the original service to the server, and simultaneously effectively reduces the mutual interference of different services.

In addition, the invention also stores the data with different dimensionalities in a non-relational database in a flexibly configured data type, so that the data retention time is low, the consistency of the data structure is weakened, the use is more flexible and the expandability is good. Meanwhile, the real-time processing state of the data stream is monitored and processed, the running states of different data streams can be seen through the log, and a reliable reference value is provided for early warning of component exception handling. Finally, the invention supports the data to be reserved in a file format in the data processing process, and provides source data guarantee for the abnormal supplementary recording of the data in the later period.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.

FIG. 1 shows a block diagram of a data aggregation system 100 according to one embodiment of the invention.

FIG. 2 shows a block diagram of a computing device 200, according to one embodiment of the invention.

FIG. 3 shows a flow diagram of a data aggregation method 300 according to one embodiment of the invention.

Fig. 4 shows a schematic diagram of a data flow completion transmission queue according to an embodiment of the invention.

FIG. 5 illustrates a schematic diagram of data indexing, according to one embodiment of the invention.

FIG. 6 shows a schematic diagram of monitoring a data aggregation flow, according to one embodiment of the invention.

FIG. 7 shows a schematic diagram of data flow input/output parameters, according to one embodiment of the invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

FIG. 1 shows a schematic diagram of a data aggregation system 100 according to one embodiment of the invention. As shown in the figure, system 100 includes at least one compute node 110, message middleware 120, a server 130, and a data index repository 140. It should be noted that the data aggregation system 100 in fig. 1 is merely exemplary, and in a specific practical situation, there may be different numbers of computing nodes 110, message middleware 120, servers 130, and data index libraries 140 in the system 100, and the present invention does not limit the number of devices included in the system 100.

The computing nodes 110 may be deployed in the same geographical location, or may be deployed in different geographical locations, which is not limited in the present invention. Each compute node 110 has multiple CPU processors to process the assigned jobs. Meanwhile, the computing node 110 sends the current device status, such as CPU status, network status, disk status, etc., to the messaging middleware 120 communicatively connected thereto. The messaging middleware 120 is communicatively coupled to the server 130 and is adapted to send the log information obtained from the computing node 110 to the server 130 according to information on different topics for different nodes in response to a subscription request from the server 130. The server 130 is connected to the data index base 140 in a communication manner, and is adapted to store the data obtained from the message middleware 120 into the data index base 140 after aggregating the data.

According to one embodiment, the message middleware 120 is NATs middleware. The server 130 may be a single server, or a server cluster composed of a plurality of servers, or a cloud computing service center, and a plurality of servers for constituting the server cluster or the cloud computing service center may reside in a plurality of geographic locations. The data index library 140 includes a search database, such as an ElasticSearch Server (ES), and a time series database, such as an infiluxdb database.

According to AN embodiment of the present invention, the various components of the data aggregation system 100 described above may communicate over one or more networks, such as a local area network (L AN) or a Wide Area Network (WAN), such as the Internet, wherein the computing node 110, the message middleware 120, the server 130, and the data index repository 140 may each be implemented by a computing device 200 as described below.

FIG. 2 shows a block diagram of a computing device 200, according to one embodiment of the invention. In a basic configuration 202, computing device 200 typically includes system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.

The processor 204 may be any type of processing, including but not limited to a microprocessor (μ P), a microcontroller (μ C), a digital information processor (DSP), or any combination thereof, depending on the desired configuration the processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, a processor core 214, and registers 216 the example processor core 214 may include an arithmetic logic unit (A L U), a Floating Point Unit (FPU), a digital signal processing core (DSP core), or any combination thereof the example memory controller 218 may be used with the processor 204 or, in some implementations, the memory controller 218 may be an internal part of the processor 204.

Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 206 may include an operating system 220, one or more applications 222, and program data 224. In some implementations, the application 222 can be arranged to operate with program data 224 on an operating system. The program data 224 comprises instructions, and in the computing device 200 according to the invention the program data 224 comprises instructions for performing the data aggregation method 300.

Computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 252. Example peripheral interfaces 244 can include a serial interface controller 254 and a parallel interface controller 256, which can be configured to facilitate communications with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 258. An example communication device 246 may include a network controller 260, which may be arranged to facilitate communications with one or more other computing devices 262 over a network communication link via one or more communication ports 264.

A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.

Computing device 200 may be implemented as a server, such as a file server, a database server, an application server, a WEB server, etc., or as part of a small-sized portable (or mobile) electronic device, such as a cellular telephone, a Personal Digital Assistant (PDA), a wireless WEB-watch device, an application specific device, or a hybrid device that include any of the above functions. Computing device 200 may also be implemented as a personal computer including both desktop and notebook computer configurations. In some embodiments, the computing device 200 is configured to perform the data aggregation method 300.

Fig. 3 shows a flow diagram of a data aggregation method 300 according to an embodiment of the invention. The method 300 is performed in a server (e.g., in server 130) to complete the aggregate storage of log data.

As shown in fig. 3, the method begins at step S310. In step S310, topic data for different topic types for each compute node is read from the message middleware. The theme data includes a timestamp and at least one index value, and may further include a host name, a host identifier, an IP address, and the like.

The theme data is in a data stream form, and the theme data adopts a RingBuffer data structure. RingBuffer is a circular buffer, is used for transmitting data among 2 threads, and is a standard first-in first-out (FIFO) model. Two pointers, a head pointer and a tail pointer, are arranged in the RingBuffer, the head pointer points to the position of the next reading, and the tail pointer points to the position of the next writing. The RingBuffer may be stored in an array where the memory addresses of the elements in the array are contiguous.

The theme type comprises at least one of a CPU index, a disk IO index, a database index and a file system index. The database index is, for example, Mysql database index, and the file system index is, for example, NFS system index. Each large-class index is used as a theme, and a plurality of small-class indexes are possible under the theme. For example, the CPU index includes at least one of a total number of CPU cores, a user mode duty ratio, a core mode duty ratio, an IO wait duty ratio, a hard interrupt duty ratio, a soft interrupt duty ratio, and a virtual machine stealing duty ratio. The disk IO indexes comprise at least one of the number of disks, the disk reading rate, the disk writing rate, the IO waiting time per second, the IO service time per second, the queue request number per second, the IO reading times per second and the IO writing times per second.

In the data transmission process of the computing nodes and the message middleware, the computing nodes send data in a single-node mode, so that each node has data of a plurality of corresponding subjects. In one implementation, each computing node sends log information, and the message middleware extracts topic data corresponding to each topic of the computing node from the log information. In another implementation, the compute node sends the message middleware directly in the data format of the topic data. A compute node may have multiple CPUs, each CPU (e.g., CPU0, CPU1, CPU2, etc.) has a complete set of subclasses. After the server starts the application program, the server subscribes the specified theme data in the specified message middleware service. For example, CPU theme data of the computing node 1 is specified.

According to one embodiment, after the subscribed topic data is received, the topic data read from the message middleware can be assembled into a canonical data set according to the defined field correspondence, so as to realize data formatting. The canonical data set is then stored in a data flow queue. The following is an example of a segment of CPU metrics with a message topic of topic: r.slurmccluster.load.cpu:

{

"cluster": "slurmcluster",

"subcid":"slurmcluster",

"hostid": "002aee7396f16544fd5b794c8bbd6a29",

"hostname": "slurm-node01",

"ip": "1.11.1.11",

"format": "1.0.0L",

"timestamp": 1588918080001,

"cpu": {

"count": 4,

"cpunum": 2,

"user": 0.32,

"nice": 0,

"sys": 0.63,

"idle": 99.03,

"iowait": 0,

"irq": 0,

"softirq": 0.02,

"steal": 0,

"guest": 0,

"guest_nice": 0

},

"cpu0": {

"user": 0.33,

"nice": 0,

"sys": 1.33,

"idle": 98.27,

"iowait": 0,

"irq": 0,

"softirq": 0.07,

"steal": 0,

"guest": 0,

"guest_nice": 0

},

"cpu1": {

"user": 0.4,

"nice":0,

"sys": 0.6,

"idle": 99,

"iowait": 0,

"irq": 0,

"softirq": 0,

"steal": 0,

"guest": 0,

"guest_nice": 0

}。

subsequently, in step S320, the theme data is temporarily stored in a preset data flow queue, and the theme data is extracted from the data flow queue item by using a preset filtering flow queue.

Fig. 4 is a schematic diagram of a data flow complete transmission queue according to an embodiment of the present invention, wherein a preset data flow queue RowFlow is included. The data flow queue is used for temporarily storing subscribed data, and avoids possible abnormal conditions caused by untimely processing of the application program under a large-batch data scene. According to different use scenes, different data output objects are predefined, and the data output objects comprise time sequence database output, search database output, alarm data output and aggregated data output. The first three are that the output data of the data flow queue RowFlow is directly used as source data and output to a corresponding output object; and the aggregated data output is to aggregate the source data and then output.

The time-series database output is, for example, the data output infiluxdbauutput of infiluxdb, the search database output is, for example, the data output esInOut of the ES system, the alarm data output is, for example, dispatchingnout, and the aggregation data output is, for example, midIO1 Out. In this way, the data in the data flow queue RowFlow is output to each predefined output object.

For the output object of the aggregated data in the third column, the present invention may create an initial queue midlO1 to perform a regular replacement on the data transmitted from the data stream queue, for example, to uniformly replace a certain type of words with canonical words for subsequent data processing and storage monitoring. After that, the invention creates the filter flow queue, which is used to play the role of a multithread producer in the whole data inflow process, and can gradually extract data from the data flow queue to form a buffer channel. And the data flow output of the filtering flow queue is processed by a filtering flow core FilterCore, the FilterCore plays a role of multithread consumption in the whole data inflow process, and the processing result is used as source data aggregated in different dimensions.

Subsequently, in step S330, the same topic data of the same computing node is aggregated according to at least one aggregation cycle, and an index statistic of the computing node in each aggregation cycle is computed.

Specifically, step S330 includes the steps of:

firstly, an aggregation flow queue and an aggregation flow core corresponding to each aggregation period are created, and the aggregation flow queue CollectFlow and the aggregation flow core Collectcore are numbered respectively according to the total class number of the aggregation period. The polymerization period may be 5min, 1h, 1day, etc., and may be configured according to the actual environment requirement, which is not limited by the present invention. If there are four aggregation cycles, the total number of classes is 4, and the aggregation flow queues and the aggregation flow cores are numbered in sequence, as numbered 0, 1, n, etc. in fig. 4. For example, aggregation period of 5min is the first aggregation flow queue, and 1h is the second aggregation flow queue.

And then, aggregating the same subject data of the same computing node according to the corresponding aggregation period by adopting the aggregation flow queue corresponding to each aggregation period, and computing the index statistic value of the computing node in each aggregation period by adopting the corresponding core flow queue.

Here, the same theme data of the same computing node after being processed by the filtering flow core is aggregated according to different aggregation cycles through each aggregation flow queue. Furthermore, an aggregation object midInOuts may be created before the aggregation stream queue to divide the data objects to be aggregated per batch according to the time stamp and the aggregation period. For example, midInOuts0 divides a batch of data every 5min, and all the data are processed in the aggregation flow queue after transmission; midInOuts1 divides a batch of data every 1h, and so on.

The index statistic value in the aggregation period comprises at least one of the maximum value, the minimum value, the average value and the total number of the same index in the aggregation period. Taking the CPU index as an example, the data flow of the computing node A in 5min has a plurality of pieces of theme data, and each piece of data has a set of subclass index values. And counting the statistical values such as the average value, the maximum value, the minimum value and the like within 5min by using the same subclass index. The source data is replaced by these statistics, i.e. 5min of data aggregation is completed. It should be understood that there may be multiple CPUs for the same compute node in the source data, and data aggregation may be implemented for the compute node hierarchy, that is, multiple similar indexes of the same node in an aggregation period; the method can also be realized according to the CPU level, namely, a plurality of similar indexes of the same CPU of the same node are aggregated.

And similarly, counting the index statistical values in other periods. Therefore, the same subject data of the same computing node is aggregated into a plurality of aggregated data according to different aggregation periods, so that the data storage capacity is reduced, and meanwhile, business personnel can conveniently check the aggregated data according to various dimensionality requirements.

The system records the initial time of the current aggregation record in the aggregation process, and judges whether each piece of data meets the requirement of the aggregation according to the aggregation period and the timestamp of each piece of data. For example, currently, 5min of aggregation data needs to be made, and the initial time is: 1589251020000 (2020/5/1210: 37: 0), if the data obtained is within 1589251320000 (2020/5/1210: 42: 0), then calculate directly to the average, maximum, minimum values at that point in time. If the timestamp of the next acquired record is greater than 1589251320000 (2020/5/1210: 42: 0), the aggregation calculation operation is stopped, the calculated aggregated data is prepared to be put in storage, and the next aggregation is continued, wherein the initial time is 1589251320000.

In the whole data processing flow, aiming at the data inflow and processing of each data stream node, the invention is regarded as the processing flow of a plurality of message producers- > RingBuffer- > a plurality of message consumers in RingBuffer. For large batch of data, efficient processing is performed by using a thread pool, taking 5-minute data aggregation as an example: after a data output object midInOuts5 of 5min is acquired, the object is used as a data flow object to flow into CollectFlow5, CollectFlow5 integrates data concurrently, and the data in the data buffer queue is continuously updated by combining with the consumption production and consumption mechanism of RingBuffer. For the data buffer queue, the core class CollectCore performs 5min aggregation according to the timestamp and the subject type of each piece of data: calculate the maximum, minimum, total, average over a specified time period. Until the timestamp of the next data exceeds 5 minutes. Of course, for each update, appropriate read-write lock operation is performed on the premise that thread security needs to be guaranteed.

Taking the theme of the CPU type as an example, according to the queue array of the CPU data in the time sequence, the average value avg, the maximum value max, the minimum value min, and the number cnt of each subclass index are counted among a plurality of CPUs in the same node, and the actual aggregate output format is as follows:

{

"cluster": "slurmcluster",

"cpu": {

"count": 4,

"cpunum": 2,

"cpunum-avg": 2,

"cpunum-cnt": 1,

"cpunum-max": 2,

"cpunum-min": 2,

"guest": 0,

"guest-avg": 0,

"guest-cnt": 1,

"guest-max": 0,

"guest-min": 0,

"guest_nice": 0,

"guest_nice-avg": 0,

"guest_nice-cnt": 1,

"guest_nice-max": 0,

"guest_nice-min": 0,

"idle": 98.77,

"idle-avg": 98.77,

"idle-cnt": 1,

"idle-max": 98.77,

"idle-min": 98.77,

……。

subsequently, in step S340, the index statistics of multiple aggregation cycles in the preset index period are summarized as the data index of the computing node in the preset index period, and stored in the data index database.

Here, the aggregation result of the aggregation flow core is used as data input, a corresponding processing result output flow StreamFlow is created, and is output to the unified Elasticsearch and infix db output objects, such as collecteseinuout and collectenfluxin, so as to prepare for data output and landing to the Elasticsearch and infix db. In summary, the present invention creates input stream CollectFlow, data processing core class CollectCore, output stream StreamFlow for the Elasticsearch database, infiluxdb database to output aggregate data result and put in storage according to different indexes.

According to one embodiment, the preset index period is 1 week when the aggregation period is 5 min; when the polymerization cycle is 1h, the preset index time period is 1 month; when the polymerization cycle is 1day, the preset index period is 1 year. Of course, other aggregation periods and corresponding preset index periods may also be set, and the present invention is not limited thereto. Different aggregation cycle systems automatically calculate index names from the time when the first piece of data is put in storage to the corresponding time interval. Real-time data generates an index for each topic type of data each day; aggregating five minutes of data per topic type, generating one index per week; aggregating one hour of data to generate one index per type, per month; aggregating data for one day generates an index per type per year.

FIG. 5 is an example of a segment of a data index generated. The computing node is cluster1, and the aggregation period is identified by taking seconds as a unit and comprises three types of 300s (5 min), 3600s (1 h) and 86400s (1 day). The theme type comprises indexes such as a disk loading rate, an NFS file system index, a Mysql database index and the like. From these classifications, a variety of file indexes are generated.

The process of storing the data to be written in the data stream into the database is similar to the process of writing into a relational database, after Json format data including index information are assembled, an http request is directly sent to a target ES server, and the request body of the request contains the data to be written. Then, performing subsequent operation according to the status code returned by the ES server, and if the writing is successful, continuing to write next time; if the writing fails, recording the failed data to a file, and combining a RingBuffer data structure to write the subsequent data.

According to an embodiment of the invention, the subject data read from the message middleware can be stored into the search database and the time sequence database respectively as source data. And then, monitoring whether the source data is abnormal or not in real time by using a monitoring queue, and sending an alarm indication when the source data is abnormal. Meanwhile, original data when the abnormity occurs can be marked, original records are provided for abnormity tracking, and meanwhile, data supplement and recording are facilitated.

Furthermore, the invention can also respectively calculate the input/output flow rate of the data flow in each queue, monitor the input/output flow rate in real time by adopting the monitoring queue or whether the index value in each aggregation cycle is abnormal, and send out an alarm indication when the abnormality occurs. The specific calculation logic includes: the data flow rate per unit time is calculated according to the input quantity and the output quantity of the data flow with the attributes thereof and the time variable, the default unit is byte/s, and the data flow rate can be configured in other units. Data condition monitoring may be performed based on the input/output flow rate of the data stream at each node. Usually, the data flow of the historical data or the historical index value of each aggregation cycle can be referred to for comparison to determine whether the data is abnormal. Fig. 6 is a data flow of each node aggregated, which corresponds to the processCollectFlow data flow, collect2ESFlow data flow, processcollecttonfluxflow data flow, and the like in fig. 3. The input/output parameter of each data stream is shown in FIG. 7, when the byte is 0byte/s, it represents that there is no data input or output currently.

According to the technical scheme of the invention, a convenient data processing mode is provided, the data of different service scenes are processed more efficiently on the premise of meeting the requirement of normal storage of the data, and the requirements of real-time performance and safety of acquisition and analysis of mass data are met. Compared with the traditional data processing mode, the streaming parallel data aggregation mode can realize real-time, rapid and efficient data aggregation, support the parallel processing of mass data stream queues in a large-scale cluster environment, and ensure that data aggregation operations with different dimensions are performed in parallel and independently. In addition, the invention monitors the processing process of each data stream in real time and displays logs by calculating the data stream summarizing input/output flow rate. Moreover, the invention depends on the design idea of the Disproptor, adopts a RingBuffer data structure, and stably finishes data aggregation under the conditions of high-speed reading and safe writing. Meanwhile, the invention keeps the stream data to the data file, provides the original record for the abnormal tracking and is convenient for data supplement and recording.

The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.

In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to perform the method of the invention according to instructions in said program code stored in the memory.

By way of example, and not limitation, readable media may comprise readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.

In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with examples of this invention. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.

As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense with respect to the scope of the invention, as defined in the appended claims.

Claims (13)

1. A data aggregation method adapted to be executed in a server communicatively coupled to message middleware and a data index repository, respectively, the message middleware communicatively coupled to at least one compute node, the method comprising the steps of:

reading topic data of each computing node for different topic types from the message middleware, wherein the topic data comprises a timestamp and at least one index value;

temporarily storing the theme data in a preset data flow queue, and extracting the theme data from the data flow queue one by adopting a preset filtering flow queue;

aggregating the same subject data of the same computing node according to at least one aggregation period, and computing the index statistic value of the computing node in each aggregation period;

summarizing the index statistics values of a plurality of aggregation cycles in a preset index time period, using the summary values as data indexes of the computing nodes in the preset index time period, and storing the data indexes into the data index database.

2. The method of claim 1, wherein the theme type comprises at least one of a CPU metric, a disk IO metric, a database metric, and a file system metric.

3. The method of claim 2, wherein the CPU metrics include at least one of a total number of CPU cores, a user mode duty ratio, a kernel mode duty ratio, an IO wait duty ratio, a hard interrupt duty ratio, a soft interrupt duty ratio, and a virtual machine stealing duty ratio.

4. The method of claim 2, wherein the disk IO metrics comprise at least one of a number of disks, a disk read rate, a disk write rate, an IO latency per second, an IO service time per second, a number of queue requests per second, a number of IO reads per second, and a number of IO writes per second.

5. The method of claim 1, wherein the statistical value of the index in the aggregation period comprises at least one of a maximum value, a minimum value, an average value and a total number of the same index in the aggregation period.

6. The method of any one of claims 1-5, wherein the step of aggregating the same subject data of the same computing node according to at least one aggregation cycle and calculating the index statistics of the computing node in each aggregation cycle comprises:

establishing an aggregation flow queue and an aggregation flow core corresponding to each aggregation period, and numbering the aggregation flow queue and the aggregation flow core respectively according to the total class number of the aggregation period; and

and aggregating the same subject data of the same computing node according to the corresponding aggregation period by adopting the aggregation flow queue corresponding to each aggregation period, and computing the index statistical value of the computing node in each aggregation period by adopting the corresponding core flow queue.

7. The method of claim 1, wherein,

when the polymerization cycle is 5min, the preset index time period is 1 week;

when the polymerization cycle is 1h, the preset index time period is 1 month;

when the aggregation cycle is 1day, the preset index period is 1 year.

8. The method of claim 1, wherein the step of temporarily storing the subject data in a preset data flow queue comprises:

and according to a preset field corresponding relation, assembling the subject data read from the message middleware into a standard data set and storing the standard data set into the data stream queue.

9. The method of claim 1, wherein the database into which the data index is stored comprises a search database and a timing database, the method further comprising the steps of:

respectively storing the subject data read from the message middleware as source data into the search database and the time sequence database;

and monitoring whether the source data is abnormal or not in real time by adopting a monitoring queue, and sending an alarm indication when the source data is abnormal.

10. The method of claim 1, further comprising the steps of:

respectively calculating the input/output flow rate of the data flow in each queue;

and monitoring whether the input/output flow rate or the index value in each aggregation period is abnormal or not in real time by adopting a monitoring queue, and sending an alarm indication when the abnormality occurs.

11. The method of claim 1, wherein the subject data employs a RingBuffer data structure.

12. A server, comprising:

at least one processor; and

at least one memory including computer program instructions;

the at least one memory and the computer program instructions are configured to, with the at least one processor, cause the server to perform the method of any of claims 1-11.

13. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a server, cause the server to perform any of the methods of claims 1-11.

Images