CN113791955A - Data aggregation device and method for monitoring system and server - Google Patents
Data aggregation device and method for monitoring system and server Download PDFInfo
- Publication number
- CN113791955A CN113791955A CN202111093632.3A CN202111093632A CN113791955A CN 113791955 A CN113791955 A CN 113791955A CN 202111093632 A CN202111093632 A CN 202111093632A CN 113791955 A CN113791955 A CN 113791955A
- Authority
- CN
- China
- Prior art keywords
- data
- aggregation
- data aggregation
- module
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002776 aggregation Effects 0.000 title claims abstract description 245
- 238000004220 aggregation Methods 0.000 title claims abstract description 245
- 238000012544 monitoring process Methods 0.000 title claims abstract description 58
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 72
- 230000002688 persistence Effects 0.000 claims abstract description 23
- 238000001914 filtration Methods 0.000 claims description 25
- 230000004044 response Effects 0.000 claims description 14
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000001419 dependent effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
Abstract
The invention discloses a data aggregation device, a method and a server for a monitoring system, wherein the device comprises the following steps: rule engine module, data assemble processing module and storage module, wherein: the rule engine module is used for configuring data aggregation rules; the data aggregation processing module comprises a plurality of data aggregation channels, and is used for acquiring data, transmitting the data to the corresponding data aggregation channels, processing the data based on the data aggregation rules configured by the rule engine module, and outputting an aggregation result; the data aggregation processing module is further configured to determine whether a convergence result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to the other data aggregation channels, and store the convergence result in a cache layer or a persistence layer of the storage module based on the determination result. According to the invention, the data aggregation rule and the data aggregation channel can be conveniently and dynamically adjusted, the data aggregation processing speed is accelerated, and the data aggregation processing efficiency is improved.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data aggregation device, a data aggregation method and a server for a monitoring system.
Background
In the big data era, as the scale of the data center equipment increases and as the time goes on, the monitoring data volume of the equipment also becomes larger and larger, the diversity of data processing also becomes larger and larger, a convergence dimension is generally required to be added to revise the data convergence logic, and the efficiency of data processing is greatly reduced.
Disclosure of Invention
In view of this, the invention provides a data aggregation device, a data aggregation method and a server for a monitoring system, which can adjust data aggregation conveniently and dynamically, accelerate the processing of data aggregation and improve the data aggregation efficiency.
Based on the above object, an aspect of the embodiments of the present invention provides a data aggregation device for a monitoring system, where the data aggregation device specifically includes: rule engine module, data assemble processing module and storage module, wherein:
the rule engine module is used for configuring data aggregation rules;
the data aggregation processing module comprises a plurality of data aggregation channels, is used for acquiring data, transmitting the data to the corresponding data aggregation channels, processing the data based on the data aggregation rules configured by the rule engine module, and outputting an aggregation result;
the data aggregation processing module is further configured to determine whether an aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to the other data aggregation channels, and store the aggregation result in a cache layer or a persistence layer of the storage module based on the determination result.
In some embodiments, the data aggregation processing module includes a determining submodule, and the determining submodule is configured to store the aggregation result in the cache layer in response to that a dependency relationship exists between the aggregation result output by the corresponding data aggregation channel and data transmitted to the other data aggregation channels.
In some embodiments, the determining sub-module is further configured to store the aggregation result in the persistence layer in response to that the aggregation result output by the corresponding data aggregation channel does not have a dependency relationship with data transmitted to the remaining data aggregation channels.
In some embodiments, the data aggregation device further comprises: and the data receiving module is used for receiving the monitoring data, filtering the monitoring data and adding the filtered monitoring data into the buffer queue.
In some embodiments, the data aggregation processing module is configured to obtain the filtered monitoring data from the data receiving module or obtain the aggregation result from the cache layer.
In some embodiments, the data receiving module is configured to configure a data filtering rule, and perform filtering processing on the received monitoring data based on the data filtering rule.
In another aspect of the embodiments of the present invention, a data aggregation method for a monitoring system is further provided, including:
configuring a data aggregation rule based on the rule engine module;
acquiring data based on a data aggregation processing module, transmitting the data to corresponding data aggregation channels in a plurality of data aggregation channels of the data aggregation processing module, processing the data based on a data aggregation rule configured by a rule engine module, and outputting an aggregation result;
and judging whether the aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to other data aggregation channels based on a data aggregation processing module, and storing the aggregation result into a cache layer or a persistence layer of the storage module based on the judgment result.
In some embodiments, storing the aggregated result to the caching layer or the persistence layer based on the determination comprises:
in response to that the aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to other data aggregation channels, storing the aggregation result in the cache layer;
and in response to that the aggregation result output by the corresponding data aggregation channel has no dependency relationship with the data transmitted to the other data aggregation channels, storing the aggregation result in the persistence layer.
In some embodiments, the method further comprises:
and receiving the monitoring data based on a data receiving module, filtering the monitoring data, and adding the filtered monitoring data into a buffer queue.
In another aspect of the embodiments of the present invention, a server is further provided, which includes the data aggregation device for a monitoring system according to the present invention.
The invention has at least the following beneficial technical effects: by the scheme of the invention, when data processing is converged, the data convergence rule and the data convergence channel can be conveniently and dynamically adjusted, so that the convergence processing of the data is accelerated; and the storage module is divided into a cache layer and a persistence layer, so that the processed data and the convergence result dependent data are cached and stored persistently according to the data convergence processing result, the data convergence processing efficiency is accelerated, and the data convergence efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of an embodiment of a data aggregation device for a monitoring system according to the present invention;
fig. 2 is a schematic structural diagram of a data receiving module provided in the present invention;
FIG. 3 is a schematic structural diagram of a data aggregation device for a monitoring system according to another embodiment of the present invention;
FIG. 4 is a block diagram of an embodiment of a data aggregation method for a monitoring system according to the present invention;
fig. 5 is a schematic diagram of an embodiment of a server provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In view of the above, a first aspect of the embodiments of the present invention provides a data aggregation device for a monitoring system. As shown in fig. 1, the data aggregation device specifically includes: a rule engine module 110, a data aggregation processing module 120, and a storage module 130, wherein:
the rule engine module 110 is used to configure data aggregation rules.
By setting the convergence channel, different dimensionalities of data can be processed, and cooperative data processing can also be performed by combining processing results of a plurality of different convergence channels;
the data aggregation processing module 120 includes a plurality of data aggregation channels, and the data aggregation processing module 120 is configured to acquire data, transmit the data to a corresponding data aggregation channel, process the data based on a data aggregation rule configured by the rule engine module 110, and output an aggregation result.
By setting the convergence channel, different dimensionalities of data can be processed.
The data aggregation processing module 120 is further configured to determine whether an aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to the other data aggregation channels, and store the aggregation result in a cache layer or a persistent layer of the storage module 130 based on the determination result.
The storage module is divided into the cache layer and the persistence layer, so that the processed data and the convergence result dependent data are cached and stored persistently according to the data convergence processing result, and the efficiency of data convergence processing is improved.
For a better understanding of the embodiments of the present invention, the following data aggregation scenario of CPU utilization is illustrated.
A data aggregation Rule is configured through the Rule engine module 110, the Rule is to average 60 times of input data, and the Rule is named Rule 1;
two data aggregation channels are created by the data aggregation processing module 120.
The data aggregation channel 1 is named Pipeline1, data input is set to be 1 parameter, parameter values are CPU utilization rate performance data of a receiving module, and an aggregation Rule with the aggregation Rule name of Rule1 is associated;
the data aggregation channel 2 is named Pipeline2, the data input is set to be 1 parameter, the parameter value is the output value of Pipeline1, and the associated aggregation Rule is the aggregation Rule of Rule 1.
After the data aggregation module 120 obtains the CPU utilization data, the CPU utilization data is transmitted to a data aggregation channel named Pipeline1, and the Rule engine module 110 associates the aggregation Rule1, so that after 60 times of transmission of the CPU utilization data to the data aggregation channel, the data aggregation channel obtains an aggregation result and outputs the aggregation result.
In the second-level monitoring system, the reporting frequency of data is 1 second and 1 time, and the average utilization rate of the CPU in the minute level can be obtained through the aggregation configuration.
The data aggregation processing module 120 determines that the aggregation output result of the data aggregation channel 1 is the dependent data of the data aggregation channel 2, and thus stores the aggregation result in the cache layer of the storage module 130 for standby.
The data aggregation module 120 obtains the average usage rate of the CPU at the minute level output by the data aggregation channel 1, transmits the average usage rate of the CPU at the minute level to the data aggregation channel named Pipeline2, and associates the aggregation Rule1 with the Rule engine module 110, so that after 60-time transmission of the CPU usage rate data at the minute level to the data aggregation channel 2, the data aggregation channel 2 obtains an aggregation result and outputs the aggregation result, that is, the average usage rate aggregation result of the CPU at the hour level is obtained.
The data aggregation module 120 obtains data from two sources, one is obtained from the monitoring system, for example, the CPU utilization data transmitted to the data aggregation channel Pipeline1 in the above example; one is the aggregate result output by the other data aggregation channels, for example, the average of the 60 CPU utilization data output by the data aggregation channel Pipeline1 in the above example.
For aggregation results with dependency relationship, the aggregation results may be cached in the cache layer of the storage module 130 first, and obtained from the cache layer when waiting for the data aggregation processing module 120 to use; or directly output to the corresponding data gathering channel, and use after the data gathering channel triggers the associated rule.
By the embodiment of the invention, when data processing is converged, the data convergence rule and the data convergence channel can be conveniently and dynamically adjusted, the convergence processing of the data is accelerated, and the data convergence efficiency is improved.
The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions described herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.
In some embodiments, the data aggregation processing module includes a determining submodule, and the determining submodule is configured to store the aggregation result in the cache layer in response to that a dependency relationship exists between the aggregation result output by the corresponding data aggregation channel and data transmitted to the other data aggregation channels.
In some embodiments, the determining sub-module is further configured to store the aggregation result in the persistence layer in response to that the aggregation result output by the corresponding data aggregation channel does not have a dependency relationship with data transmitted to the remaining data aggregation channels.
The storage module for storing data generally selects a database, if the data is stored in the database for storage after being sorted, the data is inquired from the database when the data is required to be used, the time for inquiring the data is required, and particularly, the data acquisition time is prolonged when the data stored in the database is large or the data inquiry operation is frequently performed, so that the data is reduced. Therefore, the storage modules are layered, and two databases are established, wherein one database is used as a cache layer of the cache module, and the other database is used as a persistence layer of the storage module. The data to be used or the convergence result is changed by the cache layer, so that the query to the database can be reduced, the acquisition of the data is accelerated, and the data convergence efficiency is improved.
In some embodiments, the data aggregation device further comprises: and the data receiving module is used for receiving the monitoring data, filtering the monitoring data and adding the filtered monitoring data into the buffer queue.
In some embodiments, the data aggregation processing module is configured to obtain the filtered monitoring data from the data receiving module or obtain the aggregation result from the cache layer.
In some embodiments, the data receiving module is configured to configure a data filtering rule, and perform filtering processing on the received monitoring data based on the data filtering rule.
Fig. 2 is a schematic structural diagram of a data receiving module.
The data receiving module receives monitoring data of hardware equipment (such as hardware on board a server, such as a CPU, a hard disk, a fan and the like) through a receiving interface, calls a data filtering rule through a filter, filters invalid data which are abnormal and do not conform to a processing format, and inputs the filtered monitoring data into a queue to buffer the data.
The data filtering rules may include: a threshold range rule, wherein numerical data which are not in the range are filtered through the threshold range rule; format validity rules, filtering data in band units, filtering specified symbols, and the like.
Filtering different types of data through configured data filtering rules, wherein the filtering function is to carry out prepositive data verification processing to prevent invalid data in abnormal and non-conforming processing formats from being carried out in a subsequent data aggregation module; and data buffering is carried out on the data through the queue, so that instantaneous high concurrent data receiving can be prevented, and the usability of the data receiving module is improved.
Several embodiments of the present invention are described below with reference to specific examples.
Fig. 3 is a schematic diagram of another embodiment of a data aggregation device for a monitoring system.
The data aggregation device includes: the system comprises a data receiving module, a data aggregation processing module, a rule engine module and a storage module, wherein the storage module comprises a cache layer and a persistence layer.
And the data receiving module receives the monitoring data based on the data receiving module, filters the monitoring data and adds the filtered monitoring data into the buffer queue.
The data aggregation processing module acquires the monitoring data from the cache queue and transmits the monitoring data to the corresponding data aggregation channel, and the data aggregation channel processes the monitoring data based on the data aggregation rule configured by the rule engine module and outputs an aggregation result;
and judging whether the aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to other data aggregation channels based on a data aggregation processing module, and storing the aggregation result into a cache layer or a persistence layer of the storage module based on the judgment result.
By the embodiment of the invention, when data processing is converged, the data convergence rule and the data convergence channel can be conveniently and dynamically adjusted, so that the convergence processing of the data is accelerated; and the storage module is divided into a cache layer and a persistence layer, so that the processed data and the convergence result dependent data are cached and stored persistently according to the data convergence processing result, the data convergence processing efficiency is accelerated, and the data convergence efficiency is improved.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 4, an embodiment of the present invention further provides a data aggregation method for a monitoring system, including:
and S101, configuring a data aggregation rule based on the rule engine module.
Step S103, acquiring data based on a data aggregation processing module, transmitting the data to corresponding data aggregation channels in a plurality of data aggregation channels of the data aggregation processing module, processing the data based on a data aggregation rule configured by the rule engine module, and outputting an aggregation result;
and step S105, judging whether the aggregation result output by the corresponding data aggregation channel has a dependency relationship with the data transmitted to the other data aggregation channels based on the data aggregation processing module, and storing the aggregation result in a cache layer or a persistence layer of the storage module based on the judgment result.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like. Embodiments of the computer program may achieve the same or similar effects as any of the preceding method embodiments to which it corresponds.
The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
In some embodiments, storing the aggregated result to the caching layer or the persistence layer based on the determination comprises:
in response to that the aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to other data aggregation channels, storing the aggregation result in the cache layer;
and in response to that the aggregation result output by the corresponding data aggregation channel has no dependency relationship with the data transmitted to the other data aggregation channels, storing the aggregation result in the persistence layer.
In some embodiments, the method further comprises:
and receiving the monitoring data based on a data receiving module, filtering the monitoring data, and adding the filtered monitoring data into a buffer queue.
The embodiment of the invention also can comprise corresponding computer equipment. The computer device comprises a memory, at least one processor and a computer program stored on the memory and executable on the processor, the processor executing the program to perform the method according to the invention.
In a second aspect of the embodiment of the present invention, a server 11 is provided. Fig. 5 is a schematic diagram of an embodiment of a server provided by the present invention. As shown in fig. 5, the server 11 includes a data aggregation device 12 for monitoring a system as follows.
The data aggregation device 12 specifically includes: rule engine module, data assemble processing module and storage module, wherein:
the rule engine module is used for configuring data aggregation rules;
the data aggregation processing module comprises a plurality of data aggregation channels, is used for acquiring data, transmitting the data to the corresponding data aggregation channels, processing the data based on the data aggregation rules configured by the rule engine module, and outputting an aggregation result;
the data aggregation processing module is further configured to determine whether an aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to the other data aggregation channels, and store the aggregation result in a cache layer or a persistence layer of the storage module based on the determination result.
In some embodiments, the data aggregation processing module includes a determining submodule, and the determining submodule is configured to store the aggregation result in the cache layer in response to that a dependency relationship exists between the aggregation result output by the corresponding data aggregation channel and data transmitted to the other data aggregation channels.
In some embodiments, the determining sub-module is further configured to store the aggregation result in the persistence layer in response to that the aggregation result output by the corresponding data aggregation channel does not have a dependency relationship with data transmitted to the remaining data aggregation channels.
In some embodiments, the data aggregation device further comprises: and the data receiving module is used for receiving the monitoring data, filtering the monitoring data and adding the filtered monitoring data into the buffer queue.
In some embodiments, the data aggregation processing module is configured to obtain the filtered monitoring data from the data receiving module or obtain the aggregation result from the cache layer.
In some embodiments, the data receiving module is configured to configure a data filtering rule, and perform filtering processing on the received monitoring data based on the data filtering rule.
In the context of the present invention, the memory, as a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the network resource coordination method in the embodiments of the present application. The processor executes various functional applications and data processing of the device by running nonvolatile software programs, instructions and modules stored in the memory, that is, the data aggregation method of the above method embodiments is realized.
The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the device, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and such remote memory may be coupled to the local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Finally, it should be noted that, as will be understood by those skilled in the art, all or part of the processes of the methods of the above embodiments may be implemented by a computer program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A data aggregation device for a monitoring system, comprising: rule engine module, data assemble processing module and storage module, wherein:
the rule engine module is used for configuring data aggregation rules;
the data aggregation processing module comprises a plurality of data aggregation channels, is used for acquiring data, transmitting the data to the corresponding data aggregation channels, processing the data based on the data aggregation rules configured by the rule engine module, and outputting an aggregation result;
the data aggregation processing module is further configured to determine whether an aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to the other data aggregation channels, and store the aggregation result in a cache layer or a persistence layer of the storage module based on the determination result.
2. The apparatus according to claim 1, wherein the data aggregation processing module includes a determining submodule, and the determining submodule is configured to store the aggregation result in the cache layer in response to that the aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to the other data aggregation channels.
3. The apparatus of claim 2, wherein the determining sub-module is further configured to store the aggregation result in the persistence layer in response to that the aggregation result output by the corresponding data aggregation channel has no dependency relationship with data transmitted to the remaining data aggregation channels.
4. The apparatus of claim 1, further comprising: and the data receiving module is used for receiving the monitoring data, filtering the monitoring data and adding the filtered monitoring data into the buffer queue.
5. The apparatus according to claim 4, wherein the data aggregation processing module is configured to obtain the filtered monitoring data from the data receiving module or obtain the aggregation result from the cache layer.
6. The device of claim 4, wherein the data receiving module is configured to configure a data filtering rule, and filter the received monitoring data based on the data filtering rule.
7. A data aggregation method for a monitoring system, comprising:
configuring a data aggregation rule based on the rule engine module;
acquiring data based on a data aggregation processing module, transmitting the data to corresponding data aggregation channels in a plurality of data aggregation channels of the data aggregation processing module, processing the data based on a data aggregation rule configured by a rule engine module, and outputting an aggregation result;
and judging whether the aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to other data aggregation channels based on a data aggregation processing module, and storing the aggregation result into a cache layer or a persistence layer of the storage module based on the judgment result.
8. The method of claim 7, wherein storing the aggregated result to the caching layer or the persistence layer based on the determination comprises:
in response to that the aggregation result output by the corresponding data aggregation channel has a dependency relationship with data transmitted to other data aggregation channels, storing the aggregation result in the cache layer;
and in response to that the aggregation result output by the corresponding data aggregation channel has no dependency relationship with the data transmitted to the other data aggregation channels, storing the aggregation result in the persistence layer.
9. The method of claim 7, further comprising:
and receiving the monitoring data based on a data receiving module, filtering the monitoring data, and adding the filtered monitoring data into a buffer queue.
10. A server, characterized in that it comprises a data aggregation device for monitoring systems according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111093632.3A CN113791955A (en) | 2021-09-17 | 2021-09-17 | Data aggregation device and method for monitoring system and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111093632.3A CN113791955A (en) | 2021-09-17 | 2021-09-17 | Data aggregation device and method for monitoring system and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113791955A true CN113791955A (en) | 2021-12-14 |
Family
ID=78878806
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111093632.3A Pending CN113791955A (en) | 2021-09-17 | 2021-09-17 | Data aggregation device and method for monitoring system and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113791955A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037729A (en) * | 2022-04-21 | 2022-09-09 | 中国建设银行股份有限公司 | Data aggregation method and device, electronic equipment and computer readable medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101491055A (en) * | 2006-07-17 | 2009-07-22 | 国际商业机器公司 | Dispatching request fragments from a response aggregating surrogate |
| CN102209118A (en) * | 2011-06-10 | 2011-10-05 | 成都勤智数码科技有限公司 | Distributed mass data gathering method |
| CN103810258A (en) * | 2014-01-26 | 2014-05-21 | 杭州东方通信软件技术有限公司 | Data aggregation scheduling method based on data warehouse |
| CN108804347A (en) * | 2017-05-05 | 2018-11-13 | 华中科技大学 | A kind of cache layer, collecting system and method for industrial big data convergence |
| WO2020078381A1 (en) * | 2018-10-16 | 2020-04-23 | 杭州海康威视数字技术股份有限公司 | Data aggregation method, device, equipment, storage medium and system |
| CN111460495A (en) * | 2020-03-27 | 2020-07-28 | 北京锐安科技有限公司 | Data grading management system and method |
| US20200311051A1 (en) * | 2019-03-25 | 2020-10-01 | Hitachi, Ltd. | Data linkage management method, data linkage management system, and node |
| CN111858739A (en) * | 2020-08-04 | 2020-10-30 | 浪潮卓数大数据产业发展有限公司 | Mapreduce-based data aggregation method and system |
-
2021
- 2021-09-17 CN CN202111093632.3A patent/CN113791955A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101491055A (en) * | 2006-07-17 | 2009-07-22 | 国际商业机器公司 | Dispatching request fragments from a response aggregating surrogate |
| CN102209118A (en) * | 2011-06-10 | 2011-10-05 | 成都勤智数码科技有限公司 | Distributed mass data gathering method |
| CN103810258A (en) * | 2014-01-26 | 2014-05-21 | 杭州东方通信软件技术有限公司 | Data aggregation scheduling method based on data warehouse |
| CN108804347A (en) * | 2017-05-05 | 2018-11-13 | 华中科技大学 | A kind of cache layer, collecting system and method for industrial big data convergence |
| WO2020078381A1 (en) * | 2018-10-16 | 2020-04-23 | 杭州海康威视数字技术股份有限公司 | Data aggregation method, device, equipment, storage medium and system |
| US20200311051A1 (en) * | 2019-03-25 | 2020-10-01 | Hitachi, Ltd. | Data linkage management method, data linkage management system, and node |
| CN111460495A (en) * | 2020-03-27 | 2020-07-28 | 北京锐安科技有限公司 | Data grading management system and method |
| CN111858739A (en) * | 2020-08-04 | 2020-10-30 | 浪潮卓数大数据产业发展有限公司 | Mapreduce-based data aggregation method and system |
Non-Patent Citations (1)
| Title |
|---|
| 朱智俊;乐孜纯;: "光突发交换中的一种自适应汇聚策略", 光学精密工程, no. 01, 15 January 2010 (2010-01-15) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037729A (en) * | 2022-04-21 | 2022-09-09 | 中国建设银行股份有限公司 | Data aggregation method and device, electronic equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021012568A1 (en) | Data processing method and related device | |
| KR20150016207A (en) | Service process control method and network device | |
| CN111143371A (en) | Data query method, device, equipment, system and medium | |
| CN110198251B (en) | Method and device for obtaining client address | |
| CN110099112B (en) | Data storage method, device, medium and terminal equipment based on point-to-point network | |
| CN113791955A (en) | Data aggregation device and method for monitoring system and server | |
| CN110311963B (en) | Message pushing method and device, computer equipment and computer readable storage medium | |
| CN111538572A (en) | Task processing method, device, scheduling server and medium | |
| CN109067744B (en) | ACL rule processing method, device and communication equipment | |
| CN112073329B (en) | Distributed current limiting method and device, electronic equipment and storage medium | |
| CN108463813B (en) | Method and device for processing data | |
| US11455293B2 (en) | Counting method, counter and storage medium | |
| CN113438169A (en) | Data scheduling method, electronic equipment and storage medium | |
| CN109327404B (en) | P2P prediction method and system based on naive Bayes classification algorithm, server and medium | |
| CN113835905B (en) | Message queue load balancing method and device, electronic equipment and medium | |
| CA3022435A1 (en) | Adaptive event aggregation | |
| CN109508433B (en) | Load fluctuation coping method and system based on performance adjustment of matching algorithm | |
| CN114071448B (en) | Data transmission method, related network node and storage medium | |
| CN109756386B (en) | Communication optimization method and device based on cloud network vehicle cooperation | |
| US10116698B1 (en) | Managing network firewall configuration utilizing source lists | |
| CN110071828A (en) | Alarm method, device, equipment and storage medium | |
| WO2018218871A1 (en) | Attribute access control method, device and vehicle-mounted control system | |
| CN110769290B (en) | Play event updating method and system and computing device | |
| CN113542012B (en) | Fault detection method, fault detection device and electronic equipment | |
| CN112003935B (en) | Connection number acquisition method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |