CN109558403B - Data aggregation method and device, computer device and computer readable storage medium - Google Patents
Data aggregation method and device, computer device and computer readable storage medium Download PDFInfo
- Publication number
- CN109558403B CN109558403B CN201811143535.9A CN201811143535A CN109558403B CN 109558403 B CN109558403 B CN 109558403B CN 201811143535 A CN201811143535 A CN 201811143535A CN 109558403 B CN109558403 B CN 109558403B
- Authority
- CN
- China
- Prior art keywords
- data
- information
- aggregation
- script
- sorting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002776 aggregation Effects 0.000 title claims abstract description 255
- 238000004220 aggregation Methods 0.000 title claims abstract description 254
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000013515 script Methods 0.000 claims abstract description 140
- 238000004364 calculation method Methods 0.000 claims abstract description 75
- 238000004590 computer program Methods 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 9
- 238000011161 development Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012935 Averaging Methods 0.000 description 4
- 238000006116 polymerization reaction Methods 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000013497 data interchange Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Stored Programmes (AREA)
Abstract
A method of data aggregation, comprising: acquiring a data aggregation script; loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script; acquiring original data from a designated data source according to the data acquisition information; sorting the original data according to the data sorting information to obtain sorted data; calculating the sorted data according to the data calculation information to obtain an aggregation result; and outputting the aggregation result according to the data output information. The invention also provides a data aggregation device, a computer device and a computer readable storage medium. The invention realizes high-efficiency data aggregation.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data aggregation method and apparatus, a computer apparatus, and a computer readable storage medium.
Background
In application system monitoring, the original data reported by the acquisition end is huge and is not suitable for direct use and storage. When the data background processes the original data, the data needs to be subjected to aggregation processing to generate aggregated data. However, the existing data aggregation method is low in efficiency, and after new aggregation requirements or adjustment of the requirements exist, the data background needs to be developed again.
Disclosure of Invention
In view of the foregoing, there is a need for a data aggregation method and apparatus, a computer apparatus, and a computer-readable storage medium that can achieve efficient data aggregation.
A first aspect of the present application provides a data aggregation method, the method comprising:
acquiring a data aggregation script;
loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script;
acquiring original data from a designated data source according to the data acquisition information;
sorting the original data according to the data sorting information to obtain sorted data;
calculating the sorted data according to the data calculation information to obtain an aggregation result;
and outputting the aggregation result according to the data output information.
In another possible implementation, the data aggregation script includes data in JSON format, through which data aggregation requirements are described.
In another possible implementation, the data aggregation script includes an ES script.
In another possible implementation manner, the parsing the data acquisition information, the data sorting information, the data calculation information and the data output information from the data aggregation script includes:
Analyzing the data acquisition information, the data sorting information, the data calculation information and the data output information from the data aggregation script according to preset keywords; or alternatively
And analyzing the data acquisition information, the data sorting information, the data calculation information and the data output information from the data aggregation script according to the structure of the data aggregation script.
In another possible implementation manner, the parsing the data acquisition information, the data sorting information, the data calculation information and the data output information from the data aggregation script includes:
analyzing a query part, an agg part, an index part and a type part from the data aggregation script;
analyzing an original data acquisition field, an original data sorting field and an original data calculation field from the query part;
the original data acquisition field, the index part and the type part are used as the data acquisition information, the original data sorting field is used as the data sorting information, the original data calculation field is used as the data calculation information, and the agg part is used as the data output information.
In another possible implementation manner, the method further includes:
The sorted data is filtered before being calculated.
In another possible implementation manner, the method further includes:
and deleting the original data after outputting the aggregation result.
A second aspect of the present application provides a data aggregation apparatus, the apparatus comprising:
the script acquisition unit is used for acquiring a data aggregation script;
the analysis unit is used for loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script;
a data acquisition unit for acquiring original data from a specified data source according to the data acquisition information;
the sorting unit is used for sorting the original data according to the data sorting information to obtain sorted data;
the calculation unit is used for calculating the sorted data according to the data calculation information to obtain an aggregation result;
and the output unit is used for outputting information according to the data and outputting the aggregation result.
A third aspect of the present application provides a computer apparatus comprising a processor for implementing the data aggregation method when executing a computer program stored in a memory.
A fourth aspect of the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data aggregation method.
The method comprises the steps of obtaining a data aggregation script; loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script; acquiring original data from a designated data source according to the data acquisition information; sorting the original data according to the data sorting information to obtain sorted data; calculating the sorted data according to the data calculation information to obtain an aggregation result; and outputting the aggregation result according to the data output information. According to the method and the device, data aggregation is carried out according to the aggregation script, when the aggregation requirement is newly increased or modified, only configuration in the script is required to be modified, code development is not required to be carried out, a server is not required to be updated, development manpower and cost are saved, and high-efficiency data aggregation is realized.
Drawings
Fig. 1 is a flowchart of a data aggregation method according to an embodiment of the present invention.
Fig. 2 is a block diagram of a data aggregation apparatus according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a computer device according to an embodiment of the present invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will be more clearly understood, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, and the described embodiments are merely some, rather than all, embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Preferably, the data aggregation method of the present invention is applied in one or more computer devices. The computer apparatus is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing equipment. The computer device can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.
Example 1
Fig. 1 is a flowchart of a data aggregation method according to an embodiment of the present invention. The data aggregation method is applied to a computer device. The data aggregation method aggregates the data of the designated data source according to the data aggregation script.
As shown in fig. 1, the data aggregation method specifically includes the following steps:
and step 101, acquiring a data aggregation script.
The pre-stored data aggregation script may be read. For example, data aggregation scripts for different data aggregation requirements may be pre-programmed and stored in a memory of the computing device, from which the required data aggregation scripts are read as needed.
Alternatively, a user-edited data aggregation script may be received. For example, a script editing interface may be provided from which a user-entered data aggregation script is received.
The data aggregation script describes data aggregation requirements.
The data aggregation script may be one or a plurality of data aggregation scripts. Each data aggregation script describes a separate data aggregation requirement. Different data aggregation requirements correspond to different data aggregation scripts.
For example, in an application scenario, the data aggregation method is used to aggregate the monitoring data of each application program. The monitoring data comprise the memory occupation amount, the frame rate, the breakdown times, the starting times and the like of the application program.
In the application scenario, a data aggregation script may describe a data aggregation requirement of memory occupation amounts of application programs, and calculate average values of memory occupation amounts of all application programs in a certain period of time, so as to obtain average memory occupation amounts in the period of time, where the average memory occupation amounts are the aggregation results as index data of memory occupation.
In the application scenario, another data aggregation script may describe a data aggregation requirement of a frame rate (FPS) of an application program, calculate an average value of the frame rates of all application programs in a certain period of time, and obtain an average frame rate in the period of time, as index data of the frame rate, where the average frame rate is an aggregation result.
In the application scenario, another data aggregation script may describe a data aggregation requirement of a crash rate of an application program, and divide the crash frequency of the application program in a certain period of time by the start frequency in the period of time to obtain the crash rate of the period of time, where the crash rate is a result of aggregation.
The data aggregation script may include data in JSON (JavaScript Object Notation, JS object profile) format by which data aggregation requirements are described. JSON is a lightweight data interchange format that stores and represents data in a text format that is completely independent of the programming language, and any JavaScript supported type can be represented by JSON, such as strings, numbers, objects, arrays, etc.
The data aggregation script may include ES (ElasticSearch) script. The data aggregation script takes the ES script as a part of the ES script, expands the ES script, and adds other fields for controlling the data aggregation operation.
The ES is a Lucene-based search server. It provides a distributed multi-user capable full text search engine based on RESTful web interface. The elastic search is developed by Java and is issued as an open source code under Apache license terms, so that real-time search can be achieved, and the method has the advantages of stability, reliability, rapidness and convenience in installation and use.
The data aggregation requirement includes data such as a data source (i.e., a source of original data), a type of the original data, a type of the aggregated data, a data aggregation mode (i.e., how to aggregate the original data), and the like.
Step 102, loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script.
The data aggregation script may be loaded into a memory of a computing device from which various information/data controlling the aggregation of data may be parsed.
In a specific embodiment, the data aggregation script includes query, agg, index, type parts, where:
the query part represents an ES script, and the ES script can be directly executed by an ES server, which defines the aggregate requirement and comprises an original data acquisition field, an original data sorting field and an original data calculation field;
the agg part defines the type of data storage after aggregation;
the index part defines which index the original data is acquired from and is used for acquiring the original data;
the type section defines the type of the original data in the index for acquiring the original data.
In this particular embodiment, portions query, agg, index, type may be parsed from the data aggregation script; the method comprises the steps of analyzing an original data acquisition field, an original data sorting field and an original data calculation field from a query part, taking the original data acquisition field, an index part and a type part as data acquisition information, taking the original data sorting field as data sorting information, taking the original data calculation field as data calculation information, and taking an agg part as data output information.
The data acquisition information, the data sorting information, the data calculation information and the data output information can be analyzed from the data aggregation script according to preset keywords (for example query, agg, index, type).
Or, the data acquisition information, the data sorting information, the data calculation information and the data output information can be analyzed from the data aggregation script according to the structure of the data aggregation script. For example, the data aggregation script is written according to a preset structure (for example, the data acquisition information in the data aggregation script is a first code segment, the data sorting information is a second code segment, the data calculation information is a third code segment, and the data output information is a fourth code segment), and the data acquisition information, the data sorting information, the data calculation information and the data output information are determined according to the preset structure.
The data acquisition information is used to obtain specified raw data from a specified data source.
The data sorting information is used for sorting the original data to obtain the sorted original data. Sorting refers to classifying the original data according to specified conditions.
The data calculation information is used for calculating the sorted original data to obtain an aggregation result. The calculation may include averaging, summing, ratioing, etc.
The data output information is used for outputting the aggregation result. The data output information may specify an output address, an output type, and the like of the aggregation result.
The data aggregation script may be loaded periodically to control the periodicity of data aggregation. For example, loading is performed in a period of 10 minutes. Thus, when there is a modification or addition to the data aggregation script, it will be effective in the next cycle of data aggregation.
Step 103, obtaining the original data from the appointed data source according to the data obtaining information.
For example, in an application scenario where the monitoring data of each application program is aggregated, the monitoring data is acquired by the application program performance monitoring system, if the frame rate of the application program is to be aggregated, step 103 is to acquire the frame rate from the monitoring data acquired by the application program performance monitoring system according to the data acquisition information.
The data acquisition information may include an index (index) and a type (type) of the original data, and the original data may be acquired from the data source according to the index and the type. For example, the index of the original data is defined as ubsa_event_info, the type is event. Fps (representing the frame rate), and then the data of the type event. Fps (i.e., the frame rate) is extracted from the index ubsa_event_info. The index represents the source of the original data.
And step 104, sorting the original data according to the data sorting information to obtain sorted data.
The sorting may have many rounds, each round being further subdivided on the basis of the previous round. In this embodiment, sorting the raw data may include sorting by application platform, sorting by version, sorting by interface name. For example, three rounds of frame rate sorting may be performed, first by application platform, then by version, and finally by interface name. There are two application platforms, 10 versions each, 10 interfaces for each application, three rounds of sorting divide the data into 200 shares of 2 x 10 total.
And 105, calculating the sorted data according to the data calculation information to obtain an aggregation result.
The calculation may include averaging, summing, ratioing, etc. For example, the frame rate is divided into 200 parts, and for each part of frame rate data, the average value of the part of frame rate data is calculated.
Before the calculation of the sorted data, the sorted data can be filtered to filter out abnormal data. For example, for a sorted frame rate, data having a frame rate of less than 15 or greater than 60 is filtered out, resulting in data having a frame rate between 15 and 60, and then the filtered frame rate (i.e., data having a frame rate between 15 and 60) is calculated.
For example, when the data aggregation is performed on the memory occupation amount, the obtained memory occupation amount can be filtered, data with the memory occupation amount larger than 800MB is filtered, and the filtered memory occupation amount (i.e., data with the memory occupation amount smaller than or equal to 800 MB) is calculated.
And step 106, outputting the aggregation result according to the data output information.
The data output information may include an output address, an output type, etc. of the aggregation result.
In an example of data aggregation for frame rates, the output of the aggregation result may include fields such as agg. Datasource, agg. Appver, agg. Fps. Avg, agg. Url, agg. Tag, etc., where:
agg.datasource represents the data source;
agg.appver represents version number;
agg.fps.avg represents the average of the frame rate;
agg.url represents the interface name;
tag indicates the type of data.
In one embodiment, the output of each aggregate result may include two fields, ag.time, ag.count, where:
agg.time represents the time of aggregation;
the count indicates how many pieces of data the aggregated result is aggregated.
The data aggregation method of the first embodiment obtains a data aggregation script; loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script; acquiring original data from a designated data source according to the data acquisition information; sorting the original data according to the data sorting information to obtain sorted data; calculating the sorted data according to the data calculation information to obtain an aggregation result; and outputting the aggregation result according to the data output information. According to the data aggregation method in the first embodiment, data aggregation is performed according to the aggregation script, when the aggregation requirement is newly increased or modified, only configuration in the script is required to be modified, code development is not required to be performed, a server is not required to be updated, development manpower and cost are saved, and high-efficiency data aggregation is achieved.
In another embodiment, the data aggregation method may further include: and deleting the original data after outputting the aggregation result.
For the aggregation of the frame rate, if the aggregation is performed every 10 minutes, 24×6=144 times are performed within 24 hours. If 200 pieces of data are produced per polymerization, the polymerization within 24 hours will produce 28800 pieces of data. Whereas the raw data within 24 hours may be millions of pieces. And the original data is deleted after the aggregation is completed, so that the storage cost can be greatly reduced. Meanwhile, when the frame rate is plotted, the time required for inquiring 28800 pieces of data and the time required for inquiring millions of pieces of data are much less, and the inquiring speed is greatly improved.
Example two
Fig. 2 is a block diagram of a data aggregation apparatus according to a second embodiment of the present invention. As shown in fig. 2, the data aggregation apparatus 10 may include: script acquisition unit 201, parsing unit 202, data acquisition unit 203, sorting unit 204, calculation unit 205, and output unit 206.
A script acquisition unit 201, configured to acquire a data aggregation script.
The pre-stored data aggregation script may be read. For example, data aggregation scripts for different data aggregation requirements may be pre-programmed and stored in a memory of the computing device, from which the required data aggregation scripts are read as needed.
Alternatively, a user-edited data aggregation script may be received. For example, a script editing interface may be provided from which a user-entered data aggregation script is received.
The data aggregation script describes data aggregation requirements.
The data aggregation script may be one or a plurality of data aggregation scripts. Each data aggregation script describes a separate data aggregation requirement. Different data aggregation requirements correspond to different data aggregation scripts.
For example, in an application scenario, the data aggregation method is used to aggregate the monitoring data of each application program. The monitoring data comprise the memory occupation amount, the frame rate, the breakdown times, the starting times and the like of the application program.
In the application scenario, a data aggregation script may describe a data aggregation requirement of memory occupation amounts of application programs, and calculate average values of memory occupation amounts of all application programs in a certain period of time, so as to obtain average memory occupation amounts in the period of time, where the average memory occupation amounts are the aggregation results as index data of memory occupation.
In the application scenario, another data aggregation script may describe a data aggregation requirement of a frame rate (FPS) of an application program, calculate an average value of the frame rates of all application programs in a certain period of time, and obtain an average frame rate in the period of time, as index data of the frame rate, where the average frame rate is an aggregation result.
In the application scenario, another data aggregation script may describe a data aggregation requirement of a crash rate of an application program, and divide the crash frequency of the application program in a certain period of time by the start frequency in the period of time to obtain the crash rate of the period of time, where the crash rate is a result of aggregation.
The data aggregation script may include data in JSON (JavaScript Object Notation, JS object profile) format by which data aggregation requirements are described. JSON is a lightweight data interchange format that stores and represents data in a text format that is completely independent of the programming language, and any JavaScript supported type can be represented by JSON, such as strings, numbers, objects, arrays, etc.
The data aggregation script may include ES (ElasticSearch) script. The data aggregation script takes the ES script as a part of the ES script, expands the ES script, and adds other fields for controlling the data aggregation operation.
The ES is a Lucene-based search server. It provides a distributed multi-user capable full text search engine based on RESTful web interface. The elastic search is developed by Java and is issued as an open source code under Apache license terms, so that real-time search can be achieved, and the method has the advantages of stability, reliability, rapidness and convenience in installation and use.
The data aggregation requirement includes data such as a data source (i.e., a source of original data), a type of the original data, a type of the aggregated data, a data aggregation mode (i.e., how to aggregate the original data), and the like.
And the parsing unit 202 is configured to load the data aggregation script, parse data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script.
The data aggregation script may be loaded into a memory of a computing device from which various information/data controlling the aggregation of data may be parsed.
In a specific embodiment, the data aggregation script includes query, agg, index, type parts, where:
the query part represents an ES script, and the ES script can be directly executed by an ES server, which defines the aggregate requirement and comprises an original data acquisition field, an original data sorting field and an original data calculation field;
the agg part defines the type of data storage after aggregation;
the index part defines which index the original data is acquired from and is used for acquiring the original data;
the type section defines the type of the original data in the index for acquiring the original data.
In this particular embodiment, portions query, agg, index, type may be parsed from the data aggregation script; the method comprises the steps of analyzing an original data acquisition field, an original data sorting field and an original data calculation field from a query part, taking the original data acquisition field, an index part and a type part as data acquisition information, taking the original data sorting field as data sorting information, taking the original data calculation field as data calculation information, and taking an agg part as data output information.
The data acquisition information, the data sorting information, the data calculation information and the data output information can be analyzed from the data aggregation script according to preset keywords (for example query, agg, index, type).
Or, the data acquisition information, the data sorting information, the data calculation information and the data output information can be analyzed from the data aggregation script according to the structure of the data aggregation script. For example, the data aggregation script is written according to a preset structure (for example, the data acquisition information in the data aggregation script is a first code segment, the data sorting information is a second code segment, the data calculation information is a third code segment, and the data output information is a fourth code segment), and the data acquisition information, the data sorting information, the data calculation information and the data output information are determined according to the preset structure.
The data acquisition information is used to obtain specified raw data from a specified data source.
The data sorting information is used for sorting the original data to obtain the sorted original data. Sorting refers to classifying the original data according to specified conditions.
The data calculation information is used for calculating the sorted original data to obtain an aggregation result. The calculation may include averaging, summing, ratioing, etc.
The data output information is used for outputting the aggregation result. The data output information may specify an output address, an output type, and the like of the aggregation result.
The data aggregation script may be loaded periodically to control the periodicity of data aggregation. For example, loading is performed in a period of 10 minutes. Thus, when there is a modification or addition to the data aggregation script, it will be effective in the next cycle of data aggregation.
A data acquisition unit 203 for acquiring the original data from the specified data source according to the data acquisition information.
For example, in an application scenario in which monitoring data of each application program is aggregated, the monitoring data is acquired by an application program performance monitoring system, and if data aggregation is to be performed on a frame rate of an application program, the data acquisition unit 203 acquires the frame rate from the monitoring data acquired by the application program performance monitoring system according to the data acquisition information.
The data acquisition information may include an index (index) and a type (type) of the original data, and the original data may be acquired from the data source according to the index and the type. For example, the index of the original data is defined as ubsa_event_info, the type is event. Fps (representing the frame rate), and then the data of the type event. Fps (i.e., the frame rate) is extracted from the index ubsa_event_info. The index represents the source of the original data.
And the sorting unit 204 is configured to sort the raw data according to the data sorting information, so as to obtain sorted data.
The sorting may have many rounds, each round being further subdivided on the basis of the previous round. In this embodiment, sorting the raw data may include sorting by application platform, sorting by version, sorting by interface name. For example, three rounds of frame rate sorting may be performed, first by application platform, then by version, and finally by interface name. There are two application platforms, 10 versions each, 10 interfaces for each application, three rounds of sorting divide the data into 200 shares of 2 x 10 total.
And the calculating unit 205 is configured to calculate the sorted data according to the data calculation information, so as to obtain an aggregation result.
The calculation may include averaging, summing, ratioing, etc. For example, the frame rate is divided into 200 parts, and for each part of frame rate data, the average value of the part of frame rate data is calculated.
Before the calculation of the sorted data, the sorted data can be filtered to filter out abnormal data. For example, for a sorted frame rate, data having a frame rate of less than 15 or greater than 60 is filtered out, resulting in data having a frame rate between 15 and 60, and then the filtered frame rate (i.e., data having a frame rate between 15 and 60) is calculated.
For example, when the data aggregation is performed on the memory occupation amount, the obtained memory occupation amount can be filtered, data with the memory occupation amount larger than 800MB is filtered, and the filtered memory occupation amount (i.e., data with the memory occupation amount smaller than or equal to 800 MB) is calculated.
And an output unit 206, configured to output information according to the data, and output the aggregation result.
The data output information may include an output address, an output type, etc. of the aggregation result.
In an example of data aggregation for frame rates, the output of the aggregation result may include fields such as agg. Datasource, agg. Appver, agg. Fps. Avg, agg. Url, agg. Tag, etc., where:
agg.datasource represents the data source;
agg.appver represents version number;
agg.fps.avg represents the average of the frame rate;
agg.url represents the interface name;
tag indicates the type of data.
In one embodiment, the output of each aggregate result may include two fields, ag.time, ag.count, where:
agg.time represents the time of aggregation;
the count indicates how many pieces of data the aggregated result is aggregated.
The data aggregation device 10 of the second embodiment acquires a data aggregation script; loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script; acquiring original data from a designated data source according to the data acquisition information; sorting the original data according to the data sorting information to obtain sorted data; calculating the sorted data according to the data calculation information to obtain an aggregation result; and outputting the aggregation result according to the data output information. The data aggregation device 10 of the second embodiment performs data aggregation according to the aggregation script, and only needs to modify the configuration in the script when the aggregation requirement is newly increased or modified, so that code development is not needed, a server is not needed to be updated, development manpower and cost are saved, and high-efficiency data aggregation is realized.
In another embodiment, the data aggregation apparatus 10 may further include: and the deleting unit is used for deleting the original data after outputting the aggregation result.
For the aggregation of the frame rate, if the aggregation is performed every 10 minutes, 24×6=144 times are performed within 24 hours. If 200 pieces of data are produced per polymerization, the polymerization within 24 hours will produce 28800 pieces of data. Whereas the raw data within 24 hours may be millions of pieces. And the original data is deleted after the aggregation is completed, so that the storage cost can be greatly reduced. Meanwhile, when the frame rate is plotted, the time required for inquiring 28800 pieces of data and the time required for inquiring millions of pieces of data are much less, and the inquiring speed is greatly improved.
Example III
The present embodiment provides a computer readable storage medium having a computer program stored thereon, which when executed by a processor, implements the steps of the above-described data aggregation method embodiment, such as steps 101-106 shown in fig. 1:
step 101, acquiring a data aggregation script;
step 102, loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script;
Step 103, according to the data acquisition information, acquiring original data from a designated data source;
step 104, sorting the original data according to the data sorting information to obtain sorted data;
step 105, calculating the sorted data according to the data calculation information to obtain an aggregation result;
and step 106, outputting the aggregation result according to the data output information.
Alternatively, the computer program, when executed by a processor, performs the functions of the modules/units in the above-described apparatus embodiments, e.g., units 201-206 in fig. 2:
a script acquisition unit 201 for acquiring a data aggregation script;
the parsing unit 202 is configured to load the data aggregation script, parse data acquisition information, data sorting information, data calculation information, and data output information from the data aggregation script;
a data acquisition unit 203 for acquiring original data from a specified data source according to the data acquisition information;
a sorting unit 204, configured to sort the raw data according to the data sorting information, so as to obtain sorted data;
a calculating unit 205, configured to calculate the sorted data according to the data calculation information, so as to obtain an aggregation result;
And an output unit 206, configured to output information according to the data, and output the aggregation result.
Example IV
Fig. 3 is a schematic diagram of a computer device according to a fourth embodiment of the invention. The computer device 1 comprises a memory 20, a processor 30 and a computer program 40, such as a data aggregation program, stored in the memory 20 and executable on the processor 30. The steps of the data aggregation method embodiment described above, such as steps 101-106 shown in fig. 1, are implemented by the processor 30 when executing the computer program 40:
step 101, acquiring a data aggregation script;
step 102, loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script;
step 103, according to the data acquisition information, acquiring original data from a designated data source;
step 104, sorting the original data according to the data sorting information to obtain sorted data;
step 105, calculating the sorted data according to the data calculation information to obtain an aggregation result;
and step 106, outputting the aggregation result according to the data output information.
Alternatively, the processor 30, when executing the computer program 40, performs the functions of the modules/units of the above-described apparatus embodiments, e.g., the units 201-206 in fig. 2:
a script acquisition unit 201 for acquiring a data aggregation script;
the parsing unit 202 is configured to load the data aggregation script, parse data acquisition information, data sorting information, data calculation information, and data output information from the data aggregation script;
a data acquisition unit 203 for acquiring original data from a specified data source according to the data acquisition information;
a sorting unit 204, configured to sort the raw data according to the data sorting information, so as to obtain sorted data;
a calculating unit 205, configured to calculate the sorted data according to the data calculation information, so as to obtain an aggregation result;
and an output unit 206, configured to output information according to the data, and output the aggregation result.
Illustratively, the computer program 40 may be partitioned into one or more modules/units that are stored in the memory 20 and executed by the processor 30 to complete the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing the specified functions, which instruction segments are used for describing the execution of the computer program 40 in the computer device 1. For example, the computer program 40 may be divided into a script acquisition unit 201, a parsing unit 202, a data acquisition unit 203, a sorting unit 204, a calculation unit 205, and an output unit 206 in fig. 2, and each unit has a specific function, see embodiment two.
The computer device 1 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. It will be appreciated by those skilled in the art that the schematic diagram 3 is merely an example of the computer apparatus 1 and does not constitute a limitation of the computer apparatus 1, and may include more or less components than illustrated, or may combine certain components, or different components, for example, the computer apparatus 1 may further include input and output devices, network access devices, buses, and the like.
The processor 30 may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor 30 may be any conventional processor or the like, the processor 30 being the control center of the computer device 1, the various interfaces and lines being used to connect the various parts of the overall computer device 1.
The memory 20 may be used to store the computer program 40 and/or modules/units, and the processor 30 may perform various functions of the computer device 1 by executing or executing the computer program and/or modules/units stored in the memory 20, and invoking data stored in the memory 20. The memory 20 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data created according to the use of the computer apparatus 1 (such as audio data, phonebook, etc.), and the like. In addition, the memory 20 may include high-speed random access memory, and may also include nonvolatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid state storage device.
The modules/units integrated in the computer device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
In the several embodiments provided by the present invention, it should be understood that the disclosed computer apparatus and method may be implemented in other ways. For example, the above-described embodiments of the computer apparatus are merely illustrative, and for example, the division of the units is merely a logical function division, and there may be other manners of division when actually implemented.
In addition, each functional unit in the embodiments of the present invention may be integrated in the same processing unit, or each unit may exist alone physically, or two or more units may be integrated in the same unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. Multiple units or computer means recited in the computer means claim may also be implemented by means of software or hardware by means of the same unit or computer means. The terms first, second, etc. are used to denote a name, but not any particular order.
Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (9)
1. A method of data aggregation, the method comprising:
acquiring a data aggregation script;
loading the data aggregation script, and analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script, wherein the data acquisition information, the data sorting information, the data calculation information and the data output information comprise:
analyzing a query part, an agg part, an index part and a type part from the data aggregation script;
analyzing an original data acquisition field, an original data sorting field and an original data calculation field from the query part;
taking the original data acquisition field, the index part and the type part as the data acquisition information, taking the original data sorting field as the data sorting information, taking the original data calculation field as the data calculation information and taking the agg part as the data output information;
Acquiring original data from a designated data source according to the data acquisition information;
sorting the original data according to the data sorting information to obtain sorted data;
calculating the sorted data according to the data calculation information to obtain an aggregation result;
and outputting the aggregation result according to the data output information.
2. The method of claim 1, wherein the data aggregation script comprises data in JSON format by which data aggregation requirements are described.
3. The method of claim 1, wherein the data aggregation script comprises an ES script.
4. The method of claim 1, wherein parsing the data acquisition information, the data sort information, the data calculation information, the data output information from the data aggregation script comprises:
analyzing the data acquisition information, the data sorting information, the data calculation information and the data output information from the data aggregation script according to preset keywords; or alternatively
And analyzing the data acquisition information, the data sorting information, the data calculation information and the data output information from the data aggregation script according to the structure of the data aggregation script.
5. The method of claim 1, wherein the method further comprises:
the sorted data is filtered before being calculated.
6. The method of claim 1, wherein the method further comprises:
and deleting the original data after outputting the aggregation result.
7. A data aggregation apparatus, the apparatus comprising:
the script acquisition unit is used for acquiring a data aggregation script;
the analyzing unit is used for loading the data aggregation script, analyzing data acquisition information, data sorting information, data calculation information and data output information from the data aggregation script, wherein the data acquisition information, the data sorting information, the data calculation information and the data output information comprise:
the method comprises the steps of analyzing a query part, an agg part, an index part and a type part from the data aggregation script;
the method comprises the steps of analyzing an original data acquisition field, an original data sorting field and an original data calculation field from the query part;
the data processing device is used for taking the original data acquisition field, the index part and the type part as the data acquisition information, taking the original data sorting field as the data sorting information, taking the original data calculation field as the data calculation information and taking the agg part as the data output information;
A data acquisition unit for acquiring original data from a specified data source according to the data acquisition information;
the sorting unit is used for sorting the original data according to the data sorting information to obtain sorted data;
the calculation unit is used for calculating the sorted data according to the data calculation information to obtain an aggregation result;
and the output unit is used for outputting information according to the data and outputting the aggregation result.
8. A computer apparatus, characterized in that: the computer apparatus comprising a processor for executing a computer program stored in a memory to implement the data aggregation method of any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: the computer program, when executed by a processor, implements the data aggregation method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811143535.9A CN109558403B (en) | 2018-09-28 | 2018-09-28 | Data aggregation method and device, computer device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811143535.9A CN109558403B (en) | 2018-09-28 | 2018-09-28 | Data aggregation method and device, computer device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109558403A CN109558403A (en) | 2019-04-02 |
| CN109558403B true CN109558403B (en) | 2024-02-02 |
Family
ID=65864722
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811143535.9A Active CN109558403B (en) | 2018-09-28 | 2018-09-28 | Data aggregation method and device, computer device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109558403B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813905B (en) * | 2020-06-17 | 2024-05-10 | 平安科技（深圳）有限公司 | Corpus generation method, corpus generation device, computer equipment and storage medium |
| CN112328708A (en) * | 2020-11-13 | 2021-02-05 | 深圳市般若大数据技术有限公司 | Mixed data warehouse technology for real-time aggregation of multiple data sources |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273519A (en) * | 2017-06-22 | 2017-10-20 | 睿视智联科技（香港）有限公司 | Data analysing method, device, terminal and storage medium |
| CN107480260A (en) * | 2017-08-16 | 2017-12-15 | 北京奇虎科技有限公司 | Big data real-time analysis method, device, computing device and computer-readable storage medium |
| CN108427725A (en) * | 2018-02-11 | 2018-08-21 | 华为技术有限公司 | Data processing method, device and system |
-
2018
- 2018-09-28 CN CN201811143535.9A patent/CN109558403B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273519A (en) * | 2017-06-22 | 2017-10-20 | 睿视智联科技（香港）有限公司 | Data analysing method, device, terminal and storage medium |
| CN107480260A (en) * | 2017-08-16 | 2017-12-15 | 北京奇虎科技有限公司 | Big data real-time analysis method, device, computing device and computer-readable storage medium |
| CN108427725A (en) * | 2018-02-11 | 2018-08-21 | 华为技术有限公司 | Data processing method, device and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109558403A (en) | 2019-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442341B (en) | Interface configuration method and device of application program and terminal equipment | |
| CN111209352B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN113704243A (en) | Data analysis method, data analysis device, computer device, and storage medium | |
| CN110647562B (en) | Data query method and device, electronic equipment and storage medium | |
| CN106557307B (en) | Service data processing method and system | |
| CN107391622B (en) | Data access method and equipment | |
| CN109558403B (en) | Data aggregation method and device, computer device and computer readable storage medium | |
| CN109359027B (en) | Method and device for Monkey test, electronic equipment and computer readable storage medium | |
| CN110955438A (en) | Method, device and equipment for monitoring performance of small program and storage medium | |
| CN112667426A (en) | Log analysis method and device | |
| CN112506950A (en) | Data aggregation processing method, computing node, computing cluster and storage medium | |
| CN111611276A (en) | Data query method, device and storage medium | |
| CN107256167B (en) | Upgrade control method and upgrade control equipment applied to application system migration | |
| CN111857781A (en) | Resource updating method and related equipment | |
| CN116775599A (en) | Data migration method, device, electronic equipment and storage medium | |
| CN115757174A (en) | Database difference detection method and device | |
| CN112667631B (en) | Automatic editing method, device, equipment and storage medium for business field | |
| CN110895460A (en) | Jenkins-based robot system integration method and device and terminal equipment | |
| CN112148470B (en) | Parameter synchronization method, computer device and readable storage medium | |
| CN112712610A (en) | Vehicle diagnosis log processing method and device, terminal device and readable storage medium | |
| CN113326237A (en) | Log data processing method and device, terminal device and storage medium | |
| CN111782479A (en) | Log processing method and device, electronic equipment and computer readable storage medium | |
| CN114371866A (en) | Version reconfiguration test method, device and equipment of service system | |
| CN111339422A (en) | Recommendation system task management platform, recommendation method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |