CN113449161A - Data collection method, device, system and storage medium - Google Patents
Data collection method, device, system and storage medium Download PDFInfo
- Publication number
- CN113449161A CN113449161A CN202010223825.5A CN202010223825A CN113449161A CN 113449161 A CN113449161 A CN 113449161A CN 202010223825 A CN202010223825 A CN 202010223825A CN 113449161 A CN113449161 A CN 113449161A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- collection node
- terminal
- collection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000013480 data collection Methods 0.000 title claims abstract description 42
- 238000003860 storage Methods 0.000 title claims abstract description 21
- 238000001914 filtration Methods 0.000 claims abstract description 49
- 230000002776 aggregation Effects 0.000 claims description 41
- 238000004220 aggregation Methods 0.000 claims description 41
- 230000015654 memory Effects 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012937 correction Methods 0.000 claims description 6
- 238000004806 packaging method and process Methods 0.000 claims description 6
- 238000012856 packing Methods 0.000 claims description 6
- 230000004931 aggregating effect Effects 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 14
- 238000013523 data management Methods 0.000 abstract description 7
- 230000008569 process Effects 0.000 description 26
- 238000013461 design Methods 0.000 description 19
- 238000010586 diagram Methods 0.000 description 15
- 238000007405 data analysis Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000009776 industrial production Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a data collection method, a device, a system and a storage medium, wherein the method is applied to a system comprising a terminal collection node, a middle collection node and a terminal collection node, and the method comprises the following steps: acquiring initial data through the terminal collection node, and uploading the initial data to the intermediate collection node; filtering the initial data according to service requirements through the intermediate collection node to obtain intermediate data, and uploading the intermediate data to the terminal collection node; and filtering the intermediate data through the terminal collection node to obtain target data. Therefore, data can be processed in a layered mode, the data processing amount of a high-level layer is reduced, the data have traceability, data management is facilitated, and the universality is high.
Description
Technical Field
The present invention relates to the field of big data technologies, and in particular, to a data processing method, apparatus, system, and storage medium.
Background
With the development of the industrialization process, the data amount generated in the industrial control process is also larger and larger. In order to effectively manage and control the industrial production process, a large amount of data needs to be collected.
The current data collection system is generally designed for a specific technical scene, and then the collected data is manually filtered by a manager.
However, the system designed in this way has strong coupling and poor versatility. When data collection is carried out, the acquired data are mixed due to the fact that the business process is not standard, and the calculation amount of data processing in the later period is greatly increased.
Disclosure of Invention
Embodiments of the present invention provide a data aggregation method, apparatus, system, and storage medium, which can perform hierarchical processing on data, reduce high-level data processing amount, make the data have traceability, facilitate data management, and have strong versatility.
In a first aspect, an embodiment of the present invention provides a data aggregation method, which is applied to a system including an end aggregation node, an intermediate aggregation node, and a terminal aggregation node, and the method includes:
acquiring initial data through the terminal collection node, and uploading the initial data to the intermediate collection node;
filtering the initial data according to service requirements through the intermediate collection node to obtain intermediate data, and uploading the intermediate data to the terminal collection node;
and filtering the intermediate data through the terminal collection node to obtain target data.
In one possible design, collecting initial data by the end sink node and uploading the initial data to the intermediate sink node includes:
acquiring the initial data according to a preset acquisition frequency;
sending a time service request to the intermediate collection node;
receiving time information sent by the intermediate sink node;
time correction is carried out through the time information, so that the time of the terminal collection node is consistent with the time of the middle collection node;
packaging the initial data to obtain a first data packet;
and after the first data packets are serialized according to the acquisition time sequence, the first data packets are sequentially uploaded to the intermediate collection node.
In one possible design, the filtering, by the intermediate sink node, the initial data according to a service requirement to obtain intermediate data, and uploading the intermediate data to the terminal sink node, includes:
according to the time sequence, all the first data packets uploaded at the same time are respectively obtained;
filtering the first data packet, and filtering data irrelevant to the service requirement to obtain the intermediate data;
packing the intermediate data to obtain a second data packet;
and after the second data packets are serialized according to the time sequence, the second data packets are sequentially uploaded to the terminal collection node.
In a possible design, after filtering the first data packet to filter out data that is not related to the service requirement, and obtain the intermediate data, the method further includes:
after summarizing the intermediate data, storing the intermediate data into a relational database;
and generating a data report corresponding to the intermediate data according to a first query request input by a user.
In one possible design, the filtering the intermediate data by the terminal aggregation node to obtain target data includes:
according to the time sequence, all the second data packets uploaded at the same time are respectively obtained;
and unpacking and filtering the second data packet to obtain the target data.
In one possible design, further comprising:
after summarizing the target data, storing the target data into a relational database;
and generating a data report corresponding to the target data according to a second query request input by the user.
In one possible design, when the number of the end sink nodes is plural and the number of the intermediate sink nodes is greater than 1; the method further comprises the following steps:
and any intermediate collection node transmits the intermediate data to another intermediate collection node, so that the intermediate data is uploaded to the terminal collection node after being filtered by the other intermediate collection node.
In a second aspect, an embodiment of the present invention provides a data collection apparatus, including:
the terminal collection node is used for collecting initial data and uploading the initial data to the intermediate collection node;
the intermediate collection node is used for filtering the initial data according to the service requirement to obtain intermediate data and uploading the intermediate data to the terminal collection node;
and the terminal collection node is used for filtering the intermediate data to obtain target data.
In one possible design, the end sink node is specifically configured to:
acquiring the initial data according to a preset acquisition frequency;
sending a time service request to the intermediate collection node;
receiving time information sent by the intermediate sink node;
time correction is carried out through the time information, so that the time of the terminal collection node is consistent with the time of the middle collection node;
packaging the initial data to obtain a first data packet;
and after the first data packets are serialized according to the acquisition time sequence, the first data packets are sequentially uploaded to the intermediate collection node.
In one possible design, the intermediate sink node is specifically configured to:
according to the time sequence, all the first data packets uploaded at the same time are respectively obtained;
filtering the first data packet, and filtering data irrelevant to the service requirement to obtain the intermediate data;
packing the intermediate data to obtain a second data packet;
and after the second data packets are serialized according to the time sequence, the second data packets are sequentially uploaded to the terminal collection node.
In one possible design, the intermediate sink node is specifically configured to:
after summarizing the intermediate data, storing the intermediate data into a relational database;
and generating a data report corresponding to the intermediate data according to a first query request input by a user.
In one possible design, the terminal aggregation node is specifically configured to:
according to the time sequence, all the second data packets uploaded at the same time are respectively obtained;
and unpacking and filtering the second data packet to obtain the target data.
In one possible design, the terminal aggregation node is specifically configured to:
after summarizing the target data, storing the target data into a relational database;
and generating a data report corresponding to the target data according to a second query request input by the user.
In one possible design, when the number of the end sink nodes is plural and the number of the intermediate sink nodes is greater than 1; and any intermediate collection node transmits the intermediate data to another intermediate collection node, so that the intermediate data is uploaded to the terminal collection node after being filtered by the other intermediate collection node.
In a third aspect, an embodiment of the present invention provides a data aggregation system, including: the device comprises a memory and a processor, wherein the memory stores executable instructions of the processor; wherein the processor is configured to perform the data compilation method of any of the first aspects via execution of the executable instructions.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the data aggregation method described in any one of the first aspects.
In a fifth aspect, an embodiment of the present invention provides a program product, where the program product includes: a computer program stored in a readable storage medium, from which the computer program can be read by at least one processor of a server, execution of the computer program by the at least one processor causing the server to perform the data aggregation method of any one of the first aspects.
The embodiment of the invention provides a data collection method, a device, a system and a storage medium, which are applied to a system comprising a terminal collection node, an intermediate collection node and a terminal collection node, wherein the terminal collection node is used for collecting initial data and uploading the initial data to the intermediate collection node; filtering the initial data according to service requirements through the intermediate collection node to obtain intermediate data, and uploading the intermediate data to the terminal collection node; and filtering the intermediate data through the terminal collection node to obtain target data. Therefore, data can be processed in a layered mode, the data processing amount of a high-level layer is reduced, the data have traceability, data management is facilitated, and the universality is high.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic diagram of a data collection topology of an application scenario of the present invention;
FIG. 2 is a schematic diagram of a data collection topology according to another application scenario of the present invention;
FIG. 3 is a schematic diagram of a data collection topology of another application scenario of the present invention;
FIG. 4 is a flowchart of a data aggregation method according to an embodiment of the present invention;
fig. 5 is a schematic service flow diagram of an end sink node according to an embodiment of the present invention;
fig. 6 is a schematic service flow diagram of an intermediate sink node according to an embodiment of the present invention;
FIG. 7 is a logical view of data collection provided by an embodiment of the present invention;
fig. 8 is a schematic service flow diagram of a terminal sink node according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a data aggregation device according to a second embodiment of the present invention;
fig. 10 is a schematic structural diagram of a data aggregation system according to a third embodiment of the present invention.
With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical solution of the present invention will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
With the development of the industrialization process, the data amount generated in the industrial control process is also larger and larger. In order to effectively manage and control the industrial production process, a large amount of data needs to be collected. The current data collection system is generally designed for a specific technical scene, and then the collected data is manually filtered by a manager. However, the system designed in this way has strong coupling and poor versatility. When data collection is carried out, the acquired data are mixed due to the fact that the business process is not standard, and the calculation amount of data processing in the later period is greatly increased.
In view of the above technical problems, the present invention provides a data collection method, apparatus, system and storage medium, which are applied to a system including a terminal collection node, a middle collection node and a terminal collection node, and can perform hierarchical processing on data, reduce the amount of high-level data processing, make the data have traceability, facilitate data management, and have strong versatility. The method provided by the implementation can also be applied to a distributed data acquisition system, is integrated by a flexible and easily-expanded real-time data analysis module, can realize data collection efficiently and in real time by a clear topological structure of each level, ensures the reusability and the usability of service realization, and is easy to realize. Fig. 1 is a schematic diagram of a data collection topology in an application scenario of the present invention, as shown in fig. 1, the data collection topology is a standard data collection topology, and the data collection topology includes three types of data collection nodes, which are an end collection node, an intermediate collection node, and a terminal collection node. One or more end collection nodes collect the data and upload the data to the middle collection node. The intermediate collection node has a function of filtering collection results besides a data collection function, can perform real-time data analysis, writes back analysis results serving as common data items to the second packet, and uploads the analysis results to the upper node together with the data collection results. The terminal sink node may sink data uploaded by the plurality of intermediate sink nodes. The data aggregation method in the end aggregation node is slightly different from the other two types of nodes, which is mainly determined by the environment in which the end node is located. The terminal collection node is located on a data acquisition computer together with the data acquisition process and is the place closest to data in the data acquisition system, and the data collection flow also plays a role in data time service and is the origin of data time. The functions contained in the end sink nodes and the intermediate sink nodes are substantially identical, the only difference being that the data upload function is masked in the end sink nodes. Only the tail end collection node in the three data collection nodes has no authority of storing data, and the other two nodes have data storage authorities, wherein the terminal and the middle collection node also comprise a function of filtering collection results, and whether the upper data is uploaded to the upper-level collection node or not is determined through filtering.
Fig. 2 is a schematic diagram of a data collection topology of another application scenario of the present invention, as shown in fig. 2, the data collection topology is a data collection topology of a multi-stage hybrid intermediate collection node, and the data collection topology includes three types of data collection nodes, which are an end collection node, an intermediate collection node, and a terminal collection node. The function of each sink node is as described in a standard data sink topology, and is not described in detail here. The difference from the standard data collection topology is that the intermediate collection node can receive data of other intermediate collection nodes besides the data of the end collection node, so as to collect the data of the end collection node and the intermediate collection node. The data collection topological structure is suitable for the field of industrial control, for example, a part produced by a production line I has data correlation with a quality inspection intermediate layer, so that data interaction is required between intermediate nodes of a level.
Fig. 3 is a schematic diagram of a data collection topology according to another application scenario of the present invention, as shown in fig. 3, the data collection topology includes three types of data collection nodes, that is, an end collection node, an intermediate collection node, and an end collection node. In practical application, data is not necessarily finally collected on one node to be uniformly processed and stored. Therefore, according to different requirements of data processing services, a plurality of terminal collection nodes can be configured for the intermediate collection node, so that data filtered by the data filtering module and of different services can be uploaded to different terminal collection nodes respectively. Finally, the system is deployed into a topological structure of a plurality of terminal collection nodes so as to achieve the purposes of pressure distribution or service correlation division and the like. In the above paragraph, both the terminal sink nodes and the intermediate sink nodes are described, and there are data filtering and storing functions, so that the intermediate sink nodes may also be terminal sink nodes for some data. From this description, it can be seen that this framework generally resembles the architecture of a channel filter schema. The data are processed and filtered layer by layer through a plurality of intermediate nodes, the data are continuously screened and stored, and a small amount of data needing high-level summary basically remain in the data finally processed by the terminal node. The design effectively improves the analysis efficiency of data, enhances the timeliness of abnormal condition alarm, fundamentally distributes the pressure of real-time data processing, and controls the system abnormal problem in part.
By the method, data can be processed in a layered mode, high-level data processing amount is reduced, the data have traceability, data management is facilitated, and universality is high.
The following describes the technical solutions of the present invention and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 4 is a flowchart of a data aggregation method according to an embodiment of the present invention, and as shown in fig. 4, the method in the embodiment is applied to a system including an end aggregation node, an intermediate aggregation node, and a terminal aggregation node, and may include:
s101, acquiring initial data through a tail end collection node, and uploading the initial data to a middle collection node.
In this embodiment, the terminal sink node collects initial data according to a preset collection frequency; sending a time service request to an intermediate collection node; receiving time information sent by an intermediate collection node; time correction is carried out through the time information, so that the time of the terminal collection node is consistent with the time of the middle collection node; packaging the initial data to obtain a first data packet; and after the first data packets are serialized according to the acquisition time sequence, the first data packets are sequentially uploaded to the intermediate collection node.
Specifically, the end sink node is a data sink node on the end collector, is located on the data collector computer together with the data collection process, and is the closest place to the data in the data collection system. The terminal collection node generally has two deployment forms, one is a multithread data collection form in the same process; the other is a data collection form based on a network protocol or an interprocess communication protocol among different processes. Both forms have an important common feature, namely that the end sink node is a data sink on the same computer. Thus, the end sink node also serves as a time service for the data, being the origin of the data time. The time transfer of data can be performed synchronously and uniformly. Although a specific data acquisition terminal acquires data on other equipment through a certain protocol in most cases, the data acquisition terminal collects the data on the end node in time. Fig. 5 is a schematic view of a service flow of an end sink node according to an embodiment of the present invention, and as shown in fig. 5, the end sink node mainly includes four services, which are data collection and packing, time calibration, real-time data analysis, and data upload, respectively. The most common log collection service is taken as an example to introduce the service logic of the terminal collection node. The general logs are in units of lines, and the time, the content and the like are generally contained in one line of logs. And the data collection process of the terminal collection node performs polling access on the log at a set frequency, and reads the key data after time and content decomposition. The acquired data is then subjected to a system normalization process, for example, to: time _ data ID _ type _ value format. The time correction process of the terminal collection node refers to the step of calibrating the time part of the data item through the time service, and the time of the terminal collection node is calibrated to be standard time of the data acquisition system. The end sink node then delivers the data to the real-time data analysis process branch (if configured at that level) and the data upload branch, respectively, according to the calibrated time. The real-time data analysis process branch of the terminal collection node is to analyze data through configured data identification and data operation scripts to obtain a result generated by data analysis. An early warning event can then be generated upon exceeding the threshold, depending on the configuration. Meanwhile, the data analysis result can be written back to the second packet as a common data item when needed, and data uploading is carried out. The data uploading branch of the end collection node is to create or take a second packet according to the calibrated time in the data item, and then push the data item into the second packet. After the second packet of the current second is created, the second packet of the last second is closed and put into an asynchronous queue. And finally, serializing the second packets and sending the second packets to a previous-stage node (an intermediate/terminal data aggregation node).
And S102, filtering the initial data according to the service requirement through the intermediate collection node to obtain intermediate data, and uploading the intermediate data to the terminal collection node.
In this embodiment, the intermediate sink node respectively acquires all the first data packets uploaded at the same time according to a time sequence; filtering the first data packet, and filtering data irrelevant to the service requirement to obtain intermediate data; packing the intermediate data to obtain a second data packet; and after the second data packets are serialized according to the time sequence, the second data packets are sequentially uploaded to the terminal collection node.
Optionally, after filtering the first data packet, and filtering out data that is not related to the service requirement to obtain intermediate data, the method further includes: after the intermediate data are summarized, the intermediate data are stored in a relational database; and generating a data report corresponding to the intermediate data according to the first query request input by the user.
Optionally, when the number of the end sink nodes is multiple and the number of the intermediate sink nodes is greater than 1; the method further comprises the following steps: and any intermediate sink node transmits the intermediate data to another intermediate sink node, so that the intermediate data is filtered by the other intermediate sink node and then uploaded to the terminal sink node.
Specifically, the intermediate sink node further includes a function of filtering the sink result, and determines whether to upload the received data to the upper-level sink node through filtering. Fig. 6 is a schematic view of a service flow of an intermediate sink node according to an embodiment of the present invention, and as shown in fig. 6, the intermediate sink node mainly includes four services, which are a data sink flow, a real-time data analysis flow, a data storage flow, and a data upload flow. The data aggregation process of the intermediate aggregation node is to receive second packets sent by all upstream nodes (the end aggregation node or the intermediate aggregation node), arrange all the second packets of the same second, and perform aggregation processing on data. Fig. 7 is a logic diagram of data aggregation according to an embodiment of the present invention, and as shown in fig. 7, an intermediate aggregation node receives data second packets from a collector a, a collector B, and a collector C, and at this time, an aggregated data packet is empty, as shown by initial data in the diagram. And the intermediate collection node judges the foremost data item of the sequenced data packets according to the time sequence, takes out the foremost data among the data packets and enters the next iteration. In this way, only after all the data in the plurality of packets are taken out, the point that needs to be noticed in the process is that when the first data item in all the packets has a plurality of same time stamps, all the data items with the same time stamps can be taken out at one time, so that the iteration times are effectively reduced. Thus, in the first round of the cycle, the timestamp of collector a is 10, the timestamp of collector B is 11, the ordering of the timestamps is 10 is the first, and the timestamps of collector a and collector B are both 10. Therefore, in the first round of the loop, the data AA of collector a and the data BA of collector B are put into the aggregate data packet, and the corresponding data are deleted in collector a and collector B. In the second round of the cycle, the timestamps 11 of the collector C are sorted first, the data CA of the collector C is put into the collected data packet, and the corresponding data is deleted in the collector C. In the third round, the timestamp 33 of collector C is sorted first, the data CB of collector C is put into the aggregate data packet, and the corresponding data is deleted in collector C. The real-time data analysis process of the intermediate collection node is to analyze data through configured data identification and data operation scripts to obtain a result generated by data analysis. An early warning event can then be generated upon exceeding the threshold, depending on the configuration. Meanwhile, the data analysis result can be written back to the second packet as a common data item when needed, and data uploading is carried out. The data uploading process of the intermediate collection node is to create or take a second packet according to the time in the data item, and then push the data item into the second packet. After the second packet of the current second is created, the second packet of the last second is closed and put into an asynchronous queue. And the intermediate collection node takes out the second packets from the queue in sequence, unpacks and filters the second packets, and carries out routing marking on the data items according to the system configuration information. And finally, respectively packaging and serializing the data according to the routing marks, and respectively storing the data packets into a time sequence database or sending the data packets to a specified upper node according to the routing marks. The data storage process of the intermediate collection node is to screen out the data items required to be processed, continue to calculate, count and store the raster data into the relational database so as to be convenient for displaying the report.
And S103, filtering the intermediate data through the terminal collection node to obtain target data.
In this embodiment, all the second data packets uploaded at the same time are respectively acquired according to a time sequence; and unpacking and filtering the second data packet to obtain target data.
Optionally, after summarizing the target data, storing the target data into a relational database; and generating a data report corresponding to the target data according to a second query request input by the user.
Specifically, fig. 8 is a schematic view of a service flow of a terminal sink node according to an embodiment of the present invention, and as shown in fig. 8, the service flow of the terminal sink node is substantially the same as the service flow of an intermediate sink node, and the only difference is that there is no data upload flow in the terminal sink node and data does not need to be uploaded to other nodes. Therefore, the service flow of the terminal collection node is described in detail in the related description of the intermediate collection node, and is not described herein again.
In this embodiment, the method is applied to a system including a terminal sink node, an intermediate sink node, and a terminal sink node, and acquires initial data through the terminal sink node, and uploads the initial data to the intermediate sink node; filtering the initial data according to the service requirement through the intermediate collection node to obtain intermediate data, and uploading the intermediate data to the terminal collection node; and filtering the intermediate data through the terminal collection node to obtain target data. Therefore, data can be processed in a layered mode, the data processing amount of a high-level layer is reduced, the data have traceability, data management is facilitated, and the universality is high.
Fig. 9 is a schematic structural diagram of a data aggregation device according to a second embodiment of the present invention, and as shown in fig. 9, the data aggregation device according to the present embodiment may include:
the terminal collection node 31 is used for collecting initial data and uploading the initial data to the intermediate collection node;
the intermediate collection node 32 is used for filtering the initial data according to the service requirement to obtain intermediate data, and uploading the intermediate data to the terminal collection node;
and the terminal collection node 33 is configured to filter the intermediate data to obtain target data.
In one possible design, the end sink node 31 is specifically configured to:
acquiring initial data according to a preset acquisition frequency;
sending a time service request to an intermediate collection node;
receiving time information sent by an intermediate collection node;
time correction is carried out through the time information, so that the time of the terminal collection node is consistent with the time of the middle collection node;
packaging the initial data to obtain a first data packet;
and after the first data packets are serialized according to the acquisition time sequence, the first data packets are sequentially uploaded to the intermediate collection node.
In one possible design, the intermediate sink node 32 is specifically configured to:
according to the time sequence, all first data packets uploaded at the same time are respectively acquired;
filtering the first data packet, and filtering data irrelevant to the service requirement to obtain intermediate data;
packing the intermediate data to obtain a second data packet;
and after the second data packets are serialized according to the time sequence, the second data packets are sequentially uploaded to the terminal collection node.
In one possible design, the intermediate sink node 32 is specifically configured to:
after the intermediate data are summarized, the intermediate data are stored in a relational database;
and generating a data report corresponding to the intermediate data according to the first query request input by the user.
In one possible design, the terminal aggregation node 33 is specifically configured to:
according to the time sequence, all second data packets uploaded at the same time are respectively acquired;
and unpacking and filtering the second data packet to obtain target data.
In one possible design, the terminal aggregation node 33 is specifically configured to:
after summarizing the target data, storing the target data into a relational database;
and generating a data report corresponding to the target data according to a second query request input by the user.
In one possible design, when the number of the end sink nodes is multiple and the number of the intermediate sink nodes is greater than 1; and any intermediate sink node transmits the intermediate data to another intermediate sink node, so that the intermediate data is filtered by the other intermediate sink node and then uploaded to the terminal sink node.
The data aggregation device of this embodiment may execute the technical solution in the method shown in fig. 2, and for the specific implementation process and the technical principle, reference is made to the relevant description in the method shown in fig. 2, which is not described herein again.
In this embodiment, the method is applied to a system including a terminal sink node, an intermediate sink node, and a terminal sink node, and acquires initial data through the terminal sink node, and uploads the initial data to the intermediate sink node; filtering the initial data according to the service requirement through the intermediate collection node to obtain intermediate data, and uploading the intermediate data to the terminal collection node; and filtering the intermediate data through the terminal collection node to obtain target data. Therefore, data can be processed in a layered mode, the data processing amount of a high-level layer is reduced, the data have traceability, data management is facilitated, and the universality is high.
Fig. 10 is a schematic structural diagram of a data aggregation system according to a third embodiment of the present invention, and as shown in fig. 10, the data aggregation system 40 according to the present embodiment may include: a processor 41 and a memory 42.
A memory 42 for storing programs; the Memory 42 may include a volatile Memory (RAM), such as a Static Random Access Memory (SRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), and the like; the memory may also comprise a non-volatile memory, such as a flash memory. The memory 42 is used to store computer programs (e.g., applications, functional modules, etc. that implement the above-described methods), computer instructions, etc., which may be stored in one or more of the memories 42 in a partitioned manner. And the above-mentioned computer program, computer instructions, data, etc. can be called by the processor 41.
The computer programs, computer instructions, etc. described above may be stored in one or more memories 42 in partitions. And the above-mentioned computer program, computer instructions, data, etc. can be called by the processor 41.
A processor 41 for executing the computer program stored in the memory 42 to implement the steps of the method according to the above embodiments.
Reference may be made in particular to the description relating to the preceding method embodiment.
The processor 41 and the memory 42 may be separate structures or may be integrated structures integrated together. When the processor 41 and the memory 42 are separate structures, the memory 42 and the processor 41 may be coupled by a bus 43.
The data collection system of this embodiment may execute the technical solution in the method shown in fig. 2, and for the specific implementation process and technical principle, reference is made to the relevant description in the method shown in fig. 2, which is not described herein again.
In addition, embodiments of the present application further provide a computer-readable storage medium, in which computer-executable instructions are stored, and when at least one processor of the user equipment executes the computer-executable instructions, the user equipment performs the above-mentioned various possible methods.
Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an ASIC. Additionally, the ASIC may reside in user equipment. Of course, the processor and the storage medium may reside as discrete components in a communication device.
The present application further provides a program product comprising a computer program stored in a readable storage medium, from which the computer program can be read by at least one processor of a server, the execution of the computer program by the at least one processor causing the server to carry out the method of any of the embodiments of the invention described above.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A data aggregation method applied to a system including a terminal aggregation node, an intermediate aggregation node, and a terminal aggregation node, the method comprising:
acquiring initial data through the terminal collection node, and uploading the initial data to the intermediate collection node;
filtering the initial data according to service requirements through the intermediate collection node to obtain intermediate data, and uploading the intermediate data to the terminal collection node;
and filtering the intermediate data through the terminal collection node to obtain target data.
2. The method of claim 1, wherein collecting initial data by the end sink node and uploading the initial data to the intermediate sink node comprises:
acquiring the initial data according to a preset acquisition frequency;
sending a time service request to the intermediate collection node;
receiving time information sent by the intermediate sink node;
time correction is carried out through the time information, so that the time of the terminal collection node is consistent with the time of the middle collection node;
packaging the initial data to obtain a first data packet;
and after the first data packets are serialized according to the acquisition time sequence, the first data packets are sequentially uploaded to the intermediate collection node.
3. The method of claim 2, wherein filtering the initial data by the intermediate sink node according to service requirements to obtain intermediate data, and uploading the intermediate data to the terminal sink node, comprises:
according to the time sequence, all the first data packets uploaded at the same time are respectively obtained;
filtering the first data packet, and filtering data irrelevant to the service requirement to obtain the intermediate data;
packing the intermediate data to obtain a second data packet;
and after the second data packets are serialized according to the time sequence, the second data packets are sequentially uploaded to the terminal collection node.
4. The method of claim 3, wherein after filtering the first data packet to filter out data that is not relevant to the service requirement, and obtain the intermediate data, the method further comprises:
after summarizing the intermediate data, storing the intermediate data into a relational database;
and generating a data report corresponding to the intermediate data according to a first query request input by a user.
5. The method of claim 3, wherein filtering the intermediate data by the terminal aggregation node to obtain target data comprises:
according to the time sequence, all the second data packets uploaded at the same time are respectively obtained;
and unpacking and filtering the second data packet to obtain the target data.
6. The method of claim 5, further comprising:
after summarizing the target data, storing the target data into a relational database;
and generating a data report corresponding to the target data according to a second query request input by the user.
7. The method according to any of claims 1-6, wherein the number of the end sink nodes is plural, and the number of the intermediate sink nodes is greater than 1; the method further comprises the following steps:
and any intermediate collection node transmits the intermediate data to another intermediate collection node, so that the intermediate data is uploaded to the terminal collection node after being filtered by the other intermediate collection node.
8. A data collection device, comprising:
the terminal collection node is used for collecting initial data and uploading the initial data to the intermediate collection node;
the intermediate collection node is used for filtering the initial data according to the service requirement to obtain intermediate data and uploading the intermediate data to the terminal collection node;
and the terminal collection node is used for filtering the intermediate data to obtain target data.
9. A data aggregation system, comprising: the memory is used for storing executable instructions of the processor; wherein the processor is configured to perform the data compilation method of claims 1-7 via execution of the executable instructions.
10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method of aggregating data of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010223825.5A CN113449161A (en) | 2020-03-26 | 2020-03-26 | Data collection method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010223825.5A CN113449161A (en) | 2020-03-26 | 2020-03-26 | Data collection method, device, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113449161A true CN113449161A (en) | 2021-09-28 |
Family
ID=77807119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010223825.5A Pending CN113449161A (en) | 2020-03-26 | 2020-03-26 | Data collection method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113449161A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002024252A (en) * | 2000-07-03 | 2002-01-25 | Nippon Telegr & Teleph Corp <Ntt> | Data aggregating method, and recording medium recorded with execution program for the method |
| AU2010219408A1 (en) * | 2009-09-16 | 2011-03-31 | Microsoft Technology Licensing, Llc | Method of managing and summarising events in an event processing network |
| CN103685486A (en) * | 2013-12-02 | 2014-03-26 | 中国科学院计算技术研究所 | Distributed system monitoring method stepping over data center clusters and system |
| CN105025020A (en) * | 2015-07-07 | 2015-11-04 | 成都英力拓信息技术有限公司 | Internet of Things implementation method |
| CN108401241A (en) * | 2018-04-27 | 2018-08-14 | 江苏信息职业技术学院 | A kind of wireless system for transmitting data |
| CN110444000A (en) * | 2019-08-09 | 2019-11-12 | 贵安新区配售电有限公司 | A kind of multilist measurement data acquisition method and system based on technology of Internet of things |
| CN110782644A (en) * | 2019-11-19 | 2020-02-11 | 云南省电子工业研究所 | Hybrid wireless meter reading and writing system capable of realizing self-networking |
-
2020
- 2020-03-26 CN CN202010223825.5A patent/CN113449161A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002024252A (en) * | 2000-07-03 | 2002-01-25 | Nippon Telegr & Teleph Corp <Ntt> | Data aggregating method, and recording medium recorded with execution program for the method |
| AU2010219408A1 (en) * | 2009-09-16 | 2011-03-31 | Microsoft Technology Licensing, Llc | Method of managing and summarising events in an event processing network |
| CN103685486A (en) * | 2013-12-02 | 2014-03-26 | 中国科学院计算技术研究所 | Distributed system monitoring method stepping over data center clusters and system |
| CN105025020A (en) * | 2015-07-07 | 2015-11-04 | 成都英力拓信息技术有限公司 | Internet of Things implementation method |
| CN108401241A (en) * | 2018-04-27 | 2018-08-14 | 江苏信息职业技术学院 | A kind of wireless system for transmitting data |
| CN110444000A (en) * | 2019-08-09 | 2019-11-12 | 贵安新区配售电有限公司 | A kind of multilist measurement data acquisition method and system based on technology of Internet of things |
| CN110782644A (en) * | 2019-11-19 | 2020-02-11 | 云南省电子工业研究所 | Hybrid wireless meter reading and writing system capable of realizing self-networking |
Non-Patent Citations (1)
| Title |
|---|
| 雷景生;郝珈玮;朱国康;: "基于"分层-汇集"模型的短期电力负荷预测", 电力建设, no. 01, 1 January 2017 (2017-01-01) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108038053B (en) | A kind of dynamic configuration buries method and apparatus a little | |
| CN107689953A (en) | A kind of vessel safety monitoring method and system towards multi-tenant cloud computing | |
| CN112714047B (en) | Industrial control protocol flow based test method, device, equipment and storage medium | |
| CN111339071A (en) | Method and device for processing multi-source heterogeneous data | |
| CN111352994B (en) | Data synchronization method and related equipment and device | |
| CN105141441B (en) | A kind of method that IP network graphically configures | |
| DE102009043278A1 (en) | Network synchronization via IP networks | |
| CN104639994A (en) | Video abstraction generating method, system and network storage equipment based on moving objects | |
| CN108205569A (en) | For updating the method and apparatus of configuration management database | |
| CN113364624A (en) | Mixed cloud flow acquisition method and system based on edge computing | |
| CN105338107A (en) | Stronghold operation synchronous management system and stronghold operation synchronous management method | |
| CN110928681A (en) | Data processing method and device, storage medium and electronic device | |
| CN104202328B (en) | A kind of method, configuration module and the subscription end of subscription GOOSE/SMV messages | |
| CN103810197A (en) | Hadoop-based data processing method and system | |
| CN104836686B (en) | A kind of method and device for the configuration file generating server | |
| CN100413257C (en) | Network element warning data obtaining method | |
| US8442947B2 (en) | Management of performance data | |
| CN112416557B (en) | Method and device for determining call relation, storage medium and electronic device | |
| CN113449161A (en) | Data collection method, device, system and storage medium | |
| CN107995026B (en) | Management and control method, management node, managed node and system based on middleware | |
| CN112448911B (en) | K-Means-based normal Server IP white list mining method | |
| CN116346650A (en) | CAN signal processing method, device and system and electronic equipment | |
| CN111259062A (en) | Method and device capable of ensuring sequence of result sets of full-table query statements of distributed database | |
| CN102307116A (en) | System and method for generating monitoring information | |
| CN113204464B (en) | Real-time service monitoring method, system, terminal and medium based on service scene |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |