CN111447229A - Large-scale data acquisition method and device based on compressed sensing theory - Google Patents
Large-scale data acquisition method and device based on compressed sensing theory Download PDFInfo
- Publication number
- CN111447229A CN111447229A CN202010232080.9A CN202010232080A CN111447229A CN 111447229 A CN111447229 A CN 111447229A CN 202010232080 A CN202010232080 A CN 202010232080A CN 111447229 A CN111447229 A CN 111447229A
- Authority
- CN
- China
- Prior art keywords
- data
- parameter matrix
- host
- characteristic parameter
- sampling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/26—Special purpose or proprietary protocols or architectures
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Arrangements For Transmission Of Measured Signals (AREA)
Abstract
The invention discloses a large-scale data acquisition method and a device based on a compressed sensing theory, which comprises the following steps: the method comprises the steps that a superior host acquires sampling data sent by a subordinate host, the sampling data are obtained by the subordinate host by collecting data in a memory bank of the subordinate host according to a dynamic sampling parameter matrix, the dynamic sampling parameter matrix is determined according to an inherent physical characteristic parameter matrix of a sampling point, and data reconstruction is carried out according to the inherent physical characteristic parameter matrix of the sampling point, a variation characteristic parameter matrix of sampled historical data and the sampling data. The subordinate host performs data compression and acquisition through the dynamic sampling parameter matrix, and then performs data reconstruction through the superior host, so that the system resource consumption in the data acquisition process can be reduced, the data acquisition efficiency is improved, and the method is suitable for large-scale data acquisition.
Description
Technical Field
The invention relates to the field of data acquisition, in particular to a large-scale data acquisition method and device based on a compressed sensing theory.
Background
With the increasingly wide application of data acquisition and monitoring systems in the industries of rail transit, intelligent buildings and the like, more and more monitoring devices and applications are provided, and the data volume is increased day by day. Generally, in order to acquire accurate data in time, a conventional data acquisition protocol acquires the whole data in a small fixed period, consumes large network and software and hardware resources, and causes great pressure on the system. When the data volume is increased, the acquisition mode can not meet the requirement quickly, so that an acquisition algorithm facing large-scale data is needed. In order to solve the problem, no relevant patent is published in the field of comprehensive monitoring. However, in other fields, for large-scale data acquisition, there are two traditional approaches, one is to add acquisition equipment and the other is to compress the data during transmission.
The two modes cannot fundamentally solve the problem that the data scale is gradually increased in the field of comprehensive monitoring, and the cost is greatly increased by increasing the acquisition equipment at first. And with the increase of collection equipment, the data volume of collection can be more, causes bigger pressure to the data processing module of software, and the requirement to hardware also can be higher, and when reaching the bottleneck of software and hardware, just one in the face of magnanimity data, the consequence that faces just one: the system does not process the same. Then, a series of problems such as loss of critical data will occur. Compressing during transmission, while alleviating the network stress, increases the stress on the data processing module because of the additional steps of compressing and decompressing data.
Disclosure of Invention
The embodiment of the invention provides a large-scale data acquisition method and device based on a compressive sensing theory, which are used for improving the data acquisition efficiency and saving network and system resources.
In a first aspect, an embodiment of the present invention provides a large-scale data acquisition method based on a compressive sensing theory, including:
a superior host acquires sampling data sent by a subordinate host, wherein the sampling data is acquired by the subordinate host by collecting data in a memory bank of the subordinate host according to a dynamic sampling parameter matrix; the dynamic sampling parameter matrix is determined according to the inherent physical characteristic parameter matrix of the sampling point;
and the superior host performs data reconstruction according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data and the sampled data.
In the technical scheme, the subordinate host performs data compression and acquisition through the dynamic sampling parameter matrix, and then performs data reconstruction through the superior host, so that the system resource consumption in the data acquisition process can be reduced, the data acquisition efficiency is improved, and the method is suitable for large-scale data acquisition.
Optionally, the determining, by the superior host, the dynamic sampling parameter matrix according to the intrinsic physical characteristic parameter matrix of the sampling point includes:
the superior host acquires an inherent physical characteristic parameter matrix of a sampling point;
the superior host generates a variation characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point;
and the superior host generates the dynamic sampling parameter matrix according to the variation characteristic parameter matrix of the sampled historical data.
Optionally, after the superior host performs data reconstruction according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data, and the sampled data, the method further includes:
the superior host corrects the change characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point, the change characteristic parameter matrix of the sampled historical data and the sampled data;
and the superior host generates the dynamic sampling parameter matrix according to the modified change characteristic parameter matrix of the sampled historical data.
Optionally, the performing, by the upper host, data reconstruction includes:
and the upper host performs data reconstruction by using a greedy algorithm.
Optionally, the sampling data is collected and sent by the lower host after confirming that the data changes, or periodically collected and sent by the lower host, or collected and sent by the lower host after receiving a data collection command sent by the upper host.
Optionally, the data acquisition command is periodically sent by the upper level host.
In a second aspect, an embodiment of the present invention provides a large-scale data acquisition device based on compressed sensing theory, where the device includes:
the device comprises an acquisition unit, a storage unit and a processing unit, wherein the acquisition unit is used for acquiring sampling data sent by a lower host, and the sampling data is acquired by the lower host by collecting data in a memory bank of the lower host according to a dynamic sampling parameter matrix; the dynamic sampling parameter matrix is determined according to the inherent physical characteristic parameter matrix of the sampling point;
and the processing unit is used for carrying out data reconstruction according to the inherent physical characteristic parameter matrix of the sampling point, the change characteristic parameter matrix of the sampled historical data and the sampled data.
Optionally, the processing unit is specifically configured to:
acquiring an inherent physical characteristic parameter matrix of a sampling point;
generating a variation characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point; and generating the dynamic sampling parameter matrix according to the variation characteristic parameter matrix of the sampled historical data.
Optionally, the processing unit is further configured to:
after data reconstruction is carried out according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data and the sampled data, the variation characteristic parameter matrix of the sampled historical data is corrected according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data and the sampled data;
and generating the dynamic sampling parameter matrix according to the modified change characteristic parameter matrix of the sampled historical data.
Optionally, the processing unit is specifically configured to:
and the upper host performs data reconstruction by using a greedy algorithm.
Optionally, the sampling data is collected and sent by the lower host after confirming that the data changes, or periodically collected and sent by the lower host, or collected and sent by the lower host after receiving a data collection command sent by the upper host.
Optionally, the data acquisition command is periodically sent by the upper level host.
In a third aspect, an embodiment of the present invention further provides a computing device, including:
a memory for storing program instructions;
and the processor is used for calling the program instructions stored in the memory and executing the large-scale data acquisition method based on the compressive sensing theory according to the obtained program.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where computer-executable instructions are stored, where the computer-executable instructions are configured to enable a computer to execute the above large-scale data acquisition method based on compressive sensing theory.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a system architecture diagram according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a large-scale data acquisition method based on a compressive sensing theory according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an acquisition algorithm according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an IDE master-slave protocol data collection flow according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data collection process for passive IDE reception according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an IDE master-slave protocol scheduling/handover acquisition process according to an embodiment of the present invention;
fig. 7 is a schematic diagram of an IDE passive reception scheduling/handover acquisition process according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a large-scale data acquisition device based on a compressive sensing theory according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 illustrates an exemplary system architecture to which an embodiment of the present invention is applicable, which may include an upper level host 100 and a lower level host 200. The upper level host 100 communicates with the lower level host 200 through an IDE (Internal Data Exchange Standards).
The upper host 100 may be a server device of a central station or a control center. The lower host 200 may be a lower station, a lower node, or an RTU (remote terminal unit). The lower host 200 may perform data collection upon receiving a data collection command transmitted from the upper host 100.
It should be noted that the structure shown in fig. 1 is only an example, and the embodiment of the present invention is not limited thereto.
In a conventional data acquisition protocol, in order to directly acquire the full amount of data, the total amount of data acquisition in a unit time (1 second) is the acquisition point number and the acquisition frequency. When the number of acquisition points is 1000 ten thousand and the acquisition frequency is 2, the total amount of data acquired and processed per second is 2000 ten thousand, and if each data is calculated by 2 bytes on average, the system is meant to transmit and process 4000 ten thousand bytes of data in one second. Then tremendous pressure is placed on the network, hardware, and software of the system. The acquisition mode consumes a large amount of system resources such as software, hardware and networks, and the like, and causes great pressure on a transmission and data processing module, so that the acquisition mode can only be qualified for the condition of small data volume, when the acquisition mode meets the requirement of large-scale data acquisition, the acquisition mode has high requirements on the network and the software and hardware, and when the bottleneck of the network and the software and hardware is met, the traditional data acquisition protocol cannot be applied.
Through analysis, a large number of collected physical quantities (such as temperature, pressure, flow, running state and the like) in an industrial control system are changed slowly, are continuous, have peaks or are related, the collection period can be increased when the change is slow, the collection period can be reduced when the peak or the change is fast, or only some physical quantities are collected, and other related physical quantities can be reconstructed according to the physical quantities, namely compression is carried out during collection according to the theory of compressive sensing, so that the key is how to design and develop an algorithm to quickly obtain the next collection point and collection time.
How to reconstruct the acquired small amount of data according to the compression algorithm becomes another key. The reconstruction algorithm is related to the acquisition algorithm. Another key of the reconstruction algorithm is the rapidity, practicality and simplicity and ease of initialization of the algorithm, which otherwise loses practical application value. The data protocol algorithm provided by the embodiment of the invention just solves the problem.
Based on the above description, fig. 2 exemplarily shows a flow of the large-scale data collection method based on compressive sensing theory according to the present invention, where the flow may be executed by a large-scale data collection device based on compressive sensing theory, and the device may be located in a superior host or the superior host.
As shown in fig. 2, the steps of the process specifically include:
in step 201, the upper host acquires the sampling data sent by the lower host.
In the embodiment of the present invention, the sampling data is obtained by the lower level host collecting data in the memory bank of the lower level host according to the dynamic sampling parameter matrix. The dynamic sampling parameter matrix is determined according to the inherent physical characteristic parameter matrix of the sampling point, specifically, the upper host can acquire the inherent physical characteristic parameter matrix of the sampling point, then generate a variation characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point, and finally generate the dynamic sampling parameter matrix according to the variation characteristic parameter matrix of the sampled historical data. The dynamic sampling parameter matrix comprises dynamic sampling periods and variation values.
The acquisition algorithm shown in fig. 3, wherein the matrix a is an inherent physical characteristic parameter table of the sampling points, the matrix B is a change correlation characteristic parameter table extracted from the sampled historical data, the matrix C is a dynamic sampling period and change value matrix, and the matrix D is a new sampling value.
After the program initially runs, an inherent physical characteristic parameter matrix A is generated, a change characteristic parameter matrix B is generated according to A, then a sampling period and a change value matrix C are generated by B, and when an acquisition task is performed, a superior acquires data of a subordinate memory base according to C to obtain a new sampling value D. Then, the matrixes A, B and D are synthesized, and the data are reconstructed by using a greedy algorithm with low complexity and less time consumption and are stored in a memory.
In the Internal Data exchange standards (IDE) provided in the embodiment of the present invention, each lower node is used as an RTU (remote terminal unit), and sends Data according to the dynamic sampling parameter matrix (matrix C) or directly sends Data according to the dynamic sampling parameter matrix (matrix C) after receiving the upper-level Data acquisition command. The IDE may connect the upper level host and the lower level host in a communication interface manner during a specific application.
Specifically, the sampling data may be collected and sent by the lower level host after confirming that the data changes, and the sampling data may be periodically collected and sent by the lower level host; the sampling data can also be acquired and sent by the lower host after receiving the data acquisition command sent by the upper host. In addition, the data acquisition command may be periodically transmitted by the upper level host.
That is, the IDE may employ two protocol modes of master-slave and passive reception:
1. the upper host (center) sends a data acquisition command, and the lower host (station) sends data after receiving the command;
2. the lower host (station) transmits the change, and the upper host (center) receives the change passively.
The IDE may also employ cycle and event driven acquisition:
1. lower host periodic transmission or upper host periodic transmission data acquisition command II
2. And transmitting a data acquisition command by the data change drive of the lower host or the event drive of the upper host. The event driver includes data change, system switching, task scheduling, and other driver acquisition, for example: section data, change data.
And 202, the superior host performs data reconstruction according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data and the sampled data.
And after the upper host receives the sampled data sent by the lower host, reconstructing the data by using a greedy algorithm according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data and the sampled data.
Meanwhile, the superior host can also correct the change characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point, the change characteristic parameter matrix of the sampled historical data and the sampled data, and then generate a dynamic sampling parameter matrix according to the modified change characteristic parameter matrix of the sampled historical data.
For example, after obtaining a new sampling value D, the host of the higher level corrects B according to the new sampling value D and the a and B themselves to obtain a dynamic sampling parameter matrix C, and when a new acquisition task arrives, the higher level acquires according to the new matrix C, and so on.
In order to better explain the embodiment of the present invention, the above-mentioned data acquisition process will be described in a specific implementation scenario.
Example one
IDE master-slave protocol data collection flow:
the DAQ of the upper host sends a data acquisition command to the RTU of the lower host through the IDE protocol, and the RTU of the lower host receives the data acquisition command, and then acquires data from the lower memory bank according to the dynamic sampling parameter matrix C, and sends the acquired data to the DAQ of the upper host through the IDE protocol, as shown in fig. 4, the IDE is a communication protocol between the upper host and the lower host, which belongs to the case where the upper host sends a data sampling command to the lower host, and the lower host feeds back the acquired data.
Example two
IDE passive reception protocol data acquisition procedure:
the RTU of the subordinate host periodically collects data from the subordinate memory bank according to the dynamic sampling parameter matrix C, and sends the collected data to the DAQ of the superior host through the IDE protocol, as shown in fig. 5.
EXAMPLE III
IDE master-slave protocol scheduling/handover acquisition procedure:
a lower node (station) is regarded as an RTU, and in a distributed system, a processing node to which a task belongs may be changed by scheduling/switching. When a master-slave protocol is adopted and when an upper task node is switched/scheduled, a higher node sends a command to the lower node through an IDE protocol by the DAQ of a changed task processing node, as shown in FIG. 6, the upper node 1 is a task node before scheduling/switching, the upper node 2 is a task node after scheduling/switching, and when the node scheduling/switching occurs, the DAQ of the upper node 2 sends a BAS command after scheduling/switching; and after receiving the command through the IDE protocol, the RTU of the lower node acquires data according to the dynamic sampling parameter matrix C and sends the data to the DAQ of the upper node 2 through the IDE protocol. When the lower task node is switched/scheduled, the DAQ of the upper node sends a command to the node which processes the task at the lower node; as shown in fig. 6, the lower node 3 is a task node before scheduling/switching, the lower node 4 is a task node after scheduling/switching, when node scheduling/switching occurs, the DAQ of the upper node 3 sends a BAS command after scheduling/switching, and the RTU of the lower node 4 receives the command, collects data according to the dynamic sampling parameter matrix C, and sends the data to the DAQ of the upper node 3 through the IDE protocol.
Example four
IDE passive reception scheduling/handover acquisition procedure:
when a passive receiving protocol is adopted and when the task of the upper node changes, the acquisition protocol of the lower data node changes the channel IP address according to the scheduling result of the upper task and sends data to the new upper node. As shown in fig. 7, the upper node 1 is a task node before scheduling/switching, the upper node 2 is a task node after scheduling/switching, and when node scheduling/switching occurs, the RTU of the lower node collects data according to the dynamic sampling parameter matrix C and sends the data to the DAQ of the upper node 2 by changing the channel IP using the IDE protocol. When the task of the lower node changes, the lower data sending node automatically changes according to the task change, and changes the sent data, as shown in fig. 7, the lower node 3 is a task node before scheduling/switching, the lower node 4 is a task node after scheduling/switching, and when the node scheduling/switching occurs, the RTU of the lower node 4 collects data and sends the data to the DAQ of the upper node 3 through the IDE protocol.
The embodiment of the invention provides a large-scale data acquisition algorithm for the field of data acquisition and monitoring, which is based on a compressed sensing theory. The compressed sensing breaks through the traditional Nyquist sampling theorem, the signal can be sampled at a rate far lower than the Nyquist sampling rate, only a small number of samples are collected to restore the original data by utilizing the redundancy characteristic of the data, the necessary condition is that the signal is sparse, and the data of the collected object in the integrated monitoring system meets the requirement on the signal in the compressed sensing theory. According to the scheme, according to the 'compressed sensing theory', aiming at the characteristics of the industrial data acquisition monitoring field, by analyzing the physical characteristics and data correlation of the acquired data, a fixed acquisition point is replaced by a small acquisition point; the dynamic acquisition period replaces the fixed acquisition period, namely the physical characteristics of the acquired historical data are acquired through an intelligent algorithm to determine that the next maximum acquisition period replaces the fixed acquisition period, the full data is recovered through the intelligent algorithm, and the precision and the quantity of the acquired data are ensured, so that the requirements of the system are met.
In the embodiment of the invention, a superior host acquires sampling data sent by a subordinate host, the sampling data is obtained by the subordinate host by collecting data in a memory bank of the subordinate host according to a dynamic sampling parameter matrix, the dynamic sampling parameter matrix is determined according to an inherent physical characteristic parameter matrix of a sampling point, and data reconstruction is carried out according to the inherent physical characteristic parameter matrix of the sampling point, a variation characteristic parameter matrix of sampled historical data and the sampling data. The subordinate host performs data compression and acquisition through the dynamic sampling parameter matrix, and then performs data reconstruction through the superior host, so that the system resource consumption in the data acquisition process can be reduced, the data acquisition efficiency is improved, and the method is suitable for large-scale data acquisition.
Based on the same technical concept, fig. 8 exemplarily shows a large-scale data acquisition apparatus based on compressive sensing theory according to an embodiment of the present invention, which may perform a process of a large-scale data acquisition method based on compressive sensing theory.
As shown in fig. 8, the apparatus specifically includes:
an obtaining unit 801, configured to obtain sampling data sent by a lower host, where the sampling data is obtained by the lower host by collecting data in a memory bank of the lower host according to a dynamic sampling parameter matrix; the dynamic sampling parameter matrix is determined according to the inherent physical characteristic parameter matrix of the sampling point;
the processing unit 802 is configured to perform data reconstruction according to the inherent physical characteristic parameter matrix of the sampling point, the change characteristic parameter matrix of the sampled historical data, and the sampled data.
Optionally, the processing unit 802 is specifically configured to:
acquiring an inherent physical characteristic parameter matrix of a sampling point;
generating a variation characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point;
and generating the dynamic sampling parameter matrix according to the variation characteristic parameter matrix of the sampled historical data.
Optionally, the processing unit 802 is further configured to:
after data reconstruction is carried out according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data and the sampled data, the variation characteristic parameter matrix of the sampled historical data is corrected according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data and the sampled data;
and generating the dynamic sampling parameter matrix according to the modified change characteristic parameter matrix of the sampled historical data.
Optionally, the processing unit 802 is specifically configured to:
and the upper host performs data reconstruction by using a greedy algorithm.
Optionally, the sampling data is collected and sent by the lower host after confirming that the data changes, or periodically collected and sent by the lower host, or collected and sent by the lower host after receiving a data collection command sent by the upper host.
Optionally, the data acquisition command is periodically sent by the upper level host.
Based on the same technical concept, an embodiment of the present invention further provides a computing device, including:
a memory for storing program instructions;
and the processor is used for calling the program instructions stored in the memory and executing the large-scale data acquisition method based on the compressive sensing theory according to the obtained program.
Based on the same technical concept, the embodiment of the invention also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used for enabling a computer to execute the large-scale data acquisition method based on the compressive sensing theory.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A large-scale data acquisition method based on compressed sensing theory is characterized by comprising the following steps:
a superior host acquires sampling data sent by a subordinate host, wherein the sampling data is acquired by the subordinate host by collecting data in a memory bank of the subordinate host according to a dynamic sampling parameter matrix; the dynamic sampling parameter matrix is determined according to the inherent physical characteristic parameter matrix of the sampling point;
and the superior host performs data reconstruction according to the inherent physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data and the sampled data.
2. The method of claim 1, wherein the superior host determines the dynamic sampling parameter matrix according to an intrinsic physical characteristic parameter matrix of sampling points, comprising:
the superior host acquires an inherent physical characteristic parameter matrix of a sampling point;
the superior host generates a variation characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point;
and the superior host generates the dynamic sampling parameter matrix according to the variation characteristic parameter matrix of the sampled historical data.
3. The method as claimed in claim 1, wherein the superior host further comprises, after performing data reconstruction according to the intrinsic physical characteristic parameter matrix of the sampling point, the variation characteristic parameter matrix of the sampled historical data, and the sampled data:
the superior host corrects the change characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point, the change characteristic parameter matrix of the sampled historical data and the sampled data;
and the superior host generates the dynamic sampling parameter matrix according to the modified change characteristic parameter matrix of the sampled historical data.
4. The method of claim 1, wherein the superordinate host performs data reconstruction including:
and the upper host performs data reconstruction by using a greedy algorithm.
5. The method of any of claims 1 to 4, wherein the sampled data is collected and sent by the subordinate host after a change in the confirmation data or periodically or after a data collection command sent by the superior host is received by the subordinate host.
6. The method of claim 5, wherein the data collection command is periodically transmitted by the superior host.
7. A large-scale data acquisition device based on compressed sensing theory is characterized in that the device comprises:
the device comprises an acquisition unit, a storage unit and a processing unit, wherein the acquisition unit is used for acquiring sampling data sent by a lower host, and the sampling data is acquired by the lower host by collecting data in a memory bank of the lower host according to a dynamic sampling parameter matrix; the dynamic sampling parameter matrix is determined according to the inherent physical characteristic parameter matrix of the sampling point;
and the processing unit is used for carrying out data reconstruction according to the inherent physical characteristic parameter matrix of the sampling point, the change characteristic parameter matrix of the sampled historical data and the sampled data.
8. The apparatus as claimed in claim 7, wherein said processing unit is specifically configured to:
acquiring an inherent physical characteristic parameter matrix of a sampling point;
generating a variation characteristic parameter matrix of the sampled historical data according to the inherent physical characteristic parameter matrix of the sampling point;
and generating the dynamic sampling parameter matrix according to the variation characteristic parameter matrix of the sampled historical data.
9. A computing device, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory to execute the method of any one of claims 1 to 6 in accordance with the obtained program.
10. A computer-readable storage medium having stored thereon computer-executable instructions for causing a computer to perform the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010232080.9A CN111447229B (en) | 2020-03-27 | 2020-03-27 | Large-scale data acquisition method and device based on compressed sensing theory |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010232080.9A CN111447229B (en) | 2020-03-27 | 2020-03-27 | Large-scale data acquisition method and device based on compressed sensing theory |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111447229A true CN111447229A (en) | 2020-07-24 |
| CN111447229B CN111447229B (en) | 2022-03-11 |
Family
ID=71654091
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010232080.9A Active CN111447229B (en) | 2020-03-27 | 2020-03-27 | Large-scale data acquisition method and device based on compressed sensing theory |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111447229B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113452022A (en) * | 2021-07-06 | 2021-09-28 | 中电科技集团重庆声光电有限公司 | Method and system for collecting power consumption data of equipment |
| CN113515673A (en) * | 2021-04-30 | 2021-10-19 | 山西大学 | High-value object storage and transportation whole-process information compression and reconstruction method and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103595414A (en) * | 2012-08-15 | 2014-02-19 | 王景芳 | Sparse sampling and signal compressive sensing reconstruction method |
| CN107147396A (en) * | 2016-03-01 | 2017-09-08 | 华为技术有限公司 | Signal sampling method, signal sampling system and signal sampling device |
| CN107231250A (en) * | 2017-04-25 | 2017-10-03 | 全球能源互联网研究院 | One kind is based on electric network information physical system perception data compressive sampling method and device |
| CN107786959A (en) * | 2017-10-18 | 2018-03-09 | 北京京航计算通讯研究所 | The compressed data collection method based on adaptive measuring in wireless sensor network |
| CN108599773A (en) * | 2018-04-16 | 2018-09-28 | 兰州理工大学 | A kind of vibration signal data compression acquisition method based on certainty calculation matrix |
| CN108924148A (en) * | 2018-07-18 | 2018-11-30 | 中南大学 | A kind of source signal collaborative compression perception data restoration methods |
-
2020
- 2020-03-27 CN CN202010232080.9A patent/CN111447229B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103595414A (en) * | 2012-08-15 | 2014-02-19 | 王景芳 | Sparse sampling and signal compressive sensing reconstruction method |
| CN107147396A (en) * | 2016-03-01 | 2017-09-08 | 华为技术有限公司 | Signal sampling method, signal sampling system and signal sampling device |
| CN107231250A (en) * | 2017-04-25 | 2017-10-03 | 全球能源互联网研究院 | One kind is based on electric network information physical system perception data compressive sampling method and device |
| CN107786959A (en) * | 2017-10-18 | 2018-03-09 | 北京京航计算通讯研究所 | The compressed data collection method based on adaptive measuring in wireless sensor network |
| CN108599773A (en) * | 2018-04-16 | 2018-09-28 | 兰州理工大学 | A kind of vibration signal data compression acquisition method based on certainty calculation matrix |
| CN108924148A (en) * | 2018-07-18 | 2018-11-30 | 中南大学 | A kind of source signal collaborative compression perception data restoration methods |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113515673A (en) * | 2021-04-30 | 2021-10-19 | 山西大学 | High-value object storage and transportation whole-process information compression and reconstruction method and system |
| CN113452022A (en) * | 2021-07-06 | 2021-09-28 | 中电科技集团重庆声光电有限公司 | Method and system for collecting power consumption data of equipment |
| CN113452022B (en) * | 2021-07-06 | 2022-09-23 | 中电科技集团重庆声光电有限公司 | Method and system for collecting power consumption data of equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111447229B (en) | 2022-03-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111447229B (en) | Large-scale data acquisition method and device based on compressed sensing theory | |
| CN103379136A (en) | Compression method and decompression method of log acquisition data, compression apparatus and decompression apparatus of log acquisition data | |
| CN101345652B (en) | Data acquisition method and data acquisition equipment | |
| CN104215853A (en) | Data acquiring method of power quality monitoring system | |
| CN108009089B (en) | Incremental machine learning method and system based on transparent computing | |
| CN114969047A (en) | Data processing system and method | |
| CN101662389A (en) | Method for carrying out statistics on performance data and device applying same | |
| CN109788251B (en) | Video processing method, device and storage medium | |
| CN112335203A (en) | Processing local area network diagnostic data | |
| WO2021147319A1 (en) | Data processing method, apparatus, device, and medium | |
| CN112072783B (en) | Method and device for transmitting second-level load data between end-side equipment and edge-side equipment | |
| CN103607731A (en) | Method and device for processing measurement reports | |
| CN106953809B (en) | 485 time-sharing communication-based equipment resource acquisition method | |
| CN113342526A (en) | Dynamic management and control method, system, terminal and medium for cloud computing mobile network resources | |
| CN117221354A (en) | Multi-source heterogeneous data real-time acquisition, storage and analysis method and system | |
| CN116980491A (en) | Frequency domain transformation-based data compression transmission method and device for low-voltage power network | |
| CN115396752A (en) | Redis-based biplane data acquisition method and system | |
| CN112422525B (en) | Fault data transmission method, device, equipment and storage medium | |
| CN112732810A (en) | Data transmission system, data transmission method, data transmission device, storage medium, and electronic device | |
| Hou et al. | Design scheme for data transmission component of electric internet of things management platform | |
| CN116186613B (en) | Intelligent acquisition processing method and system for industrial Internet data | |
| CN115051714A (en) | Waveform data compression method based on FFT (fast Fourier transform) | |
| CN104302017A (en) | Pretreatment method for wavelet data compression in sensor network | |
| CN115685817B (en) | Method, device and medium for processing data concurrency during CAN (controller area network) multi-node communication | |
| CN117762646B (en) | Digital quantity processing method and system based on cluster type shared cache |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |