CN110277998B - Lossless compression method and device for power grid data - Google Patents
Lossless compression method and device for power grid data Download PDFInfo
- Publication number
- CN110277998B CN110277998B CN201910567393.7A CN201910567393A CN110277998B CN 110277998 B CN110277998 B CN 110277998B CN 201910567393 A CN201910567393 A CN 201910567393A CN 110277998 B CN110277998 B CN 110277998B
- Authority
- CN
- China
- Prior art keywords
- data
- compression
- algorithm
- sequence
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007906 compression Methods 0.000 title claims abstract description 184
- 230000006835 compression Effects 0.000 title claims abstract description 178
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000005070 sampling Methods 0.000 claims abstract description 174
- 230000005540 biological transmission Effects 0.000 claims abstract description 44
- 238000013144 data compression Methods 0.000 claims description 43
- 230000000295 complement effect Effects 0.000 claims description 34
- 230000000153 supplemental effect Effects 0.000 claims description 23
- 230000006837 decompression Effects 0.000 claims description 12
- 230000000737 periodic effect Effects 0.000 claims description 11
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 238000004891 communication Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000036039 immunity Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention discloses a lossless compression method and device for power grid data. The method comprises the following steps: processing the acquired original data, and determining the acquisition precision and/or the sampling frequency of a single sampling point; extracting feature fields corresponding to the number of each compression sub-algorithm according to a plurality of preset compression sub-algorithms, wherein the feature fields are as follows: a single sampling point, or all sampling points describing one cycle; determining a compression ratio obtained when each compression sub-algorithm processes the original data according to a characteristic field corresponding to each compression sub-algorithm and the data width of the characteristic field; determining a target compression sub-algorithm matched with the target compression ratio according to a preset target compression ratio; and processing the original data according to the target compression sub-algorithm and the corresponding characteristic field, and generating a characteristic sequence and a supplementary description data sequence. The method reduces the bandwidth occupation time of the transmission channel and improves the transmission efficiency; is beneficial to realizing low-speed and high-efficiency transmission.
Description
Technical Field
The invention belongs to the technical field of data compression, and particularly relates to a lossless compression method and device for power grid data.
Background
At present, when the equipment for processing the power grid data is used for transmitting the power grid data, the power grid data is mostly not compressed, so that the transmission burden on a transmission channel is heavy, and the transmission efficiency is low.
Especially when devices communicate using respective local interfaces (such as serial, firewire, GPIB, USB, etc.), in order to transmit power grid data in real time, the communication rate of the local interfaces needs to be increased to increase the data transmission speed (such as baud rate when serial communication), thereby increasing the power consumption of the devices. In addition, in some scenarios, increasing the communication rate of the local interface will reduce the interference immunity of the device.
Disclosure of Invention
The invention provides a lossless compression method and device for power grid data, which are used for solving the problems of large data transmission amount and large power consumption of transmission equipment of the existing power grid data.
In a first aspect, the present invention provides a lossless compression method for power grid data, including the following steps:
processing the acquired original data, and determining the acquisition precision and/or the sampling frequency of a single sampling point;
extracting feature fields corresponding to the number of each compression sub-algorithm according to a plurality of preset compression sub-algorithms, wherein the feature fields are as follows: a single sampling point, or all sampling points describing one cycle;
determining a compression ratio obtained when each compression sub-algorithm processes the original data according to a characteristic field corresponding to each compression sub-algorithm and the data width of the characteristic field;
determining a target compression sub-algorithm matched with the target compression ratio according to a preset target compression ratio;
and processing the original data according to the target compression sub-algorithm and the corresponding characteristic field, and generating a characteristic sequence and a supplementary description data sequence.
Further, the lossless compression method of the power grid data further comprises the following steps after the feature sequence and the supplementary description data sequence are generated:
filling the characteristic field, the data width of the characteristic field and the algorithm identification of the target compression sub-algorithm into a data head part of a data frame to be transmitted; and
Filling the feature sequence and the supplementary description data sequence into a data body part of a data frame to be transmitted;
and sending the data frame to be transmitted to target equipment.
Further, in the lossless compression method for grid data, the processing the original data according to the target compression sub-algorithm and the corresponding feature field, and generating a feature sequence and a supplementary description data sequence, includes:
taking the data width of the characteristic field as the step length when the original data is processed;
for the raw data of each data width,
setting the corresponding current position in the characteristic sequence as a compression effective identifier when the target compression sub-algorithm is executed; and append the supplementary description difference value corresponding to the original data of the data width determined by the target compression sub-algorithm at the end of the supplementary description data sequence;
when the target compression sub-algorithm is not executed, setting the current bit corresponding to the feature sequence as a compression invalid identifier; and appends the original data at the end of the supplemental description data sequence.
Further, in the lossless power grid data compression method, the compression sub-algorithm is arithmetic data compression:
taking the characteristic field as a first sampling point of the original data;
for each sample point of the original data,
and the value of the sampling point is differenced from the value of the previous sampling point, and the obtained difference value is a supplementary description difference value corresponding to the sampling point in the supplementary description data sequence.
Further, in the lossless compression method for the power grid data, the compression sub-algorithm is equal-ratio data compression:
taking the characteristic field as a first sampling point of the original data;
for each sample point of the original data,
and dividing the value of the sampling point by the value of the previous sampling point to obtain a quotient value which is a complementary description difference value corresponding to the sampling point in the complementary description data sequence.
Further, in the lossless power grid data compression method, the compression sub-algorithm is periodic data compression:
taking the characteristic field as all sampling points in the first complete voltage cycle or current cycle in the original data;
dividing other sampling points or all sampling points in the original data into a plurality of data groups to be processed by taking the number of the sampling points in the voltage cycle or the current cycle as an interval;
for each of said sets of data to be processed,
and subtracting or dividing the data set to be processed from the previous data set to be processed point by point, wherein the obtained difference value or quotient value is a complementary description difference value corresponding to the data to be processed in the complementary description data sequence.
Further, in the lossless power grid data compression method, the compression sub-algorithm is sinusoidal data compression:
taking the characteristic field as all sampling points in the first complete sine cycle in the original data;
dividing other sampling points or all sampling points in the original data into a plurality of data groups to be processed by taking the number of all sampling points in the sine cycle as an interval;
for each of said sets of data to be processed,
and subtracting or dividing the data set to be processed from the previous data set to be processed point by point, wherein the obtained difference value or quotient value is a complementary description difference value corresponding to the data to be processed in the complementary description data sequence.
Further, in the lossless compression method for the power grid data, the compression ratio is as follows:
(length of feature sequence + length of supplemental description data sequence)/length of original data.
Further, the lossless compression method for power grid data further includes, after sending the data frame to be transmitted to a target device:
extracting a feature field, a data width of the feature field, a compression algorithm identification, a feature sequence and a supplementary feature data sequence from the received data frame,
executing a decompression sub-algorithm corresponding to the compression algorithm identification to obtain original power grid data; wherein,
and the decompression sub-algorithm corresponding to the compression algorithm identifier and the compression sub-algorithm corresponding to the compression algorithm identifier are in reciprocal operation.
In a second aspect, the present invention provides a lossless compression apparatus for grid data, comprising:
a source device for executing the data compression method described above;
the target device is used for executing the data decompression method;
the source device and the target device are communicatively connected using a low speed transmission channel.
Compared with the prior art, the lossless compression method and device for the power grid data provided by the invention are used for compressing the waveform data by adopting the characteristic field before transmission on the source equipment side aiming at the characteristic of high similarity of the power grid data, so that the data volume is reduced and the transmission load is lightened; the bandwidth occupation time of the transmission channel on the source equipment and the target equipment on two sides is reduced, and the transmission efficiency is improved; the method is beneficial to reducing the data transmission speed of the source equipment and the target equipment at the two sides of the transmission channel, and realizes high-efficiency data transmission on a low-rate communication medium.
Drawings
Exemplary embodiments of the present invention may be more completely understood in consideration of the following drawings:
FIG. 1 is a flow chart of a lossless compression method for power grid data according to a preferred embodiment of the present invention;
FIG. 2 is a schematic diagram of the power grid data lossless compression apparatus according to the preferred embodiment of the present invention;
fig. 3 is a flow chart of a waveform profile data compression method according to a preferred embodiment of the present invention.
Detailed Description
The exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, however, the present invention may be embodied in many different forms and is not limited to the examples described herein, which are provided to fully and completely disclose the present invention and fully convey the scope of the invention to those skilled in the art. The terminology used in the exemplary embodiments illustrated in the accompanying drawings is not intended to be limiting of the invention. In the drawings, like elements/components are referred to by like reference numerals.
Unless otherwise indicated, terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. In addition, it will be understood that terms defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense.
Currently, in various scenarios, grid data is transmitted between different devices. Such devices typically utilize a local interface and employ a low rate communication medium, such as coaxial cable or USB data lines, in transmitting the grid data. Currently, in the data transmission process, a large amount of similar waveform data needs to be transmitted by occupying bandwidth for a long time. Therefore, the transmission efficiency of the power grid data is low, and the power consumption of the equipment is high.
For example, when the power grid data acquired by the single-phase metering chip or the three-phase metering chip in real time is transmitted to each device in the intelligent substation, the data transmission efficiency is low, the power consumption of the device is high, and even the communication rate of the local interface at a higher level obviously reduces the anti-interference capability of the device.
Therefore, when the source equipment side at the two ends of the transmission channel transmits data to the target equipment side by utilizing a pre-agreed communication protocol, the network data is refilled into a data frame after being processed, and the data frame is used as a data load for transmission, so that the method for improving the data transmission efficiency is feasible.
In the embodiment of the invention, when target equipment at one end of a transmission channel transmits data by utilizing a preset communication protocol, the original power grid data acquired by a metering chip in real time is processed and then is used as a load part in a data frame for transmission; when the target equipment at the other end of the transmission channel receives data by utilizing a preset communication protocol, a load part is extracted from a data frame, and after decompression processing, the original power grid data acquired by the metering chip in real time is obtained.
The data compression method provided by the embodiment of the invention is used for processing the original power grid data, so that the scale of the data is obviously reduced, and the data transmission efficiency is improved.
According to the lossless compression method and device for the power grid data, aiming at the characteristic of high similarity of the power grid data, at the source equipment side, the waveform data is compressed by adopting the characteristic field before transmission, so that the data volume is reduced, and the transmission load is lightened; the bandwidth occupation time of source equipment and target equipment on two sides of a transmission channel is reduced, and the transmission efficiency is improved; the method is beneficial to reducing the data transmission speed of the source equipment and the target equipment at two sides of the transmission channel, and realizes low-speed and high-efficiency transmission.
The lossless compression method and device for the power grid data are particularly suitable for being applied to scenes with limited communication rate of a local interface.
The definition of each term is given below:
the curve data length is the storage space occupied by the data to be compressed, namely the byte number;
the characteristic field is the original data of one or a plurality of continuous sampling points;
the length of the characteristic field, namely the data width of the characteristic field, is the step length of each compression operation in data processing; the original curve data are processed one by one with the step length as an interval to form a characteristic sequence and a supplementary description data sequence;
the characteristic sequence is a sequence constructed by taking bits as a unit, and each bit is used for describing whether the compressed data is identical to the original data one by one according to the data width or the step length as an interval.
The length (in bytes) of the signature sequence is:
curve data length/data width/8;
as a qualitative description part, the feature sequence occupies a small storage space and can be considered as a summary part of the data.
The supplemental description data sequence is used for storing the corresponding supplemental description difference value or the original sampling point data of the original sampling point which is the same as the data width of the characteristic field;
the supplemental description data sequence is interdependent with the feature series;
if the bit of the corresponding feature sequence is 1 in the supplementary description data sequence, the supplementary description difference value corresponding to the original sampling point data is on the corresponding data width, and at this time, the data width is the compressed data width; if the bit of the corresponding feature sequence is 0, the corresponding data width is the original sampling point data, and at this time, the data width is the data width before compression.
That is, there are two types of data widths in the supplemental description data sequence; and the data width before compression is not smaller than the data width after compression.
After the compression in the above way, the compression ratio is:
(length of feature sequence + length of supplemental description data sequence)/length of original curve data
It should be appreciated that the definition of compression ratio above ignores the data width of the feature field, and the memory space occupied by the identification describing the sub-compression algorithm.
As shown in fig. 1, the lossless compression method for the power grid data in one embodiment includes the following steps:
s10: processing the acquired original data, and determining the acquisition precision and/or the sampling frequency of a single sampling point;
s20: extracting feature fields corresponding to the number of each compression sub-algorithm according to a plurality of preset compression sub-algorithms, wherein the feature fields are as follows: a single sampling point, or all sampling points describing one cycle;
s30: determining a compression ratio obtained when each compression sub-algorithm processes the original data according to the characteristic field and the data width of the characteristic field corresponding to each compression sub-algorithm;
s40: determining a target compression sub-algorithm matched with the target compression ratio according to a preset target compression ratio;
s50: and processing the original data according to the target compression sub-algorithm and the corresponding characteristic field, and generating a characteristic sequence and a supplementary description data sequence.
Specifically, after generating the feature sequence and the supplementary description data sequence, the method further includes:
filling the characteristic field, the data width of the characteristic field and the algorithm identification of the target compression sub-algorithm into the data head part of the data frame to be transmitted; and
Filling the feature sequence and the supplementary description data sequence into a data body part of a data frame to be transmitted;
and sending the data frame to be transmitted to the target equipment.
Specifically, according to the target compression sub-algorithm and the corresponding feature field, processing the original data, and generating a feature sequence and a supplementary description data sequence, including:
taking the data width of the characteristic field as the step length when processing the original data;
for the raw data of each data width,
setting the corresponding current position in the characteristic sequence as a compression effective identifier when executing the target compression sub-algorithm; and append the supplementary description difference value corresponding to the original data of the data width determined by the target compression sub-algorithm at the end of the supplementary description data sequence;
when the target compression sub-algorithm is not executed, setting the current bit corresponding to the feature sequence as a compression invalid identifier; and appends the original data at the end of the supplemental description data sequence.
Specifically, the compression sub-algorithm is arithmetic data compression:
taking the characteristic field as a first sampling point of the original data;
for each sample point of the original data,
and (3) making a difference between the value of the sampling point and the value of the previous sampling point, wherein the obtained difference is a complementary description difference value corresponding to the sampling point in the complementary description data sequence.
Specifically, the compression sub-algorithm is equal ratio data compression:
taking the characteristic field as a first sampling point of the original data;
for each sample point of the original data,
and dividing the value of the sampling point by the value of the previous sampling point to obtain a quotient value which is a complementary description difference value corresponding to the sampling point in the complementary description data sequence.
Specifically, according to the lossless power grid data compression method, the compression sub-algorithm is periodic data compression:
taking the characteristic field as all sampling points in the first complete voltage cycle or current cycle in the original data;
dividing other sampling points or all sampling points in the original data into a plurality of data groups to be processed by taking the number of the sampling points in the voltage cycle or the current cycle as an interval;
for each data set to be processed,
and subtracting or dividing the data set to be processed from the previous data set to be processed point by point, wherein the obtained difference value or quotient value is a complementary description difference value corresponding to the data to be processed in the complementary description data sequence.
Specifically, the compression sub-algorithm is sinusoidal data compression:
taking the characteristic field as all sampling points in the first complete sine cycle in the original data;
dividing other sampling points or all sampling points in the original data into a plurality of data groups to be processed by taking the number of all sampling points in the sine cycle as an interval;
for each data set to be processed,
and subtracting or dividing the data set to be processed from the previous data set to be processed point by point, wherein the obtained difference value or quotient value is a complementary description difference value corresponding to the data to be processed in the complementary description data sequence.
Specifically, the compression ratio is:
(length of feature sequence + length of supplemental description data sequence)/length of original data.
Specifically, after sending the data frame to be transmitted to the target device, the method further includes:
extracting a feature field, a data width of the feature field, a compression algorithm identification, a feature sequence and a supplementary feature data sequence from the received data frame,
executing a decompression sub-algorithm corresponding to the compression algorithm identification to obtain original power grid data; wherein,
the decompression sub-algorithm corresponding to the compression algorithm identification and the compression sub-algorithm corresponding to the compression algorithm identification are reciprocal operations.
As shown in fig. 2, the present invention provides a lossless compression apparatus 1000 for power grid data, including:
a source device 100 for performing the above-described data compression method;
a target device 200 for performing the above-described data decompression method;
the source device 100 and the target device 200 are communicatively coupled using a low-speed transmission channel.
It should be understood that at the first moment, two ends of a low-speed transmission channel are provided, wherein the first data end is the source device 100 and the second data end is the destination device 200; at the second moment, at both ends of the low-speed transmission channel, the first data end is the target device 200, and the second data end is the source device 100. That is, either one of the devices at both ends of the low-speed transmission channel performs operations of data compression and data transmission or data reception and data decompression in a time-sharing manner.
It should be appreciated that grid data is typically transferred between a source device and a target device in accordance with some protocol. Each data frame of the protocol includes a frame header portion and a frame data field portion, wherein the frame data field portion is a payload of the data frame. The compression method involves only filling the contents into the data header part or the data body part of the data area part; the processing steps of the frame header portion may be in a manner disclosed in the prior art, and will not be described herein.
Typically, the grid frequency is a constant value (e.g., 50Hz in China; 60Hz in Japan and Europe), and the duration of each voltage cycle or current cycle is the inverse of the grid frequency. In the sampling data acquired by the single-phase metering chip or the three-phase metering chip, the number of sampling points (namely the sampling frequency or the data width in the above) and the sampling data length included in each voltage cycle or each current cycle are determined by the sampling precision and the parameter setting (such as resolution or precision level) of the metering chip:
for example, the resolution of the metering chip may be 64 bits or 32 bits or 16 bits, so the number of bytes occupied by the data of one sampling point is 8 bytes at the resolution of 64 bits; at 32-bit resolution, 4 bytes; at 16-bit resolution, it is 2 bytes.
For example, the sampling accuracy (e.g., sampling frequency) of the metering chip may correspond to 16 sampling points for one voltage cycle or current cycle, 8 sampling points for one voltage cycle, 64 sampling points for one current cycle, or the like.
Therefore, in the original data returned by the metering chip, the data quantity corresponding to the voltage cycle or the current cycle of the multiple channels can be determined according to the following formula:
sampling frequency x acquisition accuracy of a single sampling point x number of channels.
Therefore, in the data returned by the metering chip, the data amount corresponding to the voltage cycle or the current cycle of the single channel in each second can be determined according to the following formula:
grid frequency x sampling frequency x acquisition accuracy of a single sampling point.
Generally, the channels corresponding to the single-phase metering chip are respectively three voltage channels, current channels and zero sequence current channels; that is, the single-phase metrology chip returns waveform profile data for the three channels.
The channels corresponding to the three-phase metering chip comprise 3 groups of single-phase voltage (phase voltage or line voltage) channels, 3 groups of single-phase current or current channels and a zero-sequence current channel; that is, the three-phase metrology chip returns waveform profile data for the seven channels.
In the above, the data size of the original data to be compressed, that is, the length of the original sampled data can be determined according to the number of channels, the data width, and the sampling accuracy.
It should be understood that the sampled data is transmitted sequentially on a single channel, i.e., within each data frame, only sampled data from a single sampled channel is transmitted.
On the other hand, the sampling accuracy and data accuracy of the multiple channels may vary, and thus each channel independently uses a respective compression algorithm, including a feature field, a feature sequence, and a supplemental description data sequence of a respective data width.
Processing the acquired original data to determine the acquisition precision and/or sampling frequency of a single sampling point, wherein the method specifically comprises the following steps:
and traversing the waveform curve data in the acquired original data, extracting the correlation between the sampling point data, and determining the acquisition precision and/or the sampling frequency of a single sampling point.
The correlation between sample point data reflects the data pattern, which can be used to describe the entire raw data using the feature fields and data pattern:
for example, a certain sampling point is used as a characteristic field, and the difference value or quotient between other sampling points and the characteristic field is used for describing other sampling points;
for example, a certain frequency is used as a characteristic field, and the difference or quotient between each point of other frequencies and each point of the characteristic field is used for describing other frequencies.
When different data compression algorithms are employed for each batch of data to be transmitted, the available compression ratios are different.
Defining a compression ratio as a ratio of the size of a storage space occupied by data before and after compression; the larger the compression ratio is, the less transmission bandwidth is occupied by the compressed data, and the transmission efficiency is higher.
The same compression algorithm can be adopted all the time for curve waveform data of a single channel; it is also possible to advance over time to replace with a different compression algorithm.
For grid data, a variety of sub-compression algorithms may be employed, including: arithmetic data compression, geometric data compression, periodic data compression, sinusoidal compression, and the like. The compression algorithm identification bit may be employed to identify the target compression algorithm employed when assembling the data frame.
For any single channel data obtained from the sampling chip, a sub-compression algorithm with the optimal compression ratio can be selected as the compression algorithm of the current original data.
The arithmetic data compression is a single point compression method. Specifically, taking a characteristic field as a sampling point or a first sampling point with the largest or smallest numerical value in a waveform curve;
for each sample point of the original data,
and (3) making a difference between the value of the sampling point and the value of the previous sampling point, wherein the obtained difference is a complementary description difference value corresponding to the sampling point in the complementary description data sequence.
For example, the acquisition precision of each sampling point in the waveform curve is 4 bytes; the data width corresponding to the difference between the values of any two sampling points in the waveform curve is 2 bytes, and the compression ratio is approximately 2 after the arithmetic data compression. The approximation here refers to ignoring the data length of the feature sequence.
Taking the difference value between the characteristic field and the first sampling point of the original data as the supplementary description difference value of the first sampling point, and performing incremental comparison with the previous sampling point by point from the second sampling point of the original data:
if the feature field is 0x1000, the value of the first sample point immediately after it is: the value of the second sample point is 0x 1005: 0x1015; then
In the supplementary description data sequence, the corresponding supplementary description difference values may be expressed as: 0x05 (i.e., 0x1005-0x 1000), 0x10 (0 x1015-0x 1005).
For example, assume that four consecutive sample values are respectively noted as: A. b, C, D, determination:
the characteristic field is A;
sampling value 1 is B;
sampling value 2 is C;
sampling value 3 is D;
the complementary description data sequences corresponding to the sampling value 1, the sampling value 2 and the sampling value 3 are as follows: B-A, C-B, D-C.
That is, by making the difference, the data width of the complementary description difference value corresponding to the sampling point is approximately compressed into one byte, which is smaller than the data width of the original sampling point.
Others are also similar, compared with the previous value
Specifically, the equal ratio data compression sub-algorithm is:
taking the characteristic field as a first sampling point of the original data;
for each sample point of the original data,
and dividing the value of the sampling point by the value of the previous sampling point to obtain a quotient value which is a complementary description difference value corresponding to the sampling point in the complementary description data sequence.
For example, the acquisition precision of each sampling point in the waveform curve is 4 bytes; the data width corresponding to the quotient of the numerical values of any two sampling points in the waveform curve is 2 bytes, and the compression ratio is approximately 2 after the equal-ratio data compression.
That is, by dividing, the data width of the complementary description difference value corresponding to the sampling point is approximately compressed into one byte, which is smaller than the data width of the original sampling point.
Specifically, the periodic data compression sub-algorithm is:
taking the characteristic field as all sampling points in the first complete voltage cycle or current cycle in the original data;
dividing other sampling points or all sampling points in the original data into a plurality of data groups to be processed by taking the number of the sampling points in the voltage cycle or the current cycle as an interval;
for each data set to be processed,
and subtracting or dividing the data set to be processed from the previous data set to be processed point by point, wherein the obtained difference value or quotient value is a complementary description difference value corresponding to the data to be processed in the complementary description data sequence.
The periodic data compression is a continuous multi-point compression method. Specifically, the characteristic field is taken as a sampling point of a complete voltage cycle or current cycle in the waveform curve (at this time, the voltage cycle or the current cycle is a periodic function other than sine or cosine, and the initial phase of the periodic function is not necessarily zero);
taking the number of sampling points in the voltage or current cycle as step length or interval, subtracting or dividing the values of other sampling points of the waveform curve and the characteristic field point by point to obtain a difference value or quotient value which is the data point equivalent to the number of sampling points of the voltage cycle or the current cycle in the supplementary description data sequence
For example, a cycle in the waveform curve has data of 16 sampling points, and the acquisition precision of each sampling point is 4 bytes, so that the width of the whole cycle is 4×16=64 bytes; taking the other sampling points of the waveform curve as a group of 16 sampling points; subtracting or dividing the values of the 16 sampling points in the groups from the characteristic field point by point; if the data width corresponding to the quotient or the difference between the values of any two sampling points in the waveform curve is 2 bytes, the width of the complementary description data sequence corresponding to the group of 16 sampling points is 2×16=32 bytes. That is, after the periodic data compression, the compression ratio is approximately 2.
The method of point-by-point subtraction or division between the data set to be processed and the previous data set to be processed is the same as the arithmetic compression or the equal ratio compression, and will not be repeated here.
Specifically, the sinusoidal data compression sub-algorithm is:
taking the characteristic field as all sampling points in the first complete sine cycle in the original data;
dividing other sampling points or all sampling points in the original data into a plurality of data groups to be processed by taking the number of all sampling points in the sine cycle as an interval;
for each data set to be processed,
and subtracting or dividing the data set to be processed from the previous data set to be processed point by point, wherein the obtained difference value or quotient value is a complementary description difference value corresponding to the data to be processed in the complementary description data sequence.
The method of point-by-point subtraction or division between the data set to be processed and the previous data set to be processed is the same as the arithmetic compression or the equal ratio compression, and will not be repeated here.
Sinusoidal data compression is a continuous multi-point compression method. Specifically, taking a characteristic field as a sampling point of a voltage cycle or a current cycle of a complete sine function in a waveform curve, or a peak value or a valley value in the sine function (at this time, the voltage or the current is sine, and the initial phase of the sine function is not necessarily zero);
taking the number of sampling points of the voltage or current cycle as an interval, subtracting or dividing the numerical value of other sampling points of the waveform curve and the characteristic field point by point, wherein the obtained difference value or quotient value is the data point equivalent to the number of the sampling points of the voltage cycle or the current cycle in the supplementary description data sequence.
For example, a cycle in the waveform curve has data of 16 sampling points, and the acquisition precision of each sampling point is 4 bytes, so that the width of the whole cycle is 4×16=64 bytes; taking the other sampling points of the waveform curve as a group of 16 sampling points; subtracting or dividing the values of the 16 sampling points in the groups from the characteristic field point by point; if the data width corresponding to the quotient or the difference between the values of any two sampling points in the waveform curve is 2 bytes, the width of the complementary description data sequence corresponding to the group of 16 sampling points is 2×16=32 bytes. That is, after the periodic data compression, the compression ratio is approximately 2.
Still further, other sampling points belonging to the same batch of original data as the feature field may be described using a sub-compression algorithm or not, and identified using a compression algorithm identification sequence, where the location is set to 1 if the current feature field description is used and to 0 if the current feature field description is not used.
On the source device side, after data compression and before starting data transmission, the following steps are adopted to assemble the data frame:
populating the data header with the feature fields;
populating the data header with the data width of the feature field;
populating the data volume with the feature sequence;
the data volume is filled with a sequence of supplemental feature data.
The feature field may be a feature field determined in real time during compression, or may be a preset feature field;
in the feature sequence, the waveform curve sampling point is described by using a current feature field, the data point at the corresponding position in the compression algorithm identification sequence is set to 1, and the data point at the corresponding position in the compression algorithm identification sequence is set to 0 without using the current feature field description.
The supplemental feature data sequence is generated by sequentially describing the subsequent waveform curve sampling points by using the feature field, and typically, the length or width of each data point in the feature sequence is smaller than the length or width of the feature field.
In addition, the processing of the acquired raw data further includes:
and determining the length of the acquired original data, and sending the length to the target equipment side, so that the target equipment performs comparison or verification when decompressing according to the received data to restore the power grid data.
At a target device side, extracting a characteristic field, a data width of the characteristic field, a compression algorithm identifier, a characteristic sequence and a supplementary characteristic data sequence from a data frame received from a transmission channel according to a stipulated protocol, and decompressing to obtain power grid data; or (b)
The feature field, the data width of the feature field, the compression algorithm identification, the feature sequence, and the supplemental feature data sequence are stored, and upon receipt of a decompression request, the decompression operation is initiated.
As shown in fig. 3, the method comprises the steps of:
searching the characteristic field and the data width of the characteristic field for curve data needing to be compressed in a traversing way;
it should be appreciated that for single point compression, the feature field is single sampled, its data width is the sampling precision of the sampling point (e.g., 16-bit precision, 2 bytes; 8-bit precision, 1 byte);
for cycle compression, the feature field is all sampling points within the entire cycle. At this time, the data width is the product of the sampling precision (e.g., 16-bit precision, 2 bytes; 8-bit precision, 1 byte) and the sampling frequency (e.g., 16 sampling points in one cycle) of the sampling point.
For any original sampled data, the data width is used to determine the step size at which the data compression is performed, i.e., the data interval at each compression operation.
For example, when the data width of a single sampling point of a voltage curve is not known to be 2 bytes in single-point compression, enumeration attempts may need to be sequentially performed on 1 byte, 2 bytes and the like, so as to find the correlation between the data, and finally determine that the data width (i.e. 2 bytes) conforming to the sampling point is most suitable as a characteristic field to perform single-point compression on subsequent data.
In case the data width of a single sampling point is known, this traversal step may be omitted, setting the data width of the feature field directly to the acquisition accuracy of the sampling point. For example, if the acquisition precision of a single sampling point is 2 bytes, for example, a voltage waveform curve is known, then the characteristic field is directly set as 2 bytes of data of the first sampling point without traversing.
According to the characteristic field, selecting corresponding compression sub-algorithms to compress the curve data respectively to obtain compressed data;
when the method is implemented, a compression algorithm library is developed. The compression algorithm library comprises a plurality of data compression sub-algorithms, such as: arithmetic data compression, geometric data compression, periodic data compression, sinusoidal compression, and the like.
When different compression sub-algorithms are respectively calculated, the ratio of the length of the original curve data to the length of the compressed data is used as the compression ratio;
and selecting a sub-compression algorithm with the maximum compression ratio as a target compression sub-algorithm of the current curve data.
For example, the phase voltage waveform data is sinusoidal; the compression ratios under each sub-compression algorithm are calculated in turn, and it can be seen that: when sinusoidal compression is employed, the compression ratio of the phase voltage waveform data is maximized.
Then, using bits in the feature sequence one by one to bit identify whether to replace the data before compression with the supplemental description difference value or to remain as the data before compression, and appending the supplemental description difference value to the end of the supplemental description data sequence;
the feature field, the data width of the feature field, the compression algorithm identification, the feature sequence and the supplementary description data sequence together form final compressed data to be transmitted.
For example, when generating a feature sequence, using the data replaced by the feature field, the corresponding bit of the feature sequence is filled with 1; non-replacement fields, corresponding bits of the signature sequence are filled with 0's. The characteristic sequence of a certain curve data is as follows: 1111001101110000, the signature sequence is expressed in hexadecimal as: 0xF370 occupies only 2 bytes of memory space or space in the data area of the data frame.
For example, when the supplemental description difference value is used to construct the supplemental description data sequence, the difference value between the feature field and the first sampling point of the original data is used as the supplemental description difference value of the first sampling point, and the incremental comparison between the feature field and the previous sampling point is performed point by point starting from the second sampling point of the original data:
if the feature field is 0x1000, the value of the first sample point immediately after it is: the value of the second sample point is 0x 1005: 0x1015; then
In the supplementary description data sequence, the corresponding supplementary description difference values may be expressed as: 0x05 (i.e., 0x1005-0x 1000), 0x10 (0 x1015-0x 1005).
At this time, the approximate compression of the data width of the complementary description difference value corresponding to the sampling point to one byte is realized by making a difference, which is smaller than the data width of the original sampling point.
In summary, the lossless compression algorithm has the following characteristics:
1) Traversing the data and searching the characteristic field;
2) The waveform data with good consistency is described by the characteristic field, so that the compression effect is better;
3) A compression algorithm with an optimal compression ratio can be selected for curve data compression;
4) In the assembled data frame, the characteristic field, the compression algorithm identifier, the characteristic sequence and the supplementary description data sequence form a payload together. The occupied memory space of the payload is adapted to the payload composed of the feature field, the compression algorithm identification, the feature sequence and the supplemental description data sequence.
The invention has been described above with reference to a few embodiments. However, as is well known to those skilled in the art, other embodiments than the above disclosed invention are equally possible within the scope of the invention, as defined by the appended patent claims.
Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise therein. All references to "a/an/the [ means, component, etc. ]" are to be interpreted openly as referring to at least one instance of said means, component, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.
Claims (9)
1. The lossless compression method for the power grid data is characterized by comprising the following steps of:
processing the acquired original data, and determining the acquisition precision and/or the sampling frequency of a single sampling point;
extracting characteristic fields corresponding to each compression sub-algorithm according to a plurality of preset compression sub-algorithms, wherein the characteristic fields are as follows: a single sampling point, or all sampling points describing one cycle;
determining a compression ratio obtained when each compression sub-algorithm processes the original data according to a characteristic field corresponding to each compression sub-algorithm and the data width of the characteristic field;
determining a target compression sub-algorithm matched with the target compression ratio according to a preset target compression ratio;
processing the original data according to the target compression sub-algorithm and the corresponding characteristic field, and generating a characteristic sequence and a supplementary description data sequence;
after generating the feature sequence and the supplemental description data sequence, further comprising:
filling the characteristic field, the data width of the characteristic field and the algorithm identification of the target compression sub-algorithm into a data head part of a data frame to be transmitted; and
Filling the feature sequence and the supplementary description data sequence into a data body part of a data frame to be transmitted;
and sending the data frame to be transmitted to target equipment.
2. The method of lossless compression of grid data according to claim 1, wherein,
the processing the original data according to the target compression sub-algorithm and the corresponding feature field, and generating a feature sequence and a supplementary description data sequence, including:
taking the data width of the characteristic field as the step length when the original data is processed;
for the raw data of each data width,
setting the corresponding current position in the characteristic sequence as a compression effective identifier when the target compression sub-algorithm is executed; and append the supplementary description difference value corresponding to the original data of the data width determined by the target compression sub-algorithm at the end of the supplementary description data sequence;
when the target compression sub-algorithm is not executed, setting the current bit corresponding to the feature sequence as a compression invalid identifier; and appends the original data at the end of the supplemental description data sequence.
3. The method of lossless compression of grid data according to claim 1, wherein,
the compression sub-algorithm is arithmetic data compression:
taking the characteristic field as a first sampling point of the original data;
for each sample point of the original data,
and the value of the sampling point is differenced from the value of the previous sampling point, and the obtained difference value is a supplementary description difference value corresponding to the sampling point in the supplementary description data sequence.
4. The method of lossless compression of grid data according to claim 1, wherein,
the compression sub-algorithm is equal-ratio data compression:
taking the characteristic field as a first sampling point of the original data;
for each sample point of the original data,
and dividing the value of the sampling point by the value of the previous sampling point to obtain a quotient value which is a complementary description difference value corresponding to the sampling point in the complementary description data sequence.
5. The method of lossless compression of grid data according to claim 1, wherein,
the compression sub-algorithm is periodic data compression:
taking the characteristic field as all sampling points in the first complete voltage cycle or current cycle in the original data;
dividing other sampling points or all sampling points in the original data into a plurality of data groups to be processed by taking the number of the sampling points in the voltage cycle or the current cycle as an interval;
for each of said sets of data to be processed,
and subtracting or dividing the data set to be processed from the previous data set to be processed point by point, wherein the obtained difference value or quotient value is a complementary description difference value corresponding to the data to be processed in the complementary description data sequence.
6. The method of lossless compression of grid data according to claim 1, wherein,
the compression sub-algorithm is sinusoidal data compression:
taking the characteristic field as all sampling points in the first complete sine cycle in the original data;
dividing other sampling points or all sampling points in the original data into a plurality of data groups to be processed by taking the number of all sampling points in the sine cycle as an interval;
for each of said sets of data to be processed,
and subtracting or dividing the data set to be processed from the previous data set to be processed point by point, wherein the obtained difference value or quotient value is a complementary description difference value corresponding to the data to be processed in the complementary description data sequence.
7. The method of lossless compression of grid data according to claim 1, wherein,
the compression ratio is: (length of feature sequence + length of supplemental description data sequence)/length of original data.
8. The method of lossless compression of grid data according to claim 1, wherein,
after sending the data frame to be transmitted to the target device, the method further comprises:
extracting a feature field, a data width of the feature field, a compression algorithm identification, a feature sequence and a supplementary feature data sequence from the received data frame,
executing a decompression sub-algorithm corresponding to the compression algorithm identification to obtain original power grid data; wherein,
and the decompression sub-algorithm corresponding to the compression algorithm identifier and the compression sub-algorithm corresponding to the compression algorithm identifier are in reciprocal operation.
9. A lossless compression device for power grid data, comprising:
a source device for performing the method of any one of claims 1 to 7;
a target device for performing the method of claim 8;
the source device and the target device are communicatively connected using a low speed transmission channel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910567393.7A CN110277998B (en) | 2019-06-27 | 2019-06-27 | Lossless compression method and device for power grid data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910567393.7A CN110277998B (en) | 2019-06-27 | 2019-06-27 | Lossless compression method and device for power grid data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110277998A CN110277998A (en) | 2019-09-24 |
| CN110277998B true CN110277998B (en) | 2024-02-27 |
Family
ID=67963576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910567393.7A Active CN110277998B (en) | 2019-06-27 | 2019-06-27 | Lossless compression method and device for power grid data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110277998B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111294054B (en) * | 2020-02-13 | 2023-05-23 | 北京天拓智领科技有限公司 | Compression method for industrial interconnection data acquisition and storage |
| CN111337133B (en) * | 2020-03-02 | 2021-08-03 | 浙江大华技术股份有限公司 | Infrared data generation method and device and infrared data analysis method and device |
| CN111665756B (en) * | 2020-06-11 | 2021-10-29 | 合肥悦芯半导体科技有限公司 | Method and device for realizing sampling of asynchronous signals based on FPGA (field programmable Gate array), electronic equipment and readable storage medium |
| CN111766471A (en) * | 2020-06-28 | 2020-10-13 | 河北旭辉电气股份有限公司 | Power waveform compression algorithm |
| EP4352626A1 (en) * | 2021-06-11 | 2024-04-17 | Huawei Technologies Co., Ltd. | Adaptive compression |
| CN113315793B (en) * | 2021-08-02 | 2021-11-23 | 华锐分布式(北京)技术有限公司 | Data transmission method, device, equipment and medium based on intelligent compression |
| CN116996076B (en) * | 2023-09-27 | 2023-12-19 | 湖北华中电力科技开发有限责任公司 | Intelligent management method for electrical energy consumption data of campus equipment |
| CN117499994B (en) * | 2023-12-29 | 2024-03-19 | 南京市计量监督检测院 | Compression and restoration method for differential data recording and transmission of GNSS (Global navigation satellite System) locator |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104320143A (en) * | 2014-10-24 | 2015-01-28 | 华北电力大学（保定） | Method and device for compressing three-phase voltage and current signal waveform sample data |
| CN106656201A (en) * | 2016-12-23 | 2017-05-10 | 燕山大学 | Compression algorithm based on amplitude-frequency characteristics of sampled data |
| CN108259515A (en) * | 2018-03-29 | 2018-07-06 | 燕山大学 | A kind of lossless source compression method suitable for transmission link under Bandwidth-Constrained |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030231799A1 (en) * | 2002-06-14 | 2003-12-18 | Craig Schmidt | Lossless data compression using constraint propagation |
| EP2663074B1 (en) * | 2012-05-11 | 2016-04-13 | XRing Technologies GmbH | Method and system for lossy compression and decompression of computed tomography data |
-
2019
- 2019-06-27 CN CN201910567393.7A patent/CN110277998B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104320143A (en) * | 2014-10-24 | 2015-01-28 | 华北电力大学（保定） | Method and device for compressing three-phase voltage and current signal waveform sample data |
| CN106656201A (en) * | 2016-12-23 | 2017-05-10 | 燕山大学 | Compression algorithm based on amplitude-frequency characteristics of sampled data |
| CN108259515A (en) * | 2018-03-29 | 2018-07-06 | 燕山大学 | A kind of lossless source compression method suitable for transmission link under Bandwidth-Constrained |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110277998A (en) | 2019-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110277998B (en) | Lossless compression method and device for power grid data | |
| CN103457609B (en) | Fault recorder data Lossless Compression, decompression method | |
| US7770091B2 (en) | Data compression for use in communication systems | |
| CN107102220A (en) | Recorder data processing method and processing device | |
| CN110058079B (en) | System for calculating electric quantity, substation comprising such a system and method for calculating electric quantity | |
| CN105530013A (en) | Waveform data compression method and system | |
| JP6021792B2 (en) | Encoder, decoder and method thereof | |
| CN103841424B (en) | The system and method for compressed data in random access memory | |
| JP2014116940A5 (en) | ||
| US20110231629A1 (en) | Data processing apparatus and data processing method | |
| WO2017131579A1 (en) | Methods, devices and systems for compressing and decompressing data | |
| CN105553937A (en) | System and method for data compression | |
| CN107565970B (en) | Hybrid lossless compression method and device based on feature recognition | |
| CN108737976A (en) | A kind of compression transmitting method based on Big Dipper short message | |
| CN115622566A (en) | High-efficiency dynamic lossless compression transmission method for real-time data of electric power | |
| CN104320143B (en) | The compression method and device of three-phase voltage, current signal waveform sampled data | |
| CN106849956A (en) | Compression method, decompression method, device and data handling system | |
| CN113312325B (en) | Track data transmission method, device, equipment and storage medium | |
| CN112449191B (en) | Method for compressing multiple images, method and device for decompressing images | |
| US8175403B1 (en) | Iterative backward reference selection with reduced entropy for image compression | |
| CN101808248A (en) | Adaptive paradigm Hoffmann decoder, method thereof and image decoder | |
| CN100401671C (en) | Method for transmitting files | |
| CN109150397B (en) | A kind of method and device constructing polarization code sequence | |
| KR101218511B1 (en) | Apparatus and method for sequencing data coding, apparatus, and system using the same | |
| US20190327343A1 (en) | Communication apparatus and communication control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |