CN117997353B - Hydraulic engineering water level data processing method - Google Patents
Hydraulic engineering water level data processing method Download PDFInfo
- Publication number
- CN117997353B CN117997353B CN202410406101.2A CN202410406101A CN117997353B CN 117997353 B CN117997353 B CN 117997353B CN 202410406101 A CN202410406101 A CN 202410406101A CN 117997353 B CN117997353 B CN 117997353B
- Authority
- CN
- China
- Prior art keywords
- water level
- level data
- data set
- compression
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 title claims abstract description 308
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 238000007906 compression Methods 0.000 claims abstract description 97
- 230000006835 compression Effects 0.000 claims abstract description 97
- 230000005856 abnormality Effects 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims description 26
- 230000002159 abnormal effect Effects 0.000 description 9
- 230000008859 change Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000013144 data compression Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention relates to the technical field of data processing, in particular to a hydraulic engineering water level data processing method, which comprises the following steps: acquiring all water level data, and classifying and processing all water level data to obtain a plurality of water level data sets; calculating the abnormality degree of each water level data set, and obtaining a compressed water level data set according to the abnormality degree; calculating the compression weight of each compression water level data set, obtaining the value range of each compression water level data set, and obtaining the adjusted value range of each compression water level data set according to the compression weight and the value range; obtaining an optimal compression value of each compression water level data set according to the adjusted value range; and performing compression storage processing according to the optimal compression value. Thereby effectively saving the storage cost.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a hydraulic engineering water level data processing method.
Background
The monitoring and management of the water level data change has important significance for reservoir operation and water resource management. Through monitoring the water level change, the change condition of reservoir water storage capacity can be known, the water storage scheme of reservoir is reasonably adjusted according to real-time requirements, and the stability and flexibility of water supply are ensured. Meanwhile, the running efficiency of the reservoir can be improved by monitoring and managing the water level change in the reservoir in real time, the safe and stable running of the reservoir is ensured, and the water demand of people is met to the greatest extent.
The existing storage method for large-scale water level data generally uses a time sequence database to store the complete water level data in a lossless manner. However, the method has the defects that a large amount of storage space is needed, the cost is high, and the flexibility of inquiry is limited, so that the scheme can be used for carrying out lossy compression on specific water level data, improves the storage efficiency under the condition that the change trend and the characteristics of the water level data are not changed, saves the storage space, and achieves efficient management on the water level data.
Disclosure of Invention
The invention provides a hydraulic engineering water level data processing method, which aims to solve the existing problems: how to ensure the change characteristics of the water level data and simultaneously reduce the storage amount of the water level data.
The hydraulic engineering water level data processing method adopts the following technical scheme:
The embodiment of the invention provides a hydraulic engineering water level data processing method, which comprises the following steps:
Acquiring all water level data, and classifying and processing all water level data to obtain a plurality of water level data sets;
calculating the abnormality degree of each water level data set, and obtaining a compressed water level data set according to the abnormality degree;
Calculating the compression weight of each compression water level data set, obtaining the value range of each compression water level data set, and obtaining the adjusted value range of each compression water level data set according to the compression weight and the value range;
obtaining an optimal compression value of each compression water level data set according to the adjusted value range;
And performing compression storage processing according to the optimal compression value.
Preferably, the classifying process of all the water level data obtains a plurality of water level data sets, including the specific method that:
and processing all water level data by using a region growing method, and forming a water level data set by all water level data belonging to one region to obtain a plurality of water level data sets.
Preferably, the calculating the abnormality degree of each water level data set includes the following specific steps:
obtaining a first parameter of the water level data set according to the mean value of all the water level data in the water level data set and the standard deviation of all the water level data in the water level data set;
The ratio of the data quantity of the water level data in the ith water level data set to the data quantity of the water level data in all water level data sets is recorded as a first ratio of the ith water level data set Will/>A second parameter noted as the i-th water level data set;
recording the ratio of the first parameter and the second parameter of the ith water level data set as the abnormality degree of the ith water level data set, wherein An exponential function based on a natural constant is represented.
Preferably, the obtaining the first parameter of the water level data set according to the mean value of all the water level data in the water level data set and the standard deviation of all the water level data in the water level data set includes the following specific steps:
The ratio of the average value of all water level data in the ith water level data set to the average value of all water level data in all water level data sets is recorded as a second ratio of the ith water level data set ；
The ratio of the standard deviation of all water level data in the ith water level data set to the average value of all water level data in the ith water level data set is recorded as a third ratio of the ith water level data set；
Will beThe first parameter, denoted as the ith water level data set, wherein/>Representing the acquisition of absolute values.
Preferably, the method for obtaining the compressed water level data set according to the abnormality degree includes the following specific steps:
and taking the water level data set with the abnormality degree smaller than the preset abnormality degree threshold Y1 as the compressed water level data set.
Preferably, the calculating the compression weight of each compression water level data set includes the following specific steps:
Wherein, Compression weight representing the ith compressed water level data set,/>Representing the minimum degree of abnormality for all compressed water level data sets,/>Maximum value of abnormality degree representing all compression water level data sets,/>Representing the difference between the maximum water level data and the minimum water level data in the ith compressed water level data set,/>Indicating the degree of abnormality of the ith water level data set.
Preferably, the acquiring the value range of each compressed water level data set includes the following specific steps:
And acquiring the maximum water level data and the minimum water level data in each compressed water level data set, and forming the minimum water level data and the maximum water level data into a value range of each compressed water level data set.
Preferably, the obtaining the adjusted value range of each compressed water level data set according to the compression weight and the value range includes the following specific methods:
Wherein, Representing the upper limit value of the value range of the ith compressed water level data set,/>Representing the lower limit value of the value range of the ith compressed water level data set,/>Representing the upper limit value of the value range after adjustment of the ith compressed water level data set,/>Representing the lower limit value of the value range after adjustment of the ith compressed water level data set,/>Representing a downward rounding symbol,/>Representing the compression weight of the ith compressed water level data set.
Preferably, the method for obtaining the optimal compression value of each compression water level data set according to the adjusted value range includes the following specific steps:
For each compressed water level data set, marking any integer in the adjusted value range as a target value, taking the absolute value of the difference value between each water level data in the compressed water level data set and the target value as the first difference value between each water level data and the target value, taking the ratio of the sum of the first differences of all water level data and the target value and the variance of all water level data in the compressed water level data set as the loss value of the target value, and acquiring the integer with the minimum loss value from all integers in the adjusted value range as the optimal compression value of each compressed water level data set.
Preferably, the compressing and storing according to the optimal compression value comprises the following specific steps:
each compressed digital data set only has the best compressed value, and for other water level data sets, each water level data in the water level data set needs to be stored.
The technical scheme of the invention has the beneficial effects that: acquiring all water level data, and classifying and processing all water level data to obtain a plurality of water level data sets; calculating the abnormality degree of each water level data set, and obtaining a compressed water level data set according to the abnormality degree; the normal water level data exists in the compressed water level data set, so that only the normal water level data is subjected to lossy compression, the abnormal water level data is reserved, and the information loss of the abnormal water level data caused by data compression is prevented from affecting the subsequent water level analysis. Calculating the compression weight of each compression water level data set, obtaining the value range of each compression water level data set, and obtaining the adjusted value range of each compression water level data set according to the compression weight and the value range; obtaining an optimal compression value of each compression water level data set according to the adjusted value range; the optimal compression value is a compression value with the smallest loss, and the compression loss can be reduced by compressing the water level data with this value.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of steps of a hydraulic engineering water level data processing method of the present invention.
Detailed Description
In order to further describe the technical means and effects adopted by the present invention to achieve the preset purpose, the following detailed description refers to the specific implementation, structure, characteristics and effects of a hydraulic engineering water level data processing method according to the present invention with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" means that the embodiments are not necessarily the same. Furthermore, the particular features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The invention provides a concrete scheme of a hydraulic engineering water level data processing method, which is specifically described below with reference to the accompanying drawings.
Referring to fig. 1, a flowchart of steps of a hydraulic engineering water level data processing method according to an embodiment of the invention is shown, and the method includes the following steps:
step S001: and acquiring a water level data area.
In order to prevent natural disasters caused by abnormal water levels, the water levels in the reservoirs need to be detected in real time, and a large amount of water level data is collected in the real-time detection process, and has an important effect on aspects such as natural disasters research, so that the water level data collected in real time need to be stored, a large storage space is needed to store the water level data collected in real time, and in order to save the storage space, the collected water level data needs to be compressed.
Specifically, in order to implement the hydraulic engineering water level data processing method provided in this embodiment, all water level data of the reservoir needs to be acquired first. And processing all water level data by using a region growing method, and forming a water level data set by all water level data belonging to one region to obtain a plurality of water level data sets.
So far, a plurality of water level data sets are obtained through the method.
Step S002: and calculating the abnormality degree of each water level data set, and obtaining the compressed water level data set according to the abnormality degree.
When the fluctuation of the water level data is large, the water level data is highly likely to be abnormal, and the abnormal water level data generally plays an important role in natural disaster research and the like, so that the abnormal water level data should not be compressed, and the characteristic loss of the abnormal data caused by compression is prevented. Therefore, the data set needing to be compressed needs to be screened out according to the fluctuation condition of the data.
Specifically, the method for obtaining the abnormality degree of each water level data set comprises the following steps:
Wherein, Representing the degree of abnormality of the ith water level data set,/>Representing standard deviation of all water level data in the ith water level data set,/>Representing the mean value of all water level data in the ith water level data set,/>Representing the mean value of all water level data in all water level data sets,/>Representing the data amount of the water level data in the ith water level data set,/>Representing the data volume of the water level data in all water level data sets.
The deviation degree of the average value of the water level data in the ith water level data set from the average value of all the water level data is reflected, and the larger the deviation degree is the larger the water level data in the ith water level data set deviates from the whole water level data, namely, the water level data in the water level data set is more likely to be abnormal water level data. /(I)The variation coefficient of the ith water level data set is reflected and is used for measuring the fluctuation amplitude of the data, the larger the value of the variation coefficient is, the larger the fluctuation amplitude of the data in the water level data set is, the more discrete the data is, and the more the water level data in the water level data set is likely to be abnormal water level data. /(I)Reflecting the ratio of the data volume in the ith water level data set to the overall data volume, wherein the larger the data volume in the set is, the larger the ratio is, which indicates that the water level data in the water level data set is more likely to be normal water level data,/>An exponential function based on a natural constant is represented.
Further, the water level data set with the abnormality degree smaller than the preset abnormality degree threshold Y1 is taken as the compressed water level data set, in this embodiment, 0.6 is taken as an example for Y1, other embodiments may take other values, and the embodiment is not particularly limited.
Step S003: and calculating the compression weight of each compression water level data set, adjusting the value range of each compression water level data set according to the compression weight to obtain an adjusted value range of each compression water level data set, and obtaining the optimal compression value of each compression water level data set according to the adjusted value range.
The compressed water level data set is obtained in the above process, and the water level data in the compressed water level data set is required to be compressed. The difference between the water level data of each water level data set obtained by the region growing method is small, and the fluctuation amount of the water level data in the compressed water level data set screened out by the process is small. Therefore, the water level data of the compression water level data set at the fluctuation center has small difference from other water level data, so that the water level data of the compression water level data set at the fluctuation center can be selected as the compressed data of each compression water level data set, and data compression is realized.
Specifically, the method for obtaining the compression weight of each compression water level data set according to the abnormality degree of each compression digital data set comprises the following steps:
Wherein, Compression weight representing the ith compressed water level data set,/>Representing the minimum degree of abnormality for all compressed water level data sets,/>Maximum value of abnormality degree representing all compression water level data sets,/>Representing the difference between the maximum water level data and the minimum water level data in the ith compressed water level data set,/>The influence factor reflecting the degree of abnormality of the ith compression water level data set, the larger the value is, which indicates that the data in the compression water level data set is more discrete, and the larger the compression weight for adjusting the value range of the compression water level data set is. /(I)The influence factor reflecting the fluctuation degree of the ith compression water level data set is larger, and the larger the value is, the larger the data extreme value in the compression water level data set is, and the larger the compression weight for adjusting the value range of the compression water level data set is.
Further, the maximum water level data and the minimum water level data in each compressed water level data set are obtained, and the minimum water level data and the maximum water level data form a value range of each compressed water level data set.
The method for acquiring the adjusted value range of each compressed water level data set comprises the following steps:
Wherein, The upper limit value of the value range of the ith compression water level data set is also the minimum water level data of the ith compression water level data set,/>The lower limit value of the value range of the ith compression water level data set is also the maximum water level data of the ith compression water level data set,/>Representing the upper limit value of the value range after adjustment of the ith compressed water level data set,/>Representing the lower limit value of the value range after adjustment of the ith compressed water level data set,/>Representing the adjusted value range of the ith compressed water level data set,/>Representing rounding down symbols. The higher the weight is, the smaller the value lower limit lifting degree is, and the higher the weight is, the smaller the value upper limit dropping degree is. The larger the weight is, the smaller the reduction degree of the compression value range is, and the larger the contrary is. Thus, the aim of obtaining a unique compression value range for a certain compression area is achieved by the method.
Further, the method for obtaining the optimal compression value of each compression water level data set comprises the following steps:
And for each compressed water level data set, marking any integer in the adjusted value range as a target value, taking the absolute value of the difference value between each water level data in the compressed water level data set and the target value as a first difference value between each water level data and the target value, and taking the ratio of the sum of the accumulated first difference values of all water level data and the target value and the variance of all water level data in the compressed water level data set as a loss value of the target value. And obtaining the integer with the minimum loss value from all integers in the adjusted value range as the optimal compression value of each compression water level data set.
Step S004: and performing compression storage according to the optimal compression value of each compression water level data set.
Specifically, each compressed digital data set only has the best compression value, and for other water level data sets, each water level data in the village water level data set is required.
This embodiment is completed.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the invention, but any modifications, equivalent substitutions, improvements, etc. within the principles of the present invention should be included in the scope of the present invention.
Claims (4)
1. A hydraulic engineering water level data processing method is characterized by comprising the following steps:
Acquiring all water level data, and classifying and processing all water level data to obtain a plurality of water level data sets;
calculating the abnormality degree of each water level data set, and obtaining a compressed water level data set according to the abnormality degree;
the method for calculating the abnormality degree of each water level data set comprises the following specific steps:
Wherein, Representing the degree of abnormality of the ith water level data set,/>Representing standard deviation of all water level data in the ith water level data set,/>Representing the mean value of all water level data in the ith water level data set,/>Representing the mean value of all water level data in all water level data sets,/>Representing the data amount of the water level data in the ith water level data set,/>Representing the data amount of the water level data in all water level data sets;
the compressed water level data set is obtained according to the degree of abnormality, and the method comprises the following specific steps:
Taking a water level data set with the abnormality degree smaller than a preset abnormality degree threshold Y1 as a compressed water level data set;
Calculating the compression weight of each compression water level data set, obtaining the value range of each compression water level data set, and obtaining the adjusted value range of each compression water level data set according to the compression weight and the value range;
The specific method for calculating the compression weight of each compression water level data set comprises the following steps:
Wherein, Compression weight representing the ith compressed water level data set,/>Representing the minimum degree of abnormality for all compressed water level data sets,/>Maximum value of abnormality degree representing all compression water level data sets,/>Representing the difference between the maximum water level data and the minimum water level data in the ith compressed water level data set;
The method for obtaining the adjusted value range of each compressed water level data set according to the compression weight and the value range comprises the following specific steps:
Wherein, Representing the upper limit value of the value range of the ith compressed water level data set,/>Representing the lower limit value of the value range of the ith compressed water level data set,/>Representing the upper limit value of the value range after adjustment of the ith compressed water level data set,/>Representing the lower limit value of the value range after adjustment of the ith compressed water level data set,/>Representing a downward rounding symbol;
obtaining an optimal compression value of each compression water level data set according to the adjusted value range;
The method for obtaining the optimal compression value of each compression water level data set according to the adjusted value range comprises the following specific steps:
For each compressed water level data set, marking any integer in the adjusted value range as a target value, taking the absolute value of the difference value between each water level data in the compressed water level data set and the target value as a first difference value between each water level data and the target value, taking the ratio of the sum of the first difference values of all water level data and the target value to the variance of all water level data in the compressed water level data set as a loss value of the target value, and acquiring the integer with the minimum loss value from all integers in the adjusted value range as the optimal compression value of each compressed water level data set;
And performing compression storage processing according to the optimal compression value.
2. The hydraulic engineering water level data processing method according to claim 1, wherein the classifying processing of all water level data to obtain a plurality of water level data sets comprises the following specific steps:
and processing all water level data by using a region growing method, and forming a water level data set by all water level data belonging to one region to obtain a plurality of water level data sets.
3. The hydraulic engineering water level data processing method according to claim 1, wherein the obtaining the value range of each compressed water level data set comprises the following specific steps:
And acquiring the maximum water level data and the minimum water level data in each compressed water level data set, and forming the minimum water level data and the maximum water level data into a value range of each compressed water level data set.
4. The hydraulic engineering water level data processing method according to claim 1, wherein the compressing and storing according to the optimal compression value comprises the following specific steps:
each compressed digital data set only has the best compressed value, and for other water level data sets, each water level data in the water level data set needs to be stored.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410406101.2A CN117997353B (en) | 2024-04-07 | 2024-04-07 | Hydraulic engineering water level data processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410406101.2A CN117997353B (en) | 2024-04-07 | 2024-04-07 | Hydraulic engineering water level data processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117997353A CN117997353A (en) | 2024-05-07 |
| CN117997353B true CN117997353B (en) | 2024-06-18 |
Family
ID=90887727
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410406101.2A Active CN117997353B (en) | 2024-04-07 | 2024-04-07 | Hydraulic engineering water level data processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117997353B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115659070A (en) * | 2022-12-28 | 2023-01-31 | 鸿基骏业环保科技有限公司 | Water flow data transmission method based on NB-IOT intelligent water meter |
| CN117708459A (en) * | 2023-11-07 | 2024-03-15 | 福建省九龙江流域中心 | Water conservancy multivariable time sequence data loading processing optimization method and terminal |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117743870B (en) * | 2024-02-20 | 2024-05-10 | 山东齐鸿工程建设有限公司 | Water conservancy data management system based on big data |
-
2024
- 2024-04-07 CN CN202410406101.2A patent/CN117997353B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115659070A (en) * | 2022-12-28 | 2023-01-31 | 鸿基骏业环保科技有限公司 | Water flow data transmission method based on NB-IOT intelligent water meter |
| CN117708459A (en) * | 2023-11-07 | 2024-03-15 | 福建省九龙江流域中心 | Water conservancy multivariable time sequence data loading processing optimization method and terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117997353A (en) | 2024-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111241056B (en) | Power energy data storage optimization method based on decision tree model | |
| CN117278054B (en) | Intelligent power grid monitoring data storage method and system | |
| CN116760908B (en) | Agricultural information optimization management method and system based on digital twin | |
| CN116153452A (en) | Medical electronic medical record storage system based on artificial intelligence | |
| CN117764290B (en) | BIM model data management method for hydraulic engineering construction | |
| CN115802056A (en) | User data compression storage method for mobile terminal | |
| CN117608499B (en) | Intelligent traffic data optimal storage method based on Internet of things | |
| CN117997353B (en) | Hydraulic engineering water level data processing method | |
| CN115658628A (en) | Production data intelligent management method for MES system | |
| CN117290364A (en) | Intelligent market investigation data storage method | |
| CN114880318A (en) | Method and system for realizing automatic data management based on data standard | |
| CN112766590B (en) | Method and system for extracting typical residential power consumption pattern | |
| CN116627920A (en) | Data storage method based on industrial Internet | |
| CN116502802A (en) | Data management system based on big data and wireless sensing technology | |
| CN115544305A (en) | Data storage method and device for digital steel coil system | |
| CN114205393A (en) | Data reporting method and system of intelligent home system | |
| CN117767960B (en) | Sensor data optimization acquisition and storage method | |
| CN117155402B (en) | Public health intelligent physical examination service system based on RPA technology | |
| CN117708513B (en) | Planting data management method for valerian characteristic research | |
| CN118018882B (en) | Internet of things data acquisition and storage method | |
| CN117150224B (en) | User behavior data storage analysis method based on Internet of things | |
| CN117294314B (en) | Fruit and vegetable can production information data record management method | |
| CN117992808B (en) | Big data-based computer network management method and system | |
| CN117648647B (en) | Multi-energy power distribution network user data optimization classification method | |
| CN115563193B (en) | Big data analysis processing method for digital information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |