CN113111056A - Cleaning method for urban flood water monitoring data - Google Patents

Cleaning method for urban flood water monitoring data Download PDF

Info

Publication number
CN113111056A
CN113111056A CN202110500063.3A CN202110500063A CN113111056A CN 113111056 A CN113111056 A CN 113111056A CN 202110500063 A CN202110500063 A CN 202110500063A CN 113111056 A CN113111056 A CN 113111056A
Authority
CN
China
Prior art keywords
data
alpha
change
value
abnormal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110500063.3A
Other languages
Chinese (zh)
Other versions
CN113111056B (en
Inventor
刘家宏
梅超
燕文昌
王浩
杨志勇
邵薇薇
丁相毅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Institute of Water Resources and Hydropower Research
Original Assignee
China Institute of Water Resources and Hydropower Research
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Institute of Water Resources and Hydropower Research filed Critical China Institute of Water Resources and Hydropower Research
Priority to CN202110500063.3A priority Critical patent/CN113111056B/en
Publication of CN113111056A publication Critical patent/CN113111056A/en
Application granted granted Critical
Publication of CN113111056B publication Critical patent/CN113111056B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2474—Sequence data queries, e.g. querying versioned data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/248—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06395—Quality analysis or management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/26—Government or public services
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A10/00—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE at coastal zones; at river basins
    • Y02A10/40—Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • Educational Administration (AREA)
  • Development Economics (AREA)
  • Data Mining & Analysis (AREA)
  • Economics (AREA)
  • Quality & Reliability (AREA)
  • Computational Linguistics (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Entrepreneurship & Innovation (AREA)
  • General Health & Medical Sciences (AREA)
  • Operations Research (AREA)
  • Game Theory and Decision Science (AREA)
  • Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • Fuzzy Systems (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Software Systems (AREA)
  • Testing And Monitoring For Control Systems (AREA)
  • Testing Or Calibration Of Command Recording Devices (AREA)
  • Alarm Systems (AREA)

Abstract

The invention discloses a cleaning method for urban flood water monitoring data, S1, carrying out data interpolation on a missing value of the urban flood water monitoring data by adopting an interpolation algorithm; s2, screening abnormal data points based on a difference discrimination algorithm; s3, judging the change of the first peak value or the first valley value of the data to be abnormal fluctuation according to the screened abnormal data points; s4, calculating the fluctuation change frequency of the data subsequent to the first peak value or the first valley value change in the total execution time, and when the change frequency is greater than the empirical value of the minimum frequency of continuous fluctuation of the accumulated water, judging that the data fluctuation in the total execution time has continuity; and S5, correcting the abnormal data in S2, S3 and S4 by adopting a moving average algorithm. The method disclosed by the invention is based on a cleaning method framework of secondary diagnosis, so that abnormal data can be screened out to the maximum extent, and error screening can be avoided, and therefore, the urban accumulated water monitoring data can be effectively cleaned.

Description

Cleaning method for urban flood water monitoring data
Technical Field
The invention belongs to the technical field of cleaning and quality control of urban flood water monitoring data, and particularly relates to a cleaning method of urban flood water monitoring data.
Background
Urban flood disasters become main natural disasters affecting urban development in China, and the phenomenon of 'urban sea watching' in the annual flood season is repeated frequently, so that urban operation and resident production and life are seriously affected. Because China belongs to continental monsoon climate, the distribution of rainfall time is extremely uneven, and at present, China is in a period of novel urbanization rapid promotion, and it is foreseeable that urban flood disasters will exist for a long time in a future period.
In urban flood control, the monitoring of accumulated water is an important basis for forecasting, early warning, planning and designing and scientific research, and a monitoring system covering main accumulated water points of cities is built in many cities. Along with the accumulation of data, a large amount of data is found to have abnormal phenomena, particularly, the problem of ponding data oscillation caused by vehicle running is solved, and the data quality is difficult to effectively support the prevention and control of urban flood disasters.
Disclosure of Invention
The invention aims to provide a cleaning method for urban flood waterlogging monitoring data aiming at the defects in the prior art, so as to solve the problem of waterlogging data oscillation caused by vehicle running in the existing waterlogging monitoring.
In order to achieve the purpose, the invention adopts the technical scheme that:
a cleaning method for urban flood monitoring data comprises the following steps:
s1, performing data interpolation on the default value of the urban flood water monitoring data by adopting an interpolation algorithm;
s2, screening abnormal data points based on a difference discrimination algorithm;
s3, judging the change of the first peak value or the first valley value of the data to be abnormal fluctuation according to the screened abnormal data points;
s4, calculating the fluctuation change frequency of the data subsequent to the first peak value or the first valley value change in the total execution time, and when the change frequency is greater than the empirical value of the minimum frequency of continuous fluctuation of the accumulated water, judging that the data fluctuation in the total execution time has continuity;
and S5, correcting the abnormal data in S2, S3 and S4 by adopting a moving average algorithm.
Further, in S1, performing data interpolation on the missing value of the urban flood monitoring data by using an interpolation algorithm, including:
Figure BDA0003056127630000021
wherein alpha isqiFor missing data, i is 1,2,3 … … is the time sequence of the missing data, αmFor the last measured data before the missing measurement, alphanFor the first measured data after lack of measurement, delta t is alphamAnd alphanTime interval of (1), Δ t0Is a monitoring period.
Further, the screening of abnormal data points based on the difference discrimination algorithm in S2 includes:
calculate two adjacent data points alphaiAnd alphai+1The difference of (a):
Δα＝αi+1-αi
when the | Δ α | < β, the β is a threshold, and the depth change of the data point is a normal condition; when | Δ α | ≧ β, then the latter data point is an anomalous change.
Further, the step of determining, in S3, that the change of the first peak or the first valley of the data is an abnormal fluctuation according to the screened abnormal data point includes:
calculating the change rate omega of two adjacent data pointsi：
Figure BDA0003056127630000022
Sequentially calculating the change rate of two adjacent points when omega appears for the first timekWith a first rate of change omegajWhen the signs of the two phases are opposite, a first peak or valley alpha is generatedk-1；
Calculating the data point from alphaj-1To alphak-1Average rate of change of
Figure BDA0003056127630000023
Figure BDA0003056127630000024
When the oxygen deficiency is reached
Figure BDA0003056127630000031
When | | | is greater than or equal to the threshold value gamma, the alpha change rate is abnormal;
after the peak value or the valley value is generated and the positive and negative signs of omega are changed again or | omega | is less than or equal to gamma, judging that the value is alphajTo alphalIs a peak or a trough of a data change, and calculates alphajTo alphalAt intervals of time Δ tω：
Δtω＝tl-tj
Wherein, tlAnd tjAre respectively data alphalAnd alphajThe corresponding time.
Further, the frequency of fluctuation change of the first peak or first valley change succeeding data in the total execution time is calculated in S4
Figure BDA0003056127630000032
Figure BDA0003056127630000033
Wherein s is alphal+1To
Figure BDA0003056127630000034
The number of all peaks or valleys generated in between,
Figure BDA0003056127630000035
to be the total time of execution.
Further, in S5, the correcting the abnormal data in S2, S3 and S4 by using a moving average algorithm includes:
Figure BDA0003056127630000036
wherein, alpha'iIs alphaiCorrection value of alphazAs the first data after an abnormal data sequence or a single abnormal value, aj-1As abnormal data alphajThe last data before.
The cleaning method for the urban flood water monitoring data provided by the invention has the following beneficial effects:
the invention adopts a method of 'progressive and graded screening layer by layer' to diagnose whether accumulated water data are 'error report', 'lack of measurement' and 'oscillation', the normal data are directly put in storage after diagnosis, and abnormal data are put in storage after processing such as interpolation, correction, flattening and the like, so that the data which are put in storage and displayed have higher quality, and the urban flood control work is effectively supported.
Drawings
Fig. 1 is a flow chart of a method for cleaning urban flood monitoring data.
FIG. 2 is a measured water depth change process of the urban flood water monitoring data cleaning method.
FIG. 3 is a corrected water depth map of the urban flood water monitoring data cleaning method.
Detailed Description
The following description of the embodiments of the present invention is provided to facilitate the understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and it will be apparent to those skilled in the art that various changes may be made without departing from the spirit and scope of the invention as defined and defined in the appended claims, and all matters produced by the invention using the inventive concept are protected.
According to an embodiment of the application, referring to fig. 1, the method for cleaning urban flood monitoring data comprises the following steps:
s1, performing data interpolation on the default value of the urban flood water monitoring data by adopting an interpolation algorithm;
s2, screening abnormal data points based on a difference discrimination algorithm;
s3, judging the change of the first peak (valley) value of the data into abnormal fluctuation according to the screened abnormal data points;
s4, calculating the fluctuation frequency of the first peak (valley) value change follow-up data in the total execution time, and judging that the data fluctuation in the total execution time has continuity when the change frequency is greater than the empirical value of the minimum frequency of the continuous fluctuation of the accumulated water;
and S5, correcting the abnormal data in S2, S3 and S4 by adopting a moving average algorithm.
The method is based on a cleaning method framework of secondary diagnosis, abnormal data are screened out to the maximum extent, error screening is avoided, the first diagnosis is supplement to 'an undetected value', and the second diagnosis is recognition and correction of 'an abnormal value'.
The above steps will be described in detail below according to one embodiment of the present application.
S1, performing data interpolation on the default value of the urban flood water monitoring data by adopting an interpolation algorithm, namely, diagnosing for the first time, wherein the method specifically comprises the following steps:
firstly, the integrity of data is ensured, data loss may be caused due to equipment failure and the like, in order to solve the problem, the data is interpolated, wherein a 'default value' refers to a default value which influences the continuity of the data in a short time, and the data which is continuously detected for a long time is not interpolated.
Setting a data format:
{[αi，ti]}
wherein, α is the ponding depth data, t is the data monitoring time, and i is 1,2,3 … … is the time sequence of data.
Checking the continuity of the data:
Δt＝ti+1-ti
wherein, Δ t is the time interval of 2 adjacent data, when Δ t is more than or equal to 2 Δ t0When, Δ t0For a monitoring period, the data has an absence value.
Let T beqTo allow maximum missing length of interpolated data, when Δ T ≦ TqThen, the data is interpolated:
Figure BDA0003056127630000051
wherein alpha isqiFor missing data, i is 1,2,3 … … is the time sequence of the missing data, αmFor the last measured data before the missing measurement, alphanThe first measured data after the lack of measurement.
Step S1 is a single missing measurement data interpolation method, or multiple missing measurements of the data series may be interpolated in time sequence.
The surface water in urban areas is generated by rainfall, and after the rainfall stops or is reduced, the water evaporates, seeps or enters a drainage pipeline through self-flowing, and the change of the depth of the water during the period belongs to a normal condition. However, urban road surface traffic is busy, and the ponding monitoring equipment is usually set up at the roadside, and is influenced by the car driving greatly, mainly reflects as the unusual fluctuation of monitoring data, compares the normal change of ponding degree of depth, and its characteristics mainly have the fluctuation to produce and have the abruptness, and the range is great, the fluctuation frequency is higher, and the fluctuation weakens fast, probably produces continuous fluctuation according to road surface traffic flow.
Therefore, the second diagnosis is adopted for identifying and correcting the abnormal value, and the method specifically comprises the following steps:
s2, screening abnormal data points based on a difference discrimination algorithm;
first, a preliminary judgment is made by a difference discrimination algorithm, which is mainly directed to the outliers of a single data point. By setting a threshold range, calculating and comparing whether the difference value of two adjacent data points falls within the threshold range, screening out data points with obvious abnormality:
Δα＝αi+1-αi
when the | Δ α | < β, the β is a threshold, and the depth change of the data point is a normal condition; when | Δ α | ≧ β, the latter data point is an abnormal change, and the next step of analysis needs to be entered.
S3, judging abnormal fluctuation of the data by adopting a single peak (valley) algorithm;
based on the discrimination in step S2, a data point α of a suspected abnormal change is found in the data sequencejThen from alphaj-1The calculation of the rate of change omega of two adjacent data points is startedi：
Figure BDA0003056127630000061
By calculating the change rate of two adjacent points in sequence, for omegaiWhen ω appears for the first timekWith a first rate of change omegajWhen the signs of (a) are opposite, i.e. when ω isjOmega at > 0k< 0 (or ω)jOmega at < 0k> 0), the first peak (valley) value α is considered to have occurredk-1。
Calculating the data point from alphaj-1To alphak-1Average rate of change of
Figure BDA0003056127630000062
Figure BDA0003056127630000063
Let gamma be the threshold value of average change rate when gamma is not zero
Figure BDA0003056127630000064
When | ≧ threshold gamma, the rate of change of alpha is abnormal, and the peak (valley) change duration needs to be distinguished.
After the peak (valley) value is generated, the sign change of ω or | ω ≦ γ occurs again, assuming that this time the data is αlA rate of change of ωlThen, consider from αjTo alphalCalculating alpha for the peak (valley) of a data changejTo alphalThe interval time of (c):
Δtω＝tl-tj
setting an empirical value TωWhen Δ t isω＜TωThen, the secondary peak (valley) change is judged to be abnormal fluctuation, and correction processing is needed.
S4, judging the continuity of abnormal fluctuation of the data by adopting a multi-peak (valley) value algorithm;
the abnormal fluctuation of the accumulated water is usually a fluctuation which is gradually reduced in amplitude for a plurality of times after a large fluctuation is detected, and after the algorithm of the step S3 checks and finds the first abnormal fluctuation, the continuity of the plurality of fluctuations needs to be analyzed to judge whether the fluctuation is the remaining wave of the same abnormal fluctuation, so as to facilitate the subsequent correction processing.
For data subsequent to the first peak (valley) change, i.e. alphal+1The subsequent data continues to execute the algorithm of the step S3, and assuming that the total execution time is
Figure BDA0003056127630000071
Figure BDA0003056127630000072
Wherein the content of the first and second substances,
Figure BDA0003056127630000073
is composed of
Figure BDA0003056127630000074
Number of data points detected within time, pairFrom alphal+1To
Figure BDA0003056127630000075
Counting the number s of all peak values or valley values generated in the process, and calculating the frequency f of fluctuation change:
Figure BDA0003056127630000076
according to the empirical value f of the minimum frequency of the continuous fluctuation of the accumulated waterminWhen is coming into contact with
Figure BDA0003056127630000079
When it is, then it is considered that
Figure BDA0003056127630000077
The fluctuation in time is continuous, and the continuous fluctuation is subjected to correction processing.
S5, correcting abnormal data in S2, S3 and S4 by adopting a moving average algorithm;
correcting abnormal data by adopting a moving average algorithm:
Figure BDA0003056127630000078
wherein, alpha'iIs alphaiCorrection value of alphazAs the first data after an abnormal data sequence or a single abnormal value, aj-1As abnormal data alphajThe last data before.
The conditions for executing step S5 are:
if the step S2 verifies that no suspected abnormal value exists, the step S5 correction algorithm is not started;
if the step S2 tests that the suspected abnormal value exists and the step S3 tests that the fluctuation does not exist and is only a single abnormal value, the abnormal value is corrected by adopting the step S5 after the test of the step S2;
if the step S2 and the step S3 are executed, the step S4 calculates and checks that no continuous movement exists, and the abnormal data checked in the step S3 is returned and corrected by adopting the step S5 sequence;
after the steps S2, S3, and S4 are executed once, the abnormal data sequence after the inspection is corrected by using the step S5.
According to an embodiment of the present application, a specific example is described below.
After a city without waterlogging and a smart water project are built in a certain city, more than 120 urban waterlogging monitoring points are built together, and the urban waterlogging monitoring points are important infrastructure for urban waterlogging monitoring. Under normal conditions, each ponding point can upload the monitored ponding water depth data in real time, the data volume is huge, manual check is difficult to realize, and correction can be carried out through the cleaning algorithm provided by the invention. The reasonability and the effectiveness of the content of the invention are illustrated by taking partial data of a flood monitoring point in a certain city during rainfall in 8/18/2020 as an implementation case and comparing the data before and after the processing of a cleaning algorithm.
TABLE 1 actually measured Water depth data
Figure BDA0003056127630000081
The actual measurement data is substituted into the algorithm for cleaning, firstly, the missing value is interpolated through the step S1, and the length of the missing time accords with the interpolation requirement through the inspection, so that the interpolation can be carried out. Then, the abnormal value is checked and identified through a third progressive algorithm of the second diagnosis, namely, step S2-step S4, and finally, the abnormal value is corrected through a correction algorithm step S5.
TABLE 2 corrected water depth data
Figure BDA0003056127630000091
Referring to fig. 2 and 3, by comparing data sequences before and after data cleaning, it can be known that the data cleaning method of the present invention is reasonable and effective, and can effectively solve the problem of ponding data oscillation caused by vehicle driving in the existing ponding monitoring.
While the embodiments of the invention have been described in detail in connection with the accompanying drawings, it is not intended to limit the scope of the invention. Various modifications and changes may be made by those skilled in the art without inventive step within the scope of the appended claims.

Claims (6)

1. A cleaning method for urban flood water monitoring data is characterized by comprising the following steps:
s1, performing data interpolation on the default value of the urban flood water monitoring data by adopting an interpolation algorithm;
s2, screening abnormal data points based on a difference discrimination algorithm;
s3, judging the change of the first peak value or the first valley value of the data to be abnormal fluctuation according to the screened abnormal data points;
s4, calculating the fluctuation change frequency of the data subsequent to the first peak value or the first valley value change in the total execution time, and when the change frequency is greater than the empirical value of the minimum frequency of continuous fluctuation of the accumulated water, judging that the data fluctuation in the total execution time has continuity;
and S5, correcting the abnormal data in S2, S3 and S4 by adopting a moving average algorithm.
2. The method for cleaning urban flood monitoring data according to claim 1, wherein in S1, the data interpolation of the missing value of the urban flood monitoring data by using an interpolation algorithm comprises:
Figure FDA0003056127620000011
wherein alpha isqiFor missing data, i is 1,2,3 … … is the time sequence of the missing data, αmFor the last measured data before the missing measurement, alphanFor the first measured data after lack of measurement, delta t is alphamAnd alphanTime interval of (1), Δ t0Is a monitoring period.
3. The method for cleaning urban flood monitoring data according to claim 2, wherein the step of screening abnormal data points based on a difference discrimination algorithm in step S2 comprises:
calculate two adjacent data points alphaiAnd alphai+1The difference of (a):
Δα＝αi+1-αi
when the | Δ α | < β, the β is a threshold, and the depth change of the data point is a normal condition; when | Δ α | ≧ β, then the latter data point is an anomalous change.
4. The method for cleaning urban flood monitoring data according to claim 3, wherein the step of judging the change of the first peak value or the first valley value of the data as abnormal fluctuation according to the screened abnormal data points in the step S3 comprises:
calculating the change rate omega of two adjacent data pointsi：
Figure FDA0003056127620000021
Sequentially calculating the change rate of two adjacent points when omega appears for the first timekWith a first rate of change omegajWhen the signs of the two phases are opposite, a first peak or valley alpha is generatedk-1；
Calculating the data point from alphaj-1To alphak-1Average rate of change of
Figure FDA0003056127620000022
Figure FDA0003056127620000023
When in use
Figure FDA0003056127620000029
Then the rate of change of alphaAn anomaly;
after the peak value or the valley value is generated and the positive and negative signs of omega are changed again or | omega | is less than or equal to gamma, judging that the value is alphajTo alphalIs a peak or a trough of a data change, and calculates alphajTo alphalAt intervals of time Δ tω：
Δtω＝tl-tj
Wherein, tlAnd tjAre respectively data alphalAnd alphajThe corresponding time.
5. The method of claim 4, wherein the frequency of fluctuation changes of the data subsequent to the first peak or the first valley change in the total execution time is calculated in S4
Figure FDA0003056127620000024
Figure FDA0003056127620000025
Wherein s is alphal+1To
Figure FDA0003056127620000026
The number of all peaks or valleys generated in between,
Figure FDA0003056127620000027
to be the total time of execution.
6. The method for cleaning urban flood monitoring data according to claim 5, wherein the step of correcting abnormal data in the steps S2, S3 and S4 by using a moving average algorithm in the step S5 comprises the steps of:
Figure FDA0003056127620000028
wherein, a'iIs alphaiCorrection value of alphazAs the first data after an abnormal data sequence or a single abnormal value, aj-1As abnormal data alphajThe last data before.
CN202110500063.3A 2021-05-08 2021-05-08 Cleaning method for urban flood water monitoring data Active CN113111056B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110500063.3A CN113111056B (en) 2021-05-08 2021-05-08 Cleaning method for urban flood water monitoring data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110500063.3A CN113111056B (en) 2021-05-08 2021-05-08 Cleaning method for urban flood water monitoring data

Publications (2)

Publication Number Publication Date
CN113111056A true CN113111056A (en) 2021-07-13
CN113111056B CN113111056B (en) 2021-10-22

Family

ID=76721449

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110500063.3A Active CN113111056B (en) 2021-05-08 2021-05-08 Cleaning method for urban flood water monitoring data

Country Status (1)

Country Link
CN (1) CN113111056B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114757589A (en) * 2022-06-14 2022-07-15 深圳市拓安信计控仪表有限公司 Data processing method, server and storage medium
CN116186109A (en) * 2022-12-26 2023-05-30 中国长江电力股份有限公司 Method for inquiring time sequence data with value changed by information system

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100063773A1 (en) * 2007-04-30 2010-03-11 Marvasti Mazda A Nonparametric method for determination of anomalous event states in complex systems exhibiting non-stationarity
CN103365105A (en) * 2012-04-11 2013-10-23 上海微电子装备有限公司 Method for screening and correcting light intensity sampling points in alignment process
CN103620383A (en) * 2012-04-19 2014-03-05 松下电器产业株式会社 Device for detecting fluctuation in moisture content, method for detecting fluctuation in moisture content, vacuum gauge, and method for detecting fluctuation in vacuum degree
CN104066900A (en) * 2011-11-21 2014-09-24 约纳·西纳什 Method and apparatus for monitoring a network of conduits
US20180083833A1 (en) * 2016-09-16 2018-03-22 Oracle International Corporation Method and system for performing context-aware prognoses for health analysis of monitored systems
CN108920429A (en) * 2018-06-12 2018-11-30 河海大学 A kind of abnormal data analysis method of Water level trend monitoring
CN109325692A (en) * 2018-09-27 2019-02-12 清华大学合肥公共安全研究院 The data real-time analysis method and device of pipe network
CN110852577A (en) * 2019-10-25 2020-02-28 镇江德威乐普能源环保科技有限公司 Urban flood assessment method based on urban toughness and urban drainage basin hydrological model
US10585863B2 (en) * 2015-06-18 2020-03-10 Ihs Global Inc. Systems and methods for providing information services associated with natural resource extraction activities
US10635691B1 (en) * 2011-10-05 2020-04-28 Google Llc Database replication
CN111222698A (en) * 2020-01-06 2020-06-02 重庆邮电大学 Long-and-short-term memory network-based ponding water level prediction method for Internet of things
CN112286924A (en) * 2020-11-20 2021-01-29 中国水利水电科学研究院 Data cleaning technology for dynamic identification of data abnormality and multi-mode self-matching
CN112527788A (en) * 2020-12-17 2021-03-19 北京中恒博瑞数字电力科技有限公司 Method and device for detecting and cleaning abnormal value of transformer monitoring data
CN112734128A (en) * 2021-01-19 2021-04-30 重庆大学 7-day power load peak value prediction method based on optimized RBF

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100063773A1 (en) * 2007-04-30 2010-03-11 Marvasti Mazda A Nonparametric method for determination of anomalous event states in complex systems exhibiting non-stationarity
US10635691B1 (en) * 2011-10-05 2020-04-28 Google Llc Database replication
CN104066900A (en) * 2011-11-21 2014-09-24 约纳·西纳什 Method and apparatus for monitoring a network of conduits
CN103365105A (en) * 2012-04-11 2013-10-23 上海微电子装备有限公司 Method for screening and correcting light intensity sampling points in alignment process
CN103620383A (en) * 2012-04-19 2014-03-05 松下电器产业株式会社 Device for detecting fluctuation in moisture content, method for detecting fluctuation in moisture content, vacuum gauge, and method for detecting fluctuation in vacuum degree
US10585863B2 (en) * 2015-06-18 2020-03-10 Ihs Global Inc. Systems and methods for providing information services associated with natural resource extraction activities
US20180083833A1 (en) * 2016-09-16 2018-03-22 Oracle International Corporation Method and system for performing context-aware prognoses for health analysis of monitored systems
CN108920429A (en) * 2018-06-12 2018-11-30 河海大学 A kind of abnormal data analysis method of Water level trend monitoring
CN109325692A (en) * 2018-09-27 2019-02-12 清华大学合肥公共安全研究院 The data real-time analysis method and device of pipe network
CN110852577A (en) * 2019-10-25 2020-02-28 镇江德威乐普能源环保科技有限公司 Urban flood assessment method based on urban toughness and urban drainage basin hydrological model
CN111222698A (en) * 2020-01-06 2020-06-02 重庆邮电大学 Long-and-short-term memory network-based ponding water level prediction method for Internet of things
CN112286924A (en) * 2020-11-20 2021-01-29 中国水利水电科学研究院 Data cleaning technology for dynamic identification of data abnormality and multi-mode self-matching
CN112527788A (en) * 2020-12-17 2021-03-19 北京中恒博瑞数字电力科技有限公司 Method and device for detecting and cleaning abnormal value of transformer monitoring data
CN112734128A (en) * 2021-01-19 2021-04-30 重庆大学 7-day power load peak value prediction method based on optimized RBF

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
AOQIAN ZHANG等: "Time series data cleaning: from anomaly detection to anomaly repairing", 《PROCEEDINGS OF THE VLDB ENDOWMENT》 *
S.DILLING等: "Cleaning high-frequency velocity profile data with autoregressive moving average (ARMA) models", 《FLOW MEASUREMENT AND INSTRUMENTATION》 *
唐鸣: "南水北调中线工程水位模拟预测方法及应用", 《中国优秀博硕士学位论文全文数据库(硕士)工程科技Ⅱ辑》 *
甘永德等: "考虑空气阻力作用的分层土壤降雨入渗模型", 《水利学报》 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114757589A (en) * 2022-06-14 2022-07-15 深圳市拓安信计控仪表有限公司 Data processing method, server and storage medium
CN116186109A (en) * 2022-12-26 2023-05-30 中国长江电力股份有限公司 Method for inquiring time sequence data with value changed by information system
CN116186109B (en) * 2022-12-26 2024-01-05 中国长江电力股份有限公司 Method for inquiring time sequence data with value changed by information system

Also Published As

Publication number Publication date
CN113111056B (en) 2021-10-22

Similar Documents

Publication Publication Date Title
CN113111056B (en) Cleaning method for urban flood water monitoring data
CN109359698B (en) Leakage identification method based on long-time memory neural network model
CN111114519B (en) Railway vehicle brake fault prediction method and health management system
CN111709465B (en) Intelligent identification method for rough difference of dam safety monitoring data
CN115453662B (en) Abnormal site screening method combining time dimension and space dimension
CN112418517B (en) Prediction method for freeze injury short-term early warning
CN114114198B (en) Precipitation data quality control method and device, storage medium and equipment
CN110598265A (en) Bridge health data abnormity correction method and system based on wavelet analysis
CN104850678B (en) Road bridge expansion device running service performance evaluation method based on running performance
CN110254478B (en) Roadbed deformation disease identification method and device
CN112780953B (en) Independent metering area pipe network leakage detection method based on mode detection
CN108108839B (en) Power grid information system equipment state early warning method based on reverse fuzzy hierarchical analysis
CN112508444A (en) Municipal drainage pipeline assessment method
CN107273686B (en) Rain flood runoff nutrient output load forecasting method
CN116257811B (en) Abnormality processing method based on gas flow detection deep learning
CN117290666A (en) Photovoltaic abnormal power data cleaning method
CN115713312A (en) Combined identification method and device for unfavorable working conditions of hydraulic and hydroelectric engineering environmental quantity
Qu et al. Outlier Detection and Forecasting for Bridge Health Monitoring Based on Time Series Intervention Analysis.
CN115267950A (en) Initial rainwater quantification method and device, storage medium and electronic equipment
CN111222678B (en) Road surface technical condition prediction method
CN114580703A (en) Water quantity prediction method and device based on land remote sensing identification time-space change analysis
CN117688790B (en) Waterlogging flooding point pre-judging method based on multidimensional real-time monitoring and big data analysis
CN111242945A (en) Pipeline defect detection system based on big data
CN117719440B (en) Automobile signal detection method, system and readable storage medium
CN116629618B (en) Bridge disease diagnosis system based on Internet

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant