CN116910674A - Water management monitoring method, device, equipment and medium based on data fusion inspection - Google Patents
Water management monitoring method, device, equipment and medium based on data fusion inspection Download PDFInfo
- Publication number
- CN116910674A CN116910674A CN202310862471.2A CN202310862471A CN116910674A CN 116910674 A CN116910674 A CN 116910674A CN 202310862471 A CN202310862471 A CN 202310862471A CN 116910674 A CN116910674 A CN 116910674A
- Authority
- CN
- China
- Prior art keywords
- data
- data set
- abnormal
- time series
- abnormal data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 88
- 238000000034 method Methods 0.000 title claims abstract description 65
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 title claims abstract description 62
- 230000004927 fusion Effects 0.000 title claims abstract description 50
- 238000007689 inspection Methods 0.000 title claims abstract description 10
- 230000002159 abnormal effect Effects 0.000 claims abstract description 119
- 238000001514 detection method Methods 0.000 claims abstract description 33
- 238000004364 calculation method Methods 0.000 claims abstract description 20
- 238000013450 outlier detection Methods 0.000 claims abstract description 16
- 230000005856 abnormality Effects 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims description 18
- 238000012360 testing method Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012216 screening Methods 0.000 claims description 11
- 238000012937 correction Methods 0.000 claims description 6
- 238000012806 monitoring device Methods 0.000 claims description 5
- 230000009466 transformation Effects 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000008901 benefit Effects 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 15
- 238000012423 maintenance Methods 0.000 description 14
- 238000004458 analytical method Methods 0.000 description 10
- 238000005070 sampling Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000004140 cleaning Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 238000011835 investigation Methods 0.000 description 6
- 230000000875 corresponding effect Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000000275 quality assurance Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
- G01N33/18—Water
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/251—Fusion techniques of input or preprocessed data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2123/00—Data types
- G06F2123/02—Data types in the time domain, e.g. time-series data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Health & Medical Sciences (AREA)
- Economics (AREA)
- Artificial Intelligence (AREA)
- Chemical & Material Sciences (AREA)
- Public Health (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Water Supply & Treatment (AREA)
- Food Science & Technology (AREA)
- Medicinal Chemistry (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- Immunology (AREA)
- Pathology (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention relates to the technical field of intelligent water affairs and discloses a water affair management monitoring method, device, equipment and medium based on data fusion inspection, which comprises the steps of firstly acquiring N pieces of time series data from N monitoring node sensors, respectively performing data outlier detection on the N pieces of time series data, and judging outlier data values to acquire a first abnormal data set; respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set; and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data. The method has the advantages that the continuity of time sequence data of a single sensor is considered, the data correlation among fusion acquisition of multiple sensors is considered, the accuracy and the effectiveness of abnormal data detection are improved, and the workload of manually judging data abnormality is reduced.
Description
Technical Field
The invention relates to the technical field of intelligent water affairs, in particular to a water affair management monitoring method, device, equipment and medium based on data fusion inspection.
Background
Along with the continuous development and progress of the Internet of things, the construction of the intelligent water service system has a direct and vital meaning for urban water supply. The intelligent water service system is used for effectively connecting all water supply systems so as to form a water supply management chain capable of performing intelligent management.
In the prior art, the water supply management is realized based on SCADA on-line monitoring and remote control, so that the management level of intelligent water service is effectively improved, but new problems are also exposed along with the continuous accumulation of monitoring data, such as: the intelligent sensing equipment such as the monitoring data machinery is subjected to acquisition and recording, sensors and the like, faults frequently occur, and complex environmental factors influence the like, so that various problem data appear in the monitoring. The quality of the monitoring data is greatly reduced, the analysis of the data and the extraction of effective information are limited, and the intelligent process of the system is prevented from being applied. In general, professional technical analysts need to grasp the data quality as a whole and feed back the abnormal data, but when a large number of sensing devices are accessed by the system, the amount of data generated by time sequence transmission is huge, the data types are mixed, and the abnormal data is difficult to comprehensively identify only by manual judgment operation.
Disclosure of Invention
In view of the above, the invention provides a water management monitoring method, a device, equipment and a medium based on data fusion inspection, which solve the technical problem of false alarm caused by abnormal data or data missing in the process of collecting data by a sensor and improve the accuracy and the effectiveness of water management monitoring.
According to a first aspect of the present invention, there is provided a water management monitoring method based on a data fusion test, comprising:
acquiring N time series data from N monitoring node sensors, wherein the time series data consists of continuous data acquired by the sensors within a preset time length, and N is an integer greater than 2;
respectively carrying out data outlier detection on the N time series data, and judging outlier data values to obtain a first abnormal data set;
respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set;
and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data.
According to a second aspect of the present invention, there is provided a water management monitoring device based on a data fusion test, comprising:
the acquisition module is used for acquiring N time series data from N monitoring node sensors, wherein the time series data consists of continuous data acquired by the sensors within a preset time length, and N is an integer greater than 2;
the first judging module is used for respectively carrying out data outlier detection on the N time series data and judging outlier data values so as to obtain a first abnormal data set;
the second judging module is used for respectively carrying out correlation anomaly detection on the N time series data and judging that the data value without correlation exists so as to acquire a second anomaly data set;
and the calculation module is used for carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and acquiring a third abnormal data set as monitoring abnormal data.
According to a third aspect of the present invention there is provided a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the above described data fusion check based water management monitoring method when executing the computer program.
According to a fourth aspect of the present invention, there is provided a computer readable storage medium storing a computer program which when executed by a processor performs the steps of the above water management monitoring method based on data fusion verification.
By means of the technical scheme, the water management monitoring method, the device, the equipment and the medium based on the data fusion test are provided, N pieces of time series data from N monitoring node sensors are firstly obtained, data outlier detection is respectively carried out on the N pieces of time series data, outlier data values are judged, and therefore a first abnormal data set is obtained; respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set; and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data. The method has the advantages that the continuity of time sequence data of a single sensor is considered, the data correlation among fusion acquisition of multiple sensors is considered, the accuracy and the effectiveness of abnormal data detection are improved, and the workload of manually judging data abnormality is reduced.
The foregoing description is only an overview of the present invention, and is intended to provide a better understanding of the technical means of the present invention, and is to be construed as an enabling description of the present invention in light of the accompanying detailed description of the invention, as well as the preferred embodiments of the present invention.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:
fig. 1 shows a schematic diagram of an application scenario of a water management monitoring method based on data fusion test provided in an embodiment of the present invention;
FIG. 2 shows a system architecture diagram for realizing the feedback and quality assurance of the water service Internet of things sensing data provided in the embodiment of the invention;
FIG. 3 shows a flow chart of a water management monitoring method based on data fusion test provided in an embodiment of the invention;
FIG. 4 is a schematic diagram showing another flow of a water management monitoring method based on data fusion test according to an embodiment of the present invention;
fig. 5 shows a schematic structural diagram of a water management monitoring device based on data fusion test according to an embodiment of the present invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the drawings in conjunction with embodiments. It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other.
As shown in fig. 1, the water management monitoring method based on the data fusion test is applied to a scene schematic diagram of an intelligent water service, the system is also based on an SCADA (supervisory control and data acquisition) on-line monitoring and remote control system, a system architecture diagram is shown in fig. 2, the water management monitoring method based on the data fusion test is realized from a water source to a water supply network, water related data are collected through sensors arranged at different positions, a plurality of sensors form a data collection layer, the data collection layer sends the data to an edge processing node for data fusion test, finally the data are sent to a cloud server node for storage and analysis, the edge processing node in the diagram only belongs to a logic layering concept, in practice, the edge processing node can be an independent node or a processing module locally integrated with the sensors, the edge processing node only refers to a logic layering concept, the edge processing node is responsible for executing the water management monitoring method based on the data fusion test provided by the embodiment of the invention, firstly, N time series data from N monitoring node sensors are obtained, data outlier detection is respectively carried out on the N time series data, and a cluster data value is judged to obtain a first abnormal data set; respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set; and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data. The method has the advantages that the continuity of time sequence data of a single sensor is considered, the data correlation among fusion acquisition of multiple sensors is considered, the accuracy and the effectiveness of abnormal data detection are improved, and the workload of manually judging data abnormality is reduced.
The following describes a water management monitoring method based on data fusion test according to an embodiment of the present invention with reference to a specific embodiment.
Example 1
In the practical application process, as shown in fig. 3, the water management monitoring method based on the data fusion test provided in the first embodiment includes:
step 301, acquiring N time series data from N monitoring node sensors;
the time sequence data consists of continuous data acquired by the sensors within a preset time length, N is an integer larger than 2, the concept of a sliding detection window is equivalent, the sliding window is set to be 4 hours, the data of N monitoring node sensors are continuous for 4 hours, and each monitoring node sensor corresponds to a time sequence;
step 302, respectively performing data outlier detection on the N time series data, and judging outlier data values to obtain a first abnormal data set;
step 302 may specifically include:
302-1, when the relative outlier distance of a certain time sequence data value value_i exceeds a relative outlier distance threshold and the length of the current time sequence containing the value_i is greater than a minimum length threshold of the subsequence, selecting the subsequence with the value_i as the center to calculate the relative outlier distance again;
step 302-2, when the relative outlier distance of the value_i exceeds the outlier distance threshold and the subsequence length is less than the subsequence minimum length threshold, marking the value_i as outlier data and placing it in the first outlier data set.
In step 302, the length of the sliding window is first set to len ts Minimum length threshold delta for subsequences in sliding window len Sub-sequence movement distance len move Relative outlier distance threshold τ distance Secondly, sequentially carrying out relative outlier distance passing through outlier distance and current data on time series data entering a sliding windowThe ratio is calculated as follows:
the time series of a certain sensor is simplified to be denoted tsn= { v1, v2, … vn }, where n = len ts And a certain value v t Outlier distance dist from all data in TSn sequence group The following formula is adopted for calculation:
distance to outlier
When a certain time sequence data value v t Is greater than a threshold tau distance And comprises v t The current sequence ts length is greater than delta len When selecting v t The subsequence is centered: TS (transport stream) sub ＝(v t -len move -1，v t -len move +1) recalculate v t Relative outlier distance dist of (2) r . In this way, the loop operation is performed when dist_r is greater than dist group And its length is less than delta len At the time, the value v of the sensing data at the time t t And recording as abnormal data and putting the abnormal data into a first abnormal data set.
The step reduces judgment errors generated by outlier calculation, and reduces error influence on outlier detection caused by environmental change and device parameter change by continuously focusing the subsequence on a numerical value near the outlier for judgment.
Step 303, respectively performing correlation anomaly detection on the N time series data, and judging that there is no data value of correlation so as to obtain a second anomaly data set;
step 303 may specifically include:
step 303-1, determining a correlation rule between the time series data according to the attributes of the N time series data; the correlation rule in this step may be formed by combining expert experience with theoretical relationships between physical quantities, or may be found by combining data statistics and machine learning, so that the correlation rule between time series data is determined by linear correlation, transformation correlation and transformation correlation according to the attribute of N time series data, for example, the linear correlation refers to a direct linear relationship between different time series, the combination correlation refers to a relationship between time series after combination operation, the transformation correlation refers to a relationship between time series after nonlinear computation such as logarithm, exponential operation, differentiation and the like is performed on the data of the time series, and the correlation detection in this step is similar to the feature engineering in the prior art, and is not repeated.
It should be noted here that, between the detection of correlated time sequences, a linear approximation function between the sequences, i.e. y=kx+b, needs to be solved by means of a least squares method, where k and b are determined by means of a least squares method, e.g. a direct correlation between binary time sequences (TS 1, TS 2), v 1t For the value of the time series TS1 at the point of time t, v 2t For the value of the time series TS2 at the point of time t, the corresponding v 2t The value of (c) is required to be within the range of (k-delta, k+delta) only to be normal, delta is required to be within a preset error margin, delta is determined empirically or by the fluctuation range of historical abnormal data, and if the value is not within the range, the value is determined to be second abnormal data, and a second abnormal data set is added, particularly, v is required to be calculated for the conditions of transformation correlation and conversion correlation 1t 、v 2t And carrying out corresponding conversion and then judging.
Step 303-2, judging whether the N time series data meet the correlation rule constraint between the time series according to the correlation rule;
step 303-3, when the time series data meet the correlation rule constraint among the time series, merging the time series data into a conforming constraint set, otherwise merging the time series data into a non-constraint set;
and 303-4, deleting time series data contained in the unconstrained set from the constrained set to obtain a second abnormal data set.
And 304, carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data.
Step 304 may specifically include:
step 304-1, locating the second abnormal data set based on the first abnormal data set, and removing data values overlapped with outlier data values in the first abnormal data set from the second abnormal data set to obtain a first abnormal screening data set;
step 304-2, screening the data sets conforming to the correlation relationship based on the first abnormal data set to obtain a second abnormal screening data set;
step 304-3, merging the first anomaly screening dataset with the second anomaly screening dataset to generate a third anomaly dataset.
After the abnormal data is found, the data needs to be corrected and then transmitted to the cloud storage, so after step 304, fitting correction or missing filling can be performed on the abnormal data of the third abnormal data set. Fig. 4 is a flowchart in the actual software running process, the time series data from a plurality of sensors are collected through the point distribution design of the sensors, the time series data can be temporarily stored in a memory database or directly stored in the time series database, then two processes of outlier detection and correlation detection are triggered and run in parallel, after the two processes are run, final abnormal data are obtained through fusion calculation, finally, the abnormal data are preprocessed, and the abnormal data can be corrected in a plurality of modes such as fitting correction, manual auxiliary correction, expert system correction and the like in the prior art, and then the corrected data are stored in the database, and meanwhile, the operation data of correction logs are recorded, so that the later-stage retrospection is facilitated.
In the embodiment of the present invention, the index attribute of the collected data is combined with the monitoring node sensor to cluster the edge computing nodes, the unified abnormal data detection is performed on the unified attribute sensor nodes, and the nodes with similar attributes form an edge processing unit group (corresponding to the N monitoring node sensors in the above embodiment), so before step 301, the method may include:
the monitoring node sensors collect index attributes of data, and the monitoring node sensors are clustered and divided into different edge processing unit groups.
In practical application, a plurality of edge processing unit groups may exist, so that the operation amount of data is effectively reduced, and the efficiency of abnormal data detection is improved.
In particular, under the conditions of large received data volume and complex parameters, different sequence data can be classified or grouped, and the load pressure of single-point processing is avoided by utilizing the idea of distributed processing.
The first embodiment of the invention provides a water management monitoring method based on data fusion test, which comprises the steps of firstly acquiring N pieces of time series data from N monitoring node sensors, respectively performing data outlier detection on the N pieces of time series data, and judging outlier data values to acquire a first abnormal data set; respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set; and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data. The method has the advantages that the continuity of time sequence data of a single sensor is considered, the data correlation among fusion acquisition of multiple sensors is considered, the accuracy and the effectiveness of abnormal data detection are improved, and the workload of manually judging data abnormality is reduced.
Example two
In combination with the flow of daily water service detection data guarantee operation, the whole flow method provided by the embodiment of the invention runs through the whole flow of data identification, processing, application and data quality guarantee realization, and the method is used as an auxiliary monitoring method for monitoring system users, internal industry analysts, internet of things monitoring systems and outside industry operation and maintenance personnel, and plays roles in daily investigation, problem feedback, problem investigation, problem disposal, sampling calibration, data cleaning and data application, and particularly, the water service management monitoring method based on the data fusion detection needs to be based on high-quality historical data in the implementation process, the quality of the historical data needs to be guaranteed by the outside industry operation and maintenance personnel, and the data guarantee operation needs to be correspondingly optimized in the process as follows:
daily investigation and problem feedback stage: the field operation and maintenance personnel record and upload problems found in daily inspection and maintenance through an M/S module (the M/S module corresponds to an M/S (Mobile/Server) system architecture, the architecture is an information system architecture based on a wireless network, and modules such as data acquisition and sensors in the Internet of things are all called as the M/S module), and an Internet of things monitoring system marks points of the problems identified by autonomous analysis and the problems uploaded by the M/S module. And the field analyst checks the monitoring data by using the B/S module, performs problem investigation, and completes the preliminary verification of the marked problems of the data. The monitoring system user can put forward data problems based on project construction purposes and requirements, and an inner industry analysis personnel is responsible for summarizing the multi-party problems and sorting the multi-party problems according to specific problems.
A problem investigation stage: and (3) screening the occurrence reasons of the problem data by an inner industry analyst, judging whether the problem data are caused by equipment faults, marking the details of the problem in an Internet of things monitoring system, and facilitating the assignment of professional outer industry operation and maintenance personnel to confirm the problem.
Problem handling stage: and (5) carrying out inspection treatment on the confirmed problem point positions by field operation and maintenance personnel. For the part of incomplete treatment, the field operation staff should make inspection record again and feed back the problem. For the processed problems, an operation process is required to be recorded, the marking state of the problem point location in the internet of things monitoring system is modified accordingly, in order to ensure the quality of problem treatment, the treatment rechecking is carried out again by an in-house analyst, if the treatment requirement is met, the operation and maintenance work is considered to be qualified, the marking of the problem point location in the internet of things monitoring system is canceled, and only normal inspection and maintenance work is required to be carried out for the point location in the follow-up process; if the rechecking does not meet the disposal requirement, the next stage operation is needed to be carried out on the problem point location.
Sampling and calibrating: firstly, judging whether the unqualified point location needs to execute a sampling calibration step by an internal analysis personnel, and if not, entering a data cleaning judgment. For the problem point position needing to be sampled and calibrated, field operation and maintenance personnel set an operation scheme to perform field sampling and detection, and the process is recorded in an Internet of things monitoring system and synchronously uploads sampling and detection data. The inner industry analyst should fit and verify the sampling detection data, the fitting result does not accord with the operation trend of the monitoring point, and the outer industry personnel should reform the scheme to sample and detect; fitting parameters are adopted when the trends are consistent, fitting functions are generated in the monitoring system, and a data cleaning mechanism is initially established. It should be noted that, in order to fully ensure the accuracy of the treatment data, the in-house analyst still needs to perform accuracy judgment on the fitting function, and the in-house analyst needs to reform a scheme to perform sampling detection if the accuracy requirement is not met; fitting functions meeting the precision requirements are applied in the next stage.
Data cleaning: and calibrating the monitoring points by using a fitting function meeting the precision requirement, realizing the application of the fitting function in the Internet of things monitoring system, and cleaning the problem data. If the problem data cannot be sampled and calibrated or cannot be cleaned after the process, the problem needs to be fed back and recorded again.
Data application stage: after the internet of things monitoring system finishes cleaning the problem data, generating normal time sequence data meeting the monitoring requirement, storing the normal time sequence data into a corresponding database, and carrying out data analysis application by field analysts. Meanwhile, the monitoring system user examines the data and analysis results in warehouse entry, and if the requirements of construction or targets are not met, the monitoring system user carries out problem feedback again; if the requirements of the user are met, the data quality is ensured, and the purposes of data acquisition, processing and application are achieved.
The quality of the historical data is effectively improved through the data guarantee operation of each stage, after the historical data is accumulated to a certain amount, a better application effect can be obtained by applying the water management monitoring method based on the data fusion test provided by the embodiment of the invention, and particularly, when the water management of a new environment is detected, sensor arrangement is adjusted, sensor components are added and deleted, updated sensor components are replaced and updated, the historical data is required to be collected and checked again by adopting the flow method, and a series of improvement effects are obtained through the flow of the water management detection data guarantee operation optimizing method provided by the embodiment of the invention, such as:
1. the acquisition and transmission of the whole process water service data are realized; the water service sensing system which covers water sources, water supply, water drainage and water environment is realized by utilizing information technologies such as internet of things sensing, cloud computing and the like in a unified management mode in the whole field.
2. The problems of multiple parties are summarized to carry out centralized feedback, repeated proposal of the problems is avoided, and the processing efficiency of single operation and maintenance work is improved conveniently; the problem points are marked by classification and arrangement, so that relevant personnel can reasonably and An Paiyun-dimensionally work according to classification conditions, and efficient and rapid pre-investigation and diagnosis are carried out on faults.
3. The method has the advantages that the operation limit is defined for each data related party based on the data quality problem discovery, feedback paths and guarantee mechanism, application barriers such as huge data quantity, data type mixing and abnormal problem discovery are overcome, comprehensive examination of online collected data is realized from multiple angles such as requirements, technologies and environments, data feedback and quality guarantee work are normally carried out, and data quality is effectively guaranteed.
4. Recording the processes of data quality problem treatment, sampling, calibration, detection and the like, and standardizing the development of data quality assurance work; in addition, when the final treatment does not reach the standard, whether the misoperation exists in the treatment work or not can be checked according to the process file, so that the time for the secondary feedback and treatment of the problems is effectively shortened.
5. And the processing quality of the data quality problem is grasped in multiple links, and the accuracy and the reliability of the warehouse-in application data are ensured. In the problem data feedback stage, professional technicians review the problems which are automatically analyzed and identified by the system, and the phenomenon of misjudgment on normal data caused by algorithm abnormality is avoided. Secondly, in the relevant stage of problem disposal, processing and evaluation systems such as sampling calibration, data cleaning, fitting analysis, trend judgment, precision requirement and the like are matched, so that the warehouse-in data is ensured to have analysis application value
6. The invention can construct a water service all-field Internet of things sensing system related to water source, water supply, water discharge and water environment, can efficiently integrate various water service information and resources, has the capabilities of water service monitoring, early warning, analysis, prediction and the like, coordinates multi-level cooperation of water service, and realizes high-level facility control and intelligent execution capability; based on the consideration of factors such as huge data volume, mixed data types, wide existence of data quality problems and the like of the online acquisition of the water service system, a data guarantee method is provided, so that the comprehensive and accurate identification, guidance, grasp and tracking of the data processing and operation and maintenance work of the problem data in the existing water service system can be effectively realized, and the follow-up management decision of obtaining an error result or causing hazard risk due to the analysis and application of the problem data is avoided; the method comprises the steps of comprehensively monitoring system users, inner industry analysts and outer industry operation and maintenance personnel, and forming a complete data guarantee chain of problem identification-problem verification-problem determination-problem disposal-result fitting-data warehousing application by means of an M/S module and a B/S module of an Internet of things monitoring platform; the method supports the recording and historical inquiring operation of the problem data, makes a periodic operation and maintenance plan referring to the occurrence frequency of the problem data, can effectively avoid hidden dangers of the data problem caused by the faults of sensing equipment and the like, and improves the scientificity of operation and maintenance strategy command assignment.
Further, as a specific implementation of the methods of fig. 2 and fig. 3, in an embodiment of the present invention, there is provided a water management monitoring device based on data fusion test, as shown in fig. 5, where the device includes:
the acquisition module 510 is configured to acquire N time-series data from N monitoring node sensors, where the time-series data is composed of continuous data acquired by the sensors within a preset time length, and N is an integer greater than 2;
the first determining module 520 is configured to perform data outlier detection on the N time-series data, and determine outlier data values to obtain a first abnormal data set;
a second judging module 530, configured to perform correlation anomaly detection on the N time-series data, and judge that there is no data value of correlation, so as to obtain a second abnormal data set;
the calculating module 540 is configured to perform fusion calculation on the first abnormal data set and the second abnormal data set, and obtain a third abnormal data set as monitoring abnormal data.
According to the water management monitoring device based on the data fusion test, N time series data from N monitoring node sensors are firstly obtained through an obtaining module, data outlier detection is respectively carried out on the N time series data, and outlier data values are judged to obtain a first abnormal data set; respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set; and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data. The method has the advantages that the continuity of time sequence data of a single sensor is considered, the data correlation among fusion acquisition of multiple sensors is considered, the accuracy and the effectiveness of abnormal data detection are improved, and the workload of manually judging data abnormality is reduced.
The embodiment of the invention provides a computer device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the following steps when executing the computer program:
acquiring N time series data from N monitoring node sensors, wherein the time series data consists of continuous data acquired by the sensors within a preset time length, and N is an integer greater than 2;
respectively carrying out data outlier detection on the N time series data, and judging outlier data values to obtain a first abnormal data set;
respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set;
and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data.
In an embodiment of the present invention, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
acquiring N time series data from N monitoring node sensors, wherein the time series data consists of continuous data acquired by the sensors within a preset time length, and N is an integer greater than 2;
respectively carrying out data outlier detection on the N time series data, and judging outlier data values to obtain a first abnormal data set;
respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set;
and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data.
It should be noted that, the functions or steps implemented by the computer readable storage medium or the computer device may correspond to the relevant descriptions of the server side and the client side in the foregoing method embodiments, and are not described herein for avoiding repetition.
Those skilled in the art will appreciate that implementing all or part of the above-described embodiment methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.
Claims (10)
1. The water management monitoring method based on the data fusion test is characterized by comprising the following steps of:
acquiring N time series data from N monitoring node sensors, wherein the time series data consists of continuous data acquired by the sensors within a preset time length, and N is an integer greater than 2;
respectively carrying out data outlier detection on the N time series data, and judging outlier data values to obtain a first abnormal data set;
respectively carrying out correlation anomaly detection on the N time series data, and judging that the data value of the correlation does not exist so as to acquire a second anomaly data set;
and carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and obtaining a third abnormal data set as monitoring abnormal data.
2. The method of claim 1, wherein the step of performing a fusion calculation on the first abnormal data set and the second abnormal data set to obtain a third abnormal data set as monitoring abnormal data includes:
positioning the second abnormal data set based on the first abnormal data set, and removing data values overlapped with outlier data values in the first abnormal data set from the second abnormal data set to obtain a first abnormal screening data set;
screening the data sets conforming to the correlation relationship based on the first abnormal data set to obtain a second abnormal screening data set;
and merging the first abnormality screening data set with the second abnormality screening data set to generate a third abnormality data set.
3. The method of claim 1, wherein the step of separately performing data outlier detection on the N time-series data to obtain the first abnormal data set includes:
when a certain time sequence data value v t The relative outlier distance of (2) exceeds the relative outlier distance threshold and includes v t When the current time sequence length is greater than the minimum sub-sequence length threshold, v is selected t Re-computing v for the centered subsequence t Is a relative outlier distance of (2);
when v t When the relative outlier distance of (2) exceeds the outlier distance threshold and the subsequence length is less than the subsequence minimum length threshold, v t And recording as abnormal data and putting the abnormal data into a first abnormal data set.
4. The method of claim 1, wherein the step of performing correlation anomaly detection on each of the N time series data to obtain a second anomaly data set comprises:
determining a correlation rule among the time series data according to the attributes of the N time series data;
judging whether the N time series data meet the correlation rule constraint between the time series according to the correlation rule;
when the time sequence data meets the constraint of the correlation rule between the time sequences, merging the time sequence data into a conforming constraint set, otherwise merging the time sequence data into a non-constraint set;
and deleting time series data contained in the unconstrained set from the constrained set to obtain a second abnormal data set.
5. The method of claim 4, wherein the step of determining a correlation rule between time series data based on the N time series data attributes comprises:
and determining a correlation rule among the time series data through linear correlation, transformation correlation and conversion correlation according to the attributes of the N time series data.
6. The method of claim 1, wherein the step of performing a fusion calculation on the first abnormal data set and the second abnormal data set to obtain a third abnormal data set as monitoring abnormal data, comprises:
and carrying out fitting correction or missing filling on the abnormal data of the third abnormal data set.
7. The method of claim 1, wherein prior to the step of acquiring N time series data from N monitoring node sensors, comprising:
the monitoring node sensors collect index attributes of data, and the monitoring node sensors are clustered and divided into different edge processing unit groups.
8. The utility model provides a water affair management monitoring devices based on data fusion inspection which characterized in that includes:
the acquisition module is used for acquiring N time series data from N monitoring node sensors, wherein the time series data consists of continuous data acquired by the sensors within a preset time length, and N is an integer greater than 2;
the first judging module is used for respectively carrying out data outlier detection on the N time series data and judging outlier data values so as to obtain a first abnormal data set;
the second judging module is used for respectively carrying out correlation anomaly detection on the N time series data and judging that the data value without correlation exists so as to acquire a second anomaly data set;
and the calculation module is used for carrying out fusion calculation on the first abnormal data set and the second abnormal data set, and acquiring a third abnormal data set as monitoring abnormal data.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the data fusion check-based water management monitoring method according to any one of claims 1 to 7 when the computer program is executed.
10. A computer-readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the data fusion check-based water management monitoring method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310862471.2A CN116910674A (en) | 2023-07-13 | 2023-07-13 | Water management monitoring method, device, equipment and medium based on data fusion inspection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310862471.2A CN116910674A (en) | 2023-07-13 | 2023-07-13 | Water management monitoring method, device, equipment and medium based on data fusion inspection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116910674A true CN116910674A (en) | 2023-10-20 |
Family
ID=88354323
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310862471.2A Pending CN116910674A (en) | 2023-07-13 | 2023-07-13 | Water management monitoring method, device, equipment and medium based on data fusion inspection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116910674A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117391374A (en) * | 2023-10-30 | 2024-01-12 | 江苏国贸酝领智能科技股份有限公司 | Intelligent water affair supervision method and system based on Internet of things technology |
| CN117651256A (en) * | 2023-11-28 | 2024-03-05 | 佛山科学技术学院 | Node energy consumption monitoring method and system based on outlier detection |
-
2023
- 2023-07-13 CN CN202310862471.2A patent/CN116910674A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117391374A (en) * | 2023-10-30 | 2024-01-12 | 江苏国贸酝领智能科技股份有限公司 | Intelligent water affair supervision method and system based on Internet of things technology |
| CN117391374B (en) * | 2023-10-30 | 2024-03-08 | 江苏国贸酝领智能科技股份有限公司 | Intelligent water affair supervision method and system based on Internet of things technology |
| CN117651256A (en) * | 2023-11-28 | 2024-03-05 | 佛山科学技术学院 | Node energy consumption monitoring method and system based on outlier detection |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116910674A (en) | Water management monitoring method, device, equipment and medium based on data fusion inspection | |
| CN109886430B (en) | Equipment health state assessment and prediction method based on industrial big data | |
| CN106598791B (en) | Industrial equipment fault preventive identification method based on machine learning | |
| CN112508053A (en) | Intelligent diagnosis method, device, equipment and medium based on integrated learning framework | |
| US10444746B2 (en) | Method for managing subsystems of a process plant using a distributed control system | |
| CN106446181A (en) | Big data processing and remote monitoring system | |
| US20240144043A1 (en) | Prediction model for predicting product quality parameter values | |
| US20230108309A1 (en) | Methods and systems for gas meter replacement prompt based on a smart gas internet of things | |
| CN116882823A (en) | Multi-level engineering quality assessment management platform with self-correction function | |
| CN111898673A (en) | Dissolved oxygen content prediction method based on EMD and LSTM | |
| CN111950577A (en) | Point inspection method and device | |
| CN111080484A (en) | Method and device for monitoring abnormal data of power distribution network | |
| CN116612549A (en) | Mobile terminal-based inspection track positioning analysis system | |
| CN117010549A (en) | Information tracing method and terminal based on big data management system | |
| Yang et al. | Streaming data analysis framework for cyber-physical system of metal machining processes | |
| CN113377630B (en) | Universal KPI anomaly detection framework implementation method | |
| CN111931798B (en) | Method for classifying and detecting cold head state and predicting service life | |
| CN114818460A (en) | Laboratory equipment residual service life prediction method based on automatic machine learning | |
| CN116415128A (en) | Method, system and medium for lubrication evaluation | |
| CN111271608A (en) | Leakage management system and method for compressed air system | |
| CN116401604B (en) | Method for classifying and detecting cold head state and predicting service life | |
| CN117010663B (en) | Intelligent gas data management method based on map, internet of things system and medium | |
| CN117849302A (en) | Multi-parameter water quality on-line monitoring method | |
| CN117952318A (en) | Industrial garden carbon emission data management system and method based on big data | |
| Carta et al. | Bad Data Detection and Handling in ICT Platforms for Energy Systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |