CN114742103A - City monitoring data processing method and device based on Internet of things and storage medium - Google Patents
City monitoring data processing method and device based on Internet of things and storage medium Download PDFInfo
- Publication number
- CN114742103A CN114742103A CN202210334997.9A CN202210334997A CN114742103A CN 114742103 A CN114742103 A CN 114742103A CN 202210334997 A CN202210334997 A CN 202210334997A CN 114742103 A CN114742103 A CN 114742103A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring data
- value
- preset threshold
- missing value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 99
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 44
- 230000008859 change Effects 0.000 claims abstract description 42
- 230000002159 abnormal effect Effects 0.000 claims abstract description 34
- 238000006073 displacement reaction Methods 0.000 claims abstract description 32
- 238000007781 pre-processing Methods 0.000 claims abstract description 18
- 238000009499 grossing Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 13
- 238000004590 computer program Methods 0.000 claims description 6
- 230000007812 deficiency Effects 0.000 claims description 5
- 230000006855 networking Effects 0.000 claims 1
- 238000000354 decomposition reaction Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000035772 mutation Effects 0.000 description 3
- 230000001364 causal effect Effects 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/02—Preprocessing
- G06F2218/04—Denoising
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A30/00—Adapting or protecting infrastructure or their operation
- Y02A30/60—Planning or developing urban green infrastructure
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- Educational Administration (AREA)
- Economics (AREA)
- Signal Processing (AREA)
- Development Economics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Complex Calculations (AREA)
Abstract
The invention discloses a city monitoring data processing method, a city monitoring data processing device and a storage medium based on the Internet of things, wherein the method comprises the following steps: collecting original monitoring data; comparing the original monitoring data with a preset threshold range, and removing the original monitoring data beyond the preset threshold range to obtain first preprocessing data; detecting actual displacement change of the first preprocessing data in a preset time window, if the actual displacement change exceeds a change range of a preset threshold, judging the data exceeding the change range of the preset threshold as abnormal values, and removing the abnormal values from the first preprocessing data to obtain second preprocessing data; performing noise smoothing processing on the second preprocessed data by adopting a VMD-PR algorithm to obtain de-noising data; and (4) carrying out missing value compensation processing on the de-noising data based on the improved Transformer model to obtain final monitoring data. The embodiment of the invention can effectively improve the accuracy of the city monitoring data.
Description
Technical Field
The invention relates to the technical field of city monitoring, in particular to a city monitoring data processing method and device based on the Internet of things and a storage medium.
Background
The Internet of Things (Internet of Things, IoT for short) is to collect any object or process needing monitoring, connection and interaction in real time and collect various required information such as sound, light, heat, electricity, mechanics, chemistry, biology and location through various devices and technologies such as various information sensors, radio frequency identification technologies, global positioning systems, infrared sensors and laser scanners, and to realize ubiquitous connection of objects and people through various possible network accesses, and to realize intelligent sensing, identification and management of objects and processes. Based on the internet of things technology, the system can dynamically monitor the rainfall, temperature, humidity and other relevant environmental data in the city in real time, and realizes identification and early warning of urban public safety hidden dangers, thereby protecting the safety of lives and properties of urban residents.
Sensing equipment such as building deformation monitoring sensors, rain gauges, temperature sensors and humidity sensors in cities provide real-time monitoring data, and are the basis for knowing city operation states and implementing city operation situation monitoring. The existing urban monitoring data processing method has the problems of instability of monitoring instruments, equipment failure and the like, so that the collected monitoring data has abnormity, deficiency, noise and the like, and the accuracy of the urban monitoring data is low.
Disclosure of Invention
The invention provides a city monitoring data processing method, a city monitoring data processing device and a storage medium based on the Internet of things, and aims to solve the problem that the accuracy of city monitoring data collected by the existing city monitoring data processing method is low.
One embodiment of the invention provides a city monitoring data processing method based on the Internet of things, which comprises the following steps:
collecting original monitoring data;
comparing the original monitoring data with a preset threshold range, and removing the original monitoring data beyond the preset threshold range to obtain first preprocessing data;
detecting actual displacement change of the first preprocessing data in a preset time window, if the actual displacement change exceeds a change range of a preset threshold, judging data corresponding to the change range exceeding the preset threshold as abnormal values, and removing the abnormal values from the first preprocessing data to obtain second preprocessing data;
performing noise smoothing processing on the second preprocessed data by adopting a VMD-PR algorithm to obtain de-noising data;
and carrying out missing value compensation processing on the de-noising data based on an improved Transformer model to obtain final monitoring data.
Further, if the actual displacement change exceeds a change range of a preset threshold, determining data corresponding to the actual displacement change as an abnormal value, and removing the abnormal value from the first preprocessed data to obtain second preprocessed data, including:
the abnormal value is judged by the following formula:
|A|>(0.95)and|A|>m
wherein, A is the differential displacement value, m is the minimum preset threshold value, Q (i), i belongs to [0,1] and represents that the data with the ratio i in the data set is less than Q (i).
Further, the performing noise smoothing processing on the second preprocessed data by using the VMD-PR algorithm to obtain denoised data includes:
decomposing the second preprocessed data into eigenmode function components of a plurality of different frequency scales by adopting a VMD-PR algorithm;
setting the high-frequency mode of the frame mode function component to be 0, and adopting the low-frequency signal in the frame mode function component to carry out signal reconstruction to obtain de-noising data.
Further, the setting the high-frequency mode of the frame mode function component to 0, and performing signal reconstruction by using the low-frequency signal in the frame mode function component to obtain the denoising data includes:
the reconstructed denoised data is represented as:
where m is the number of sub-modalities for signal reconstruction, and k is the number of subsequences.
Further, the performing missing value compensation processing on the denoising data based on the improved transform model to obtain final monitoring data includes:
replacing an embedding layer in the original Transformer model by TCN to obtain an improved Transformer model;
and based on an improved Transformer model, predicting a missing value according to information before and after the missing value in the de-noising data to obtain a compensation missing value, and inserting the compensation missing value into the position of the missing value to obtain final monitoring data.
An embodiment of the present invention provides an internet of things-based city monitoring data processing apparatus, including:
the data acquisition module is used for acquiring original monitoring data;
the first data removing module is used for comparing the original monitoring data with a preset threshold range and removing the original monitoring data beyond the preset threshold range to obtain first preprocessing data;
the second data removing module is used for detecting the actual displacement change of the first preprocessed data in a preset time window, judging the data corresponding to the change range exceeding a preset threshold value as an abnormal value if the actual displacement change exceeds the change range of the preset threshold value, and removing the abnormal value from the first preprocessed data to obtain second preprocessed data;
the de-noising module is used for performing noise smoothing processing on the second preprocessed data by adopting a VMD-PR algorithm to obtain de-noised data;
and the missing value compensation module is used for carrying out missing value compensation processing on the de-noising data based on the improved Transformer model to obtain final monitoring data.
Further, the second data culling module is configured to:
the abnormal value is judged by the following formula:
|A|>(0.95)and|A|>m
wherein, A is the differential displacement value, m is the minimum preset threshold value, Q (i), i belongs to [0,1] and represents that the data with the ratio i in the data set is less than Q (i).
Further, the denoising module is configured to:
decomposing the second preprocessed data into a plurality of eigenmode function components of different frequency scales by adopting a VMD-PR algorithm;
setting the high-frequency mode of the frame mode function component to be 0, and adopting the low-frequency signal in the frame mode function component to reconstruct the signal to obtain de-noising data.
Further, the deficiency value compensation module is configured to:
replacing an embedding layer in the original Transformer model with TCN to obtain an improved Transformer model;
and based on an improved Transformer model, predicting a missing value according to information before and after the missing value in the de-noising data to obtain a compensation missing value, and inserting the compensation missing value into the position of the missing value to obtain final monitoring data.
One embodiment of the present invention provides a computer-readable storage medium including a stored computer program; when the computer program runs, the device where the computer readable storage medium is located is controlled to execute the city monitoring data processing method based on the internet of things.
The embodiment of the invention collects the original monitoring data, sequentially eliminates the out-of-range data and the mutation data in the original monitoring data, and carries out noise smoothing processing and missing value compensation processing on the data, so that the urban monitoring data is more accurate. Furthermore, the VWD-PR algorithm is adopted to carry out modal decomposition on the second preprocessed data, the high-frequency mode is set to be 0, and only the low-frequency signal is adopted to carry out signal reconstruction, so that the second preprocessed data are denoised, the effect of denoising the signal can be effectively improved, and the accuracy of the urban monitoring data can be improved; in addition, the embodiment of the invention adopts the improved Transformer model to carry out missing value compensation processing on the de-noising data, realizes missing value compensation by combining information before and after the missing value, and can effectively improve the accuracy and reliability of the urban monitoring data.
Drawings
Fig. 1 is a schematic flow chart of a city monitoring data processing method based on the internet of things according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a TCN structure provided by an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of an improved Transformer model provided by an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a city monitoring data processing device based on the internet of things according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the description of the present application, it is to be understood that the terms "first", "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present application, "a plurality" means two or more unless otherwise specified.
In the description of the present application, it is to be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present application can be understood in a specific case by those of ordinary skill in the art.
Referring to fig. 1, an embodiment of the present invention provides a city monitoring data processing method based on the internet of things, including:
s1, collecting original monitoring data;
in the embodiment of the invention, a monitoring instrument can be adopted to collect original city monitoring data, and monitoring equipment comprises a building deformation monitoring sensor, a rain gauge, a temperature sensor, a humidity sensor and other sensors.
S2, comparing the original monitoring data with a preset threshold range, and eliminating the original monitoring data exceeding the preset threshold range to obtain first preprocessing data;
in the embodiment of the invention, a preset threshold range can be set according to an actual standard, data with values exceeding the preset threshold range are judged as out-of-range data, and the out-of-range data are removed from original monitoring data to obtain first preprocessing data.
S3, detecting the actual displacement change of the first preprocessed data in a preset time window, if the actual displacement change exceeds the change range of a preset threshold, judging the data corresponding to the change range exceeding the preset threshold as abnormal values, and removing the abnormal values from the first preprocessed data to obtain second preprocessed data;
in the embodiment of the present invention, the size of the time window may be set according to actual needs, and the method for monitoring the actual displacement change of the first preprocessed data includes: and drawing each data point of the first preprocessing data into a curve according to a preset time point interval, and calculating the displacement change of the current point compared with the previous point. In a specific embodiment, the displacement change of the current point compared to any historical point may also be calculated as the actual displacement change of the current point. The actual displacement change of each data point is compared with the change range of the preset threshold value in a fixed time window, and if the data point of the actual displacement change exceeding the change range of the preset threshold value exists, the data point is judged to be an abnormal value.
According to the embodiment of the invention, the abnormal value is removed from the first preprocessing data, so that the jump type data caused by various factors such as instability of a sensor, equipment failure and instrument maintenance can be effectively removed, and the accuracy of the monitoring data can be effectively improved.
S4, performing noise smoothing processing on the second preprocessed data by adopting a VMD-PR algorithm to obtain de-noising data;
and S5, carrying out missing value compensation processing on the de-noised data based on the improved Transformer model to obtain final monitoring data.
The embodiment of the invention collects the original monitoring data, sequentially eliminates the out-of-range data and the mutation data in the original monitoring data, and carries out noise smoothing processing and missing value compensation processing on the data, so that the urban monitoring data is more accurate. Furthermore, the VWD-PR algorithm is adopted to carry out modal decomposition on the second preprocessed data, the high-frequency mode is set to be 0, and only the low-frequency signal is adopted to carry out signal reconstruction, so that the second preprocessed data are denoised, the effect of denoising the signal can be effectively improved, and the accuracy of the urban monitoring data can be improved; in addition, the embodiment of the invention adopts the improved Transformer model to carry out missing value compensation processing on the de-noised data, realizes missing value compensation by combining information before and after the missing value, and can further improve the accuracy and reliability of the urban monitoring data.
In one embodiment, if the actual displacement variation exceeds the variation range of the preset threshold, determining the data corresponding to the actual displacement variation as an abnormal value, and removing the abnormal value from the first preprocessed data to obtain the second preprocessed data, including:
the abnormal value is judged by the following formula:
|A|>(0.95)and|A|>m
wherein, A is the differential displacement value, m is the minimum preset threshold value, Q (i), i belongs to [0,1] and represents that the data with the ratio i in the data set is less than Q (i).
In the embodiment of the invention, because the abnormal value often has the characteristics of short time and burst, the embodiment of the invention can further confirm the abnormal value by combining the front and back deformation rules of the data. If the difference between the abnormal value and the preorder time point and the difference between the abnormal value and the posterior time point are the same, namely, the abnormal value is larger than or smaller than the preorder time point record and the posterior time point record at the same time, the data point is determined to be the abnormal value.
In one embodiment, the noise smoothing processing is performed on the second preprocessed data by using a VMD-PR algorithm to obtain denoised data, and the method includes:
decomposing the second preprocessed data into a plurality of eigenmode function components with different frequency scales by adopting a VMD-PR algorithm;
in the embodiment of the invention, a VWM-PR algorithm is adopted to decompose the second preprocessed data into a plurality of relatively stable Intrinsic Mode Function (IMF) components with different frequency scales. Each IMF component is a bandwidth limited am-fm signal expressed as:
uk(t)＝Ak(t)cos(φk(t))
wherein A isk(t) is the instantaneous amplitude, [ phi ]kAnd (t) is the phase.
In the embodiment of the present invention, assuming that the original signal f (t), i.e. the second preprocessed data, is decomposed into K eigenmode function components, each component is approximately compact around the center frequency, and the sum of the estimated bandwidths of each component is minimum, the bandwidth is estimated by gaussian smoothness of the modulation signal, i.e. the L2 norm of the gradient, and the constrained variation problem implemented by the VMD algorithm in the embodiment of the present invention is expressed as follows:
wherein, ω iskIs the center frequency of the sub-component,
for analytic signals obtained by Hilbert transform of subcomponents, exponential terms
Converting the spectrum of the analytic signal of each component into a baseband; the optimal solution of the variational problem is solved by using a Lagrange multiplier and an alternative direction multiplier method, and then { u } can be obtainedk}:＝{u1,u2,...,uKThus, the decomposition of the original signal f (t) into K IMF components is achieved. Wherein, the value of K can be given artificially, and the number of subsequences decomposed by the EEMD method in the embodiment of the present invention is used as the value of K.
And setting the high-frequency mode of the frame mode function component as 0, and performing signal reconstruction by adopting a low-frequency signal in the frame mode function component to obtain de-noising data.
In one embodiment, setting the high-frequency mode of the frame mode function component to 0, and performing signal reconstruction by using the low-frequency signal in the frame mode function component to obtain the de-noising data, includes:
the reconstructed denoised data is represented as:
wherein m is the number of sub-modes for signal reconstruction, and k is the number of sub-sequences.
In an embodiment of the invention, the value of m is determined according to the correlation of the sub-modality with the original signal.
In the embodiment of the invention, high-frequency noise in signals can be effectively removed by adopting a VMD-PR algorithm in the urban monitoring data processing.
In one embodiment, the performing missing value compensation processing on the denoised data based on the improved Transformer model to obtain final monitoring data includes:
replacing an embedding layer in the original Transformer model with TCN to obtain an improved Transformer model;
in the embodiment of the invention, the original Transformer is mainly applied to the classification problem, and aiming at the problem of compensation of missing values of urban monitoring data, the TCN is adopted to replace an embedding layer in the original Transformer model, so that the original Transformer model is transformed to obtain the improved Transformer model. Referring to FIG. 2, TCN is mainly composed of Causal convolution (Causal convolution) and hole convolution (related convolution), and the Transformer model can be applied to the regression problem using TCN. Meanwhile, the characteristics of the front sample and the rear sample can be considered simultaneously by the aid of a self-attention mechanism, and compared with a traditional time sequence prediction model, the improved Transformer model can effectively combine information before and after a missing value to perform missing value compensation, and accuracy and reliability of city monitoring data processing can be effectively improved.
Referring to fig. 3, based on the improved Transformer model, the missing value is predicted according to information before and after the missing value in the denoised data to obtain a compensation missing value, and the compensation missing value is inserted into the position of the missing value to obtain final monitoring data.
Illustratively, the insertion of the compensation deletion value into the position of the deletion value is specifically as follows:
and after the compensation missing value passes through the TCN, splicing the characteristics at the corresponding moment, carrying out characteristic mixing on the characteristics through a full connection layer, adding the position code of the missing value, inputting the position code into a self-attention layer of an improved transform model, and completing the compensation of the missing value.
It can be understood that, in the embodiment of the present invention, for the noise influence caused by the accidental value jitter, the sliding window is adopted to remove the noise.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention collects the original monitoring data, sequentially eliminates the out-of-range data and the mutation data in the original monitoring data, and carries out noise smoothing processing and missing value compensation processing on the data, so that the city monitoring data is more accurate. Furthermore, the VWD-PR algorithm is adopted to carry out modal decomposition on the second preprocessed data, the high-frequency mode is set to be 0, and only the low-frequency signal is adopted to carry out signal reconstruction, so that the second preprocessed data are denoised, the effect of denoising the signal can be effectively improved, and the accuracy of the urban monitoring data can be improved; in addition, the embodiment of the invention adopts the improved Transformer model to carry out missing value compensation processing on the de-noised data, realizes missing value compensation by combining information before and after the missing value, and can further improve the accuracy and reliability of the urban monitoring data.
Referring to fig. 4, based on the same inventive concept as the above embodiment, an embodiment of the present invention provides an internet of things-based city monitoring data processing apparatus, including:
the data acquisition module 10 is used for acquiring original monitoring data;
the first data removing module 20 is configured to compare the original monitoring data with a preset threshold range, and remove the original monitoring data exceeding the preset threshold range to obtain first preprocessed data;
the second data removing module 30 is configured to detect an actual displacement change of the first preprocessed data within a preset time window, determine, if the actual displacement change exceeds a change range of a preset threshold, that data corresponding to the change range exceeding the preset threshold is an abnormal value, and remove the abnormal value from the first preprocessed data to obtain second preprocessed data;
the denoising module 40 is configured to perform noise smoothing processing on the second preprocessed data by using a VMD-PR algorithm to obtain denoised data;
and the deficiency value compensation module 50 is configured to perform deficiency value compensation processing on the denoised data based on the improved Transformer model to obtain final monitoring data.
In an embodiment, the second data culling module 30 is specifically configured to:
the abnormal value is judged by the following formula:
|A|>(0.95)and|A|>m
wherein, A is the differential displacement value, m is the minimum preset threshold value, Q (i), i belongs to [0,1] and represents that the data with the ratio i in the data set is less than Q (i).
In one embodiment, denoising module 40 is configured to:
decomposing the second preprocessed data into a plurality of eigenmode function components with different frequency scales by adopting a VMD-PR algorithm;
and setting the high-frequency mode of the frame mode function component as 0, and performing signal reconstruction by adopting a low-frequency signal in the frame mode function component to obtain de-noising data.
In one embodiment, the setting of the high-frequency mode of the frame mode function component to 0, and performing signal reconstruction by using the low-frequency signal in the frame mode function component to obtain the de-noising data includes:
the reconstructed denoised data is represented as:
wherein m is the number of sub-modes for signal reconstruction, and k is the number of sub-sequences.
In one embodiment, missing value compensation module 50 is configured to:
replacing an embedding layer in the original Transformer model with TCN to obtain an improved Transformer model;
based on an improved Transformer model, carrying out missing value prediction according to information before and after the missing value in the de-noising data to obtain a compensation missing value, and inserting the compensation missing value into the position of the missing value to obtain final monitoring data.
One embodiment of the present invention provides a computer-readable storage medium comprising a stored computer program; when the computer program runs, the device where the computer readable storage medium is located is controlled to execute the city monitoring data processing method based on the internet of things.
The foregoing is a preferred embodiment of the present invention, and it should be noted that it would be apparent to those skilled in the art that various modifications and enhancements can be made without departing from the principles of the invention, and such modifications and enhancements are also considered to be within the scope of the invention.
Claims (10)
1. A city monitoring data processing method based on the Internet of things is characterized by comprising the following steps:
collecting original monitoring data;
comparing the original monitoring data with a preset threshold range, and removing the original monitoring data beyond the preset threshold range to obtain first preprocessing data;
detecting actual displacement change of the first preprocessing data in a preset time window, if the actual displacement change exceeds a change range of a preset threshold, judging data corresponding to the change range exceeding the preset threshold as abnormal values, and removing the abnormal values from the first preprocessing data to obtain second preprocessing data;
performing noise smoothing processing on the second preprocessed data by adopting a VMD-PR algorithm to obtain de-noising data;
and carrying out missing value compensation processing on the de-noising data based on an improved Transformer model to obtain final monitoring data.
2. The method for processing city monitoring data based on the internet of things according to claim 1, wherein if the actual displacement change exceeds a change range of a preset threshold, determining data corresponding to the actual displacement change as an abnormal value, and removing the abnormal value from the first preprocessed data to obtain second preprocessed data, the method comprises:
the abnormal value is judged by the following formula:
|A|>(0.95)and|A|>m
wherein, A is the displacement value after the difference, m is the minimum preset threshold value, Q (i), i belongs to [0,1] and represents that the data with the occupation ratio i in the data set is less than Q (i).
3. The internet-of-things-based city monitoring data processing method of claim 1, wherein the noise smoothing processing is performed on the second preprocessed data by using a VMD-PR algorithm to obtain de-noised data, and the method comprises the following steps:
decomposing the second preprocessed data into eigenmode function components of a plurality of different frequency scales by adopting a VMD-PR algorithm;
setting the high-frequency mode of the frame mode function component to be 0, and adopting the low-frequency signal in the frame mode function component to reconstruct the signal to obtain de-noising data.
4. The method for processing city monitoring data based on the internet of things according to claim 3, wherein the setting of the high-frequency mode of the modal function component of the frame to 0, and the performing of signal reconstruction by using the low-frequency signal in the modal function component of the frame to obtain the de-noised data comprises:
the reconstructed denoised data is represented as:
wherein m is the number of sub-modes for signal reconstruction, and k is the number of sub-sequences.
5. The method for processing city monitoring data based on the internet of things as claimed in claim 1, wherein the performing missing value compensation processing on the de-noised data based on the improved Transformer model to obtain final monitoring data comprises:
replacing an embedding layer in the original Transformer model with TCN to obtain an improved Transformer model;
and based on an improved Transformer model, predicting a missing value according to information before and after the missing value in the de-noising data to obtain a compensation missing value, and inserting the compensation missing value into the position of the missing value to obtain final monitoring data.
6. The utility model provides a city monitoring data processing apparatus based on thing networking which characterized in that includes:
the data acquisition module is used for acquiring original monitoring data;
the first data removing module is used for comparing the original monitoring data with a preset threshold range and removing the original monitoring data beyond the preset threshold range to obtain first preprocessing data;
the second data removing module is used for detecting the actual displacement change of the first preprocessed data in a preset time window, judging the data corresponding to the change range exceeding a preset threshold value as an abnormal value if the actual displacement change exceeds the change range of the preset threshold value, and removing the abnormal value from the first preprocessed data to obtain second preprocessed data;
the de-noising module is used for performing noise smoothing processing on the second preprocessed data by adopting a VMD-PR algorithm to obtain de-noised data;
and the missing value compensation module is used for carrying out missing value compensation processing on the de-noising data based on the improved Transformer model to obtain final monitoring data.
7. The internet-of-things-based city monitoring data processing device of claim 6, wherein the second data culling module is configured to:
the abnormal value is judged by the following formula:
|A|>(0.95)and|A|>m
wherein, A is the differential displacement value, m is the minimum preset threshold value, Q (i), i belongs to [0,1] and represents that the data with the ratio i in the data set is less than Q (i).
8. The internet-of-things-based city monitoring data processing apparatus of claim 6, wherein the denoising module is configured to:
decomposing the second preprocessed data into eigenmode function components of a plurality of different frequency scales by adopting a VMD-PR algorithm;
setting the high-frequency mode of the frame mode function component to be 0, and adopting the low-frequency signal in the frame mode function component to reconstruct the signal to obtain de-noising data.
9. The internet-of-things-based city monitoring data processing apparatus of claim 6, wherein the deficiency value compensation module is configured to:
replacing an embedding layer in the original Transformer model by TCN to obtain an improved Transformer model;
and based on an improved Transformer model, predicting a missing value according to information before and after the missing value in the de-noising data to obtain a compensation missing value, and inserting the compensation missing value into the position of the missing value to obtain final monitoring data.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored computer program; wherein the computer program controls the device in which the computer readable storage medium is located to execute the method for processing the city monitoring data based on the internet of things according to any one of claims 1 to 5 when running.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210334997.9A CN114742103A (en) | 2022-03-31 | 2022-03-31 | City monitoring data processing method and device based on Internet of things and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210334997.9A CN114742103A (en) | 2022-03-31 | 2022-03-31 | City monitoring data processing method and device based on Internet of things and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114742103A true CN114742103A (en) | 2022-07-12 |
Family
ID=82279910
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210334997.9A Pending CN114742103A (en) | 2022-03-31 | 2022-03-31 | City monitoring data processing method and device based on Internet of things and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114742103A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116465623A (en) * | 2023-05-10 | 2023-07-21 | 安徽大学 | Gearbox service life prediction method based on sparse converter |
-
2022
- 2022-03-31 CN CN202210334997.9A patent/CN114742103A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116465623A (en) * | 2023-05-10 | 2023-07-21 | 安徽大学 | Gearbox service life prediction method based on sparse converter |
| CN116465623B (en) * | 2023-05-10 | 2023-09-19 | 安徽大学 | Gearbox service life prediction method based on sparse converter |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR100387457B1 (en) | System for surveillance of spectral signals | |
| CN106338406B (en) | The on-line monitoring of train traction electric drive system and fault early warning system and method | |
| Mechefske et al. | Fault detection and diagnosis in low speed rolling element bearings Part I: The use of parametric spectra | |
| CN111460392B (en) | Magnetic suspension train and suspension system fault detection method and system thereof | |
| CN113947017A (en) | Method for predicting residual service life of rolling bearing | |
| CN114167838B (en) | Multi-scale health assessment and fault prediction method for servo system | |
| CN110912737A (en) | Dynamic perception performance early warning method based on hybrid model | |
| CN111776006B (en) | Pre-alarming method and device for train axle temperature | |
| Janeliukstis | Continuous wavelet transform-based method for enhancing estimation of wind turbine blade natural frequencies and damping for machine learning purposes | |
| CN114623050A (en) | Method, system and medium for detecting abnormal operation data of wind turbine generator | |
| CN114742103A (en) | City monitoring data processing method and device based on Internet of things and storage medium | |
| CN115081331A (en) | Wind turbine generator running state abnormity detection method based on state parameter reconstruction error | |
| CN115165274A (en) | Self-adaptive intelligent monitoring device and method for vibration state of engineering mechanical equipment | |
| US20190245768A1 (en) | Monitoring a resource consumption of an application | |
| CN117668427A (en) | Temperature monitoring and predicting method and equipment for electricity consumption information system based on data analysis | |
| Yao et al. | A framework via impulses-oriented Gini index and extreme value distribution for rolling bearing dynamic fault alarm and identification | |
| CN115575880B (en) | Online detection method for metering error state of power transformer | |
| CN112128950A (en) | Machine room temperature and humidity prediction method and system based on multiple model comparisons | |
| CN112016800A (en) | Feature selection method and system based on effectiveness index | |
| CN111259926A (en) | Meat freshness detection method and device, computing equipment and storage medium | |
| CN116383750A (en) | Rolling bearing early-stage abnormality detection method based on windowed differential health index | |
| CN112229435A (en) | Ship main engine accidental fault detection method and system based on standard variable analysis | |
| CN112184645A (en) | Fan blade detection method and system and computer-storable medium thereof | |
| Kaddour et al. | Event Detection for Non-intrusive Load Monitoring using Tukey s Fences | |
| CN117273547B (en) | Production equipment operation data processing method based on edge calculation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |