CN113469857A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113469857A CN113469857A CN202110811739.0A CN202110811739A CN113469857A CN 113469857 A CN113469857 A CN 113469857A CN 202110811739 A CN202110811739 A CN 202110811739A CN 113469857 A CN113469857 A CN 113469857A
- Authority
- CN
- China
- Prior art keywords
- target data
- data
- abnormal
- type
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 19
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 230000002159 abnormal effect Effects 0.000 claims abstract description 81
- 238000012545 processing Methods 0.000 claims abstract description 55
- 238000000034 method Methods 0.000 claims abstract description 44
- 238000001514 detection method Methods 0.000 claims abstract description 43
- 238000012544 monitoring process Methods 0.000 claims abstract description 38
- 238000004590 computer program Methods 0.000 claims description 6
- 230000007613 environmental effect Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000009499 grossing Methods 0.000 description 5
- 238000012417 linear regression Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012935 Averaging Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000002547 anomalous effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011109 contamination Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000003344 environmental pollutant Substances 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 231100000719 pollutant Toxicity 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000013077 scoring method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000010865 sewage Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000002759 z-score normalization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
Landscapes
- Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Strategic Management (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a data processing method and device, electronic equipment and a storage medium, and belongs to the technical field of data processing. The method comprises the following steps: detecting target data based on a preset data detection rule, and determining whether the target data is abnormal; and if so, processing the target data according to an exception handling mode associated with the exception type to which the target data belongs. According to the technical scheme, abnormal data in the pollution source online monitoring system can be rapidly found, the abnormal data can be flexibly processed, human resources are saved, and a new thought is provided for finding and processing the abnormal data in the pollution source online monitoring system.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data processing method and device, electronic equipment and a storage medium.
Background
The pollution source online monitoring system is used as an important means for environment supervision, and has an important effect on improving the environment supervision level. The effectiveness of the pollution source online monitoring data is directly related to the exertion of the function of the pollution source online monitoring system. The on-line monitoring system uploads the data generated by various analytical instruments, sensors and the like to the environmental protection law enforcement department. Because the monitoring equipment is influenced by uncertain factors such as environment, equipment loss, human intervention and the like, the online monitoring data acquired by the environmental protection department often has the conditions of data error, repetition, abnormality and the like. Abnormal monitoring data cannot truly reflect the actual pollution discharge situation of an enterprise, and meanwhile, the monitoring alarm, administrative law enforcement and punishment judgment of an environmental protection department can cause wrong influence. Therefore, an effective data management method is needed to find and process abnormal online monitoring data.
Disclosure of Invention
The invention provides a data processing method, a data processing device, electronic equipment and a storage medium, which are used for realizing automatic identification and timely processing of abnormal data.
In a first aspect, an embodiment of the present invention provides a data processing method, which is applied to a pollution source monitoring system, and the method includes:
detecting target data based on a preset data detection rule, and determining whether the target data is abnormal;
and if so, processing the target data according to an exception handling mode associated with the exception type to which the target data belongs.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus configured in a pollution source monitoring system, where the apparatus includes:
the anomaly determination module is used for detecting target data based on a preset data detection rule and determining whether the target data is abnormal or not;
and if so, processing the target data according to an exception handling mode associated with the exception type to which the target data belongs.
In a third aspect, an embodiment of the present invention further provides an electronic device, including:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data processing method provided by any embodiment of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data processing method provided in any embodiment of the present invention.
According to the technical scheme of the embodiment of the invention, the target data is detected based on the preset data detection rule, whether the target data is abnormal or not is determined, and if yes, the target data is processed according to the abnormal processing mode associated with the abnormal type of the target data. According to the technical scheme, abnormal data in the pollution source online monitoring system can be rapidly found, the abnormal data can be flexibly processed, human resources are saved, and a new thought is provided for finding and processing the abnormal data in the pollution source online monitoring system.
Drawings
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, where the embodiment is applicable to a situation where abnormal data in an online pollution source monitoring system is detected and processed, and the method may be executed by a data processing apparatus, where the apparatus may be implemented by software and/or hardware, and incorporated into an electronic device that carries a data processing function, such as a server.
As shown in fig. 1, the method may specifically include:
s110, detecting the target data based on a preset data detection rule, and determining whether the target data is abnormal.
In this embodiment, the target data refers to data that needs to be subjected to anomaly detection in the pollution source online monitoring system. The preset data detection rule may include a data common detection rule and a data service detection rule, where the data common detection rule is used to detect data itself, and the data service detection rule is used to detect a service attribute to which the data belongs.
The so-called common detection rule of data can comprise null value check, unique check, code check, value range check, data type check, repeatability check, normative check, referential integrity check, outlier check and the like; the null value check is to check the values of a specific table and a specific column, such as whether the enterprise address, the legal person, the longitude and latitude and the like are null values; the unique check is a unique check on the field provided, such as a contamination source unique number; the code check is to check whether the field value is in a code table (defined in advance by the system), and the field default with the code application in the source data is checked; the value range check is to judge whether the data to be detected meets the defined value range of the index metadata; the data type check means whether the data to be detected meets the definition type of the metadata; the repeatability check means that a set is formed by a specific table and a column or a combination of columns, and whether repeated values exist or not is found, or a repeated set is found; the normative check refers to checking specific normative on a specific table and a specific column, such as constraint check of rules of identity cards, mobile phone numbers, mailboxes, postal codes, fixed phones, IP addresses, MAC addresses, dates and the like; the referential integrity check refers to the check of the integrity of the referential between the main external keys of the data and the discovery of the empty referential of the external keys to the main keys; the outlier inspection refers to the step of performing distribution analysis on the content of the numerical field, verifying the distribution of data according to the Lauda method or the Grubbs method, and feeding back the outlier found to a corresponding business department.
The data service detection rule is a corresponding detection rule specified for the service field to which the data belongs, for example, for index data of environmental quality class, the evaluation result of the environmental quality is formulated according to the service attribute rules of national standard and line standard mainly from the relevant monitoring standard and evaluation standard of the environmental quality. For example, the relevant service indexes of the atmospheric environment may refer to the technical specification of Air Quality Index (AQI), and the rules of service limit values are formulated for the Air Quality Index, the pollutant item concentration, and the like. For the abnormal detection of the environmental supervision data, business rules can be formulated for industry attributes, region attributes, name attributes, management type attributes and the like of the pollution source according to administrative permission and administrative management specifications.
Optionally, a target type of the target data may be determined; and detecting the target data based on the data detection rule associated with the target type to determine whether the target data is abnormal. Wherein, one target type can be associated with a plurality of data detection rules.
Specifically, the target type of the target data is determined according to a predefined data standard, and then the target data is detected based on a data detection rule associated with the target type to determine whether the target data is abnormal. For example, if the target data is table data, null value check may be used to detect whether each field in the table has a null value, and value range check may also be used to detect whether the value of each field data in the table meets the defined value range of the index metadata; a repeatability check may also be employed to detect whether duplicate values or overlapping sets exist in the data of a set of characteristic columns or combinations of columns in the table data, and so forth.
Before the target data is detected, data standards are formulated, for example, data asset combing is performed according to business domains and data types of online monitoring of enterprises, and relevant standards of production, management and application of the data assets are formulated. Firstly, defining a uniform identifier, specifically comprising a data identifier, an equipment identifier, an organization identifier, a type identifier and the like, and forming a standard center for online monitoring data by referring to national and local standards. Secondly, defining metadata standards related to online monitoring business data by researching and combing the online monitoring business and data, wherein the metadata standards comprise: code specification of online monitoring data assets, core metadata contents (including Chinese paraphrases, remarks, logic primary keys, quality detection rules, data sources, data types, gate department authority, analysis rules and the like), and thirdly, standardizing a data model for online monitoring, including: naming rules of each service main data model, field naming rules, dictionary table reference relations, data types, field lengths, logic grouping and the like; fourthly, the standardization of data operation mainly comprises: data cleaning processing rules, data integration naming rules, scheduling task rules, data development rules and the like.
And S120, if so, processing the target data according to the exception handling mode associated with the exception type to which the target data belongs.
In this embodiment, if it is determined that the target data is abnormal, the abnormal type to which the target data belongs is determined, and the target data is processed according to the abnormal processing mode associated with the abnormal type described in the target data.
According to the technical scheme of the embodiment of the invention, the target data is detected based on the preset data detection rule, whether the target data is abnormal or not is determined, and if yes, the target data is processed according to the abnormal processing mode associated with the abnormal type of the target data. According to the technical scheme, abnormal data in the pollution source online monitoring system can be rapidly found, the abnormal data can be flexibly processed, human resources are saved, and a new thought is provided for finding and processing the abnormal data in the pollution source online monitoring system.
On the basis of the above embodiment, as an optional manner of the embodiment of the present invention, if the target data is abnormal, the target data may be marked, and the abnormality warning information may be sent to the terminal.
Optionally, the target data may be marked in a highlight mode for the abnormal target data, and the address information and the abnormal information of the abnormal target data may be sent to the terminal through voice or a short message.
It can be understood that by marking the abnormal target data, the abnormal target data can be clearly marked for the administrator to check; and the prediction early warning information is sent to the terminal, and related personnel can be informed to process the abnormal data in time.
Example two
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention; on the basis of the above embodiment, optimization is carried out, and an alternative implementation scheme is provided.
As shown in fig. 2, the method may specifically include:
s210, based on the business scene to which the target data belongs and/or a preset standardization rule, standardizing the target data.
In this embodiment, because different online pollution source monitoring data have different dimensions and dimension units, such data may affect detection and analysis of the data, and in order to ensure that the influence of the dimensions on the monitoring data is reduced, the target data may be standardized based on a service scenario to which the target data belongs, and different processing algorithms and processing modes are adopted for the data standardization processing aiming at different information fields based on different service scenarios. For example, if the target data belongs to the air environment type data, the target data may be normalized based on the relevant business standard of the atmospheric environment. For another example, for index monitoring data acquired by the monitoring device automatically, based on the requirement of data analysis modeling, a standardized algorithm may be used for automatic preprocessing. The standardization of partial data fields needs to be mainly carried out in a manual mode, for example, for an enterprise pollution source coordinate information field of 1-3-5 kilometers of pollution source data, the coordinates expressed in different units can be uniformly converted in an automatic calculation mode, and the uniform standard of a numerical value expression format needs manual intervention processing.
Optionally, the target data may be normalized based on a preset normalization rule. For example, the target data can be set to the same dimensional level by min-max normalization, Z-score normalization, and the like.
Optionally, the target data may be standardized based on a service scenario to which the target data belongs and a preset standardization rule.
S220, detecting the target data based on a preset data detection rule, and determining whether the target data is abnormal.
In this embodiment, based on a preset data detection rule, the target data may be detected by identifying whether the target data includes outlier data; the outlier data refers to data of a boundary outlier in the data, or noise data. Further, if the target data includes outlier data, determining that the abnormal type of the target data is a noise data type.
Optionally, based on a preset data detection rule, detecting the target data may be to identify whether there is a deviation between an actual receiving time point of the target data and an ideal receiving time point corresponding to the target data; the actual receiving time point refers to the time point when the target data are transmitted to the pollution source online monitoring system; the ideal reception time point refers to a time point at which the target data should theoretically be transmitted to the pollution source online monitoring system. And if the actual receiving time point of the target data is different from the ideal receiving time point corresponding to the target data, determining that the abnormal type of the target data is the jitter data type.
Optionally, based on a preset data detection rule, detecting the target data may be detecting the integrity of the target data. Specifically, if the target data is incomplete, the abnormal type of the target data is determined to be the missing data type.
Optionally, based on a preset data detection rule, the detection of the target data may be to identify whether the target data is within a set value range. Specifically, if the value of the target data is not within the set value range, the exception type of the target data is determined to be a logical error exception type.
Optionally, based on a preset data detection rule, the detection of the target data may be consistency comparison between the target data and reported data corresponding to the target data; wherein, the reported data refers to the data of reporting records by the enterprise to the supervision department before pollution discharge; the target data refers to data which is actually detected by sensing equipment at the sewage discharge port of the enterprise and uploaded to the pollution source online monitoring system in real time. Specifically, if the target data is inconsistent with the reported data corresponding to the target data, the abnormal type of the target data is determined to be an inconsistent data type.
Optionally, the target data may be detected based on a preset data detection rule, where the detection may be repeated detection of the target data. Specifically, the target data may be repeatedly identified based on a sorting and merging method, grouping by key field, identifying context information, identifying based on domain and business knowledge, identifying data characteristics, and the like, and if the target data has repeated data, the abnormal type of the target data is determined to be a repeated data type.
And S230, if so, processing the target data according to the exception handling mode associated with the exception type to which the target data belongs.
Optionally, if the abnormal type of the target data is a noise data type, denoising processing is performed on the target data. Specifically, methods such as linear regression, binning, outlier analysis, and the like may be used to process the abnormal target data. Linear regression is to find the direct optimal straight line fitting two variables, and when a specific value of one variable is given, the linear regression can be used for predicting the other variable; when the data variable is plural, multiple linear regression may be employed. Binning refers to analyzing data by using a "neighborhood" method, and the data is smooth and effective, and the data is distributed into some bins, which may include a bin mean smoothing method, a bin number smoothing method, a bin boundary smoothing method, and the like.
Optionally, if the heterogeneous type of the target data is a jitter data type, performing jitter offset processing on the target data. Specifically, an appropriate smoothing method may be constructed based on a moving average method, an exponential average method, an SG filtering method, or the like, and the target data may be processed by the dither offset processing to improve the usability of the data. The moving average method is based on an average method, new data are gradually added or old data are reduced according to the sequence, a moving average value is calculated to eliminate accidental variation factors, data are predicted, and the method is suitable for the conditions that the real value is not large, linear variation exists, and the noise average value is zero data; the exponential averaging method is moving average with exponential descending weighting, and the exponential moving averaging method has stronger real-time performance than the sliding averaging method and is closer to the observed value at the current moment; the SG (Savitzky-Golay) filtering method is a polynomial smoothing algorithm based on the least square principle, and the core is to carry out weighted filtering on data, so that the information of data signal change can be more effectively reserved.
Optionally, if the abnormal type of the target data is the missing data type, the target data is padded. Specifically, the target data may be subjected to padding processing by an average value method, a median method, a mode method, a regression padding method, a near padding method, a trend scoring method, or the like.
Optionally, if the exception type of the target data is a logical error data type, performing logical verification on the target data. Specifically, a historical data comparison method can be used for observing the change trend, characteristics and rules of the data through analysis of historical data, so that the correctness of the data is further verified. And the data with larger increase (or decrease) of the concentration (content) of the same proportion can be mainly checked during evaluation according to the characteristics of the target data, and the correct data needs to be reported again by related units if the data is confirmed to be wrong after the verification.
Optionally, if the abnormal type of the target data is an inconsistent data type, an error prompt of the target data is sent to the user terminal. Specifically, the error prompt of the target data can be sent to the user terminal in the forms of short message, telephone, voice and the like. The error prompt message may include error data.
Optionally, if the abnormal type of the target data is a repeated data type, deleting the target data. Specifically, the duplicated target data is deleted.
According to the technical scheme of the embodiment of the invention, the target data is subjected to standardized processing based on the business scene to which the target data belongs and/or a preset standardized rule, then the target data is detected based on a preset data detection rule, whether the target data is abnormal or not is determined, and if yes, the target data is processed according to an abnormal processing mode related to the abnormal type to which the target data belongs. According to the technical scheme, abnormal data in the pollution source online monitoring system can be rapidly found, the abnormal data can be flexibly processed, human resources are saved, and a new thought is provided for finding and processing the abnormal data in the pollution source online monitoring system.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention; the device can be realized in a software and/or hardware mode and is integrated in electronic equipment bearing a data processing function, such as a server.
As shown in fig. 3, the apparatus may include an anomaly determination module 310 and a data processing module 320, wherein,
an anomaly determination module 310, configured to detect target data based on a preset data detection rule, and determine whether the target data is anomalous;
and the data processing module 320 is configured to, if yes, process the target data according to an exception handling manner associated with the exception type to which the target data belongs.
According to the technical scheme of the embodiment of the invention, the target data is detected based on the preset data detection rule, whether the target data is abnormal or not is determined, and if yes, the target data is processed according to the abnormal processing mode associated with the abnormal type of the target data. According to the technical scheme, abnormal data in the pollution source online monitoring system can be rapidly found, the abnormal data can be flexibly processed, human resources are saved, and a new thought is provided for finding and processing the abnormal data in the pollution source online monitoring system.
Further, the device further comprises a standardization processing module, and the standardization processing module is specifically configured to:
and standardizing the target data based on the service scene to which the target data belongs and/or a preset standardization rule.
Further, the anomaly determination module 310 includes a target type determination unit and an anomaly determination unit, wherein,
a target type determination unit for determining a target type of the target data;
and the abnormity determining unit is used for detecting the target data based on the data detection rule associated with the target type and determining whether the target data is abnormal.
Further, the anomaly determination module 310 is specifically configured to:
identifying whether the target data comprises outlier data;
identifying whether the actual receiving time point of the target data and the ideal receiving time point corresponding to the target data have deviation or not;
detecting the integrity of the target data;
identifying whether the target data is within a set numerical range;
carrying out consistency comparison on the target data and reported data corresponding to the target data;
and detecting the repeatability of the target data.
Further, the data processing module 320 is specifically configured to:
if the abnormal type of the target data is a noise data type, denoising the target data;
if the different type of the target data is the jitter data type, performing jitter offset processing on the target data;
if the abnormal type of the target data is the missing data type, filling the target data;
if the abnormal type of the target data is a logic error data type, performing logic check on the target data;
if the abnormal type of the target data is the inconsistent data type, sending an error prompt of the target data to the user terminal;
and if the abnormal type of the target data is the repeated data type, deleting the target data.
Further, the device further comprises an early warning information sending module, wherein the early warning information sending module is specifically used for:
and if the target data is abnormal, marking the target data and sending abnormal early warning information to the terminal.
The data processing device can execute the data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention, and fig. 4 shows a block diagram of an exemplary device suitable for implementing the embodiment of the present invention. The device shown in fig. 4 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.
As shown in FIG. 4, electronic device 12 is embodied in the form of a general purpose computing device. The components of electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory (cache 32). The electronic device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of embodiments described herein.
The processing unit 16 executes various functional applications and data processing, for example, implementing a data processing method provided by an embodiment of the present invention, by executing a program stored in the system memory 28.
EXAMPLE five
Fifth, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program (or referred to as computer-executable instructions) is stored, where the computer program is used to execute a data processing method provided in the embodiment of the present invention when executed by a processor, and the method includes:
detecting target data based on a preset data detection rule, and determining whether the target data is abnormal;
and if so, processing the target data according to the exception handling mode associated with the exception type to which the target data belongs.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the embodiments of the present invention have been described in more detail through the above embodiments, the embodiments of the present invention are not limited to the above embodiments, and many other equivalent embodiments may be included without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A data processing method is applied to a pollution source monitoring system, and comprises the following steps:
detecting target data based on a preset data detection rule, and determining whether the target data is abnormal;
and if so, processing the target data according to an exception handling mode associated with the exception type to which the target data belongs.
2. The method according to claim 1, wherein before detecting the target data based on the preset data detection rule and determining whether the target data is abnormal, the method further comprises:
and standardizing the target data based on the service scene to which the target data belongs and/or a preset standardization rule.
3. The method according to claim 1, wherein the detecting target data based on a preset data detection rule to determine whether the target data is abnormal comprises:
determining a target type of target data;
and detecting the target data based on a data detection rule associated with the target type, and determining whether the target data is abnormal.
4. The method according to claim 1, wherein the detecting the target data based on the preset data detection rule comprises at least one of the following:
identifying whether outlier data is included in the target data;
identifying whether the actual receiving time point of the target data and the ideal receiving time point corresponding to the target data have deviation or not;
detecting the integrity of the target data;
identifying whether the target data is within a set numerical range;
carrying out consistency comparison on the target data and reported data corresponding to the target data;
and detecting the repeatability of the target data.
5. The method according to claim 1, wherein the processing the target data according to the exception handling manner associated with the exception type to which the target data belongs includes:
if the abnormal type of the target data is a noise data type, denoising the target data;
if the different type of the target data is a jitter data type, performing jitter offset processing on the target data;
if the abnormal type of the target data is a missing data type, filling the target data;
if the abnormal type of the target data is a logic error data type, performing logic check on the target data;
if the abnormal type of the target data is an inconsistent data type, sending an error prompt of the target data to a user terminal;
and if the abnormal type of the target data is the repeated data type, deleting the target data.
6. The method of claim 1, further comprising:
and if the target data is abnormal, marking the target data and sending abnormal early warning information to a terminal.
7. A data processing apparatus configured for use in a pollution source monitoring system, the apparatus comprising:
the anomaly determination module is used for detecting target data based on a preset data detection rule and determining whether the target data is abnormal or not;
and if so, processing the target data according to an exception handling mode associated with the exception type to which the target data belongs.
8. The apparatus of claim 7, further comprising:
and the standardization processing module is used for carrying out standardization processing on the target data based on the business scene to which the target data belongs and/or a preset standardization rule.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data processing method as claimed in any one of claims 1-6.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110811739.0A CN113469857A (en) | 2021-07-19 | 2021-07-19 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110811739.0A CN113469857A (en) | 2021-07-19 | 2021-07-19 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113469857A true CN113469857A (en) | 2021-10-01 |
Family
ID=77880956
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110811739.0A Pending CN113469857A (en) | 2021-07-19 | 2021-07-19 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113469857A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113901043A (en) * | 2021-12-13 | 2022-01-07 | 西安中电环通数字科技有限公司 | Pollution source intelligent supervision and data fusion analysis method and system |
| CN114567536A (en) * | 2022-02-24 | 2022-05-31 | 北京百度网讯科技有限公司 | Abnormal data processing method and device, electronic equipment and storage medium |
| CN114897516A (en) * | 2022-07-12 | 2022-08-12 | 山东乐习信息科技有限公司 | Method for overall process data management of special equipment |
| CN117312295A (en) * | 2023-11-29 | 2023-12-29 | 中国电力科学研究院有限公司 | Method, device, medium and equipment for managing power grid material spot check data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617360A (en) * | 2013-12-03 | 2014-03-05 | 力合科技（湖南）股份有限公司 | Early-warning event processing method and device |
| CN108830765A (en) * | 2018-04-18 | 2018-11-16 | 中国地质大学（武汉） | A kind of checking method and system of pollution entering the water monitoring data |
| CN109783552A (en) * | 2018-11-28 | 2019-05-21 | 中国电力科学研究院有限公司 | A kind of data cleansing restorative procedure |
| CN111752936A (en) * | 2020-06-30 | 2020-10-09 | 中国科学院西北生态环境资源研究院 | Data detection management method, device, server and readable storage medium |
| CN111814203A (en) * | 2020-09-08 | 2020-10-23 | 江苏神彩科技股份有限公司 | Method, device and equipment for processing environmental data and storage medium |
| CN112379656A (en) * | 2020-10-09 | 2021-02-19 | 爱普（福建）科技有限公司 | Processing method, device, equipment and medium for detecting abnormal data of industrial system |
-
2021
- 2021-07-19 CN CN202110811739.0A patent/CN113469857A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617360A (en) * | 2013-12-03 | 2014-03-05 | 力合科技（湖南）股份有限公司 | Early-warning event processing method and device |
| CN108830765A (en) * | 2018-04-18 | 2018-11-16 | 中国地质大学（武汉） | A kind of checking method and system of pollution entering the water monitoring data |
| CN109783552A (en) * | 2018-11-28 | 2019-05-21 | 中国电力科学研究院有限公司 | A kind of data cleansing restorative procedure |
| CN111752936A (en) * | 2020-06-30 | 2020-10-09 | 中国科学院西北生态环境资源研究院 | Data detection management method, device, server and readable storage medium |
| CN111814203A (en) * | 2020-09-08 | 2020-10-23 | 江苏神彩科技股份有限公司 | Method, device and equipment for processing environmental data and storage medium |
| CN112379656A (en) * | 2020-10-09 | 2021-02-19 | 爱普（福建）科技有限公司 | Processing method, device, equipment and medium for detecting abnormal data of industrial system |
Non-Patent Citations (3)
| Title |
|---|
| 乌兰: "环境监测中的异常数据处理研究", 《资源节约与环保》 * |
| 雷敏珊等: "关于环境监测数据中的异常数据分析与处理", 《当代化工研究》 * |
| 马琳等: "基于离群点诊断的污染源在线监测异常数据识别方法探究", 《油气田环境保护》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113901043A (en) * | 2021-12-13 | 2022-01-07 | 西安中电环通数字科技有限公司 | Pollution source intelligent supervision and data fusion analysis method and system |
| CN114567536A (en) * | 2022-02-24 | 2022-05-31 | 北京百度网讯科技有限公司 | Abnormal data processing method and device, electronic equipment and storage medium |
| CN114567536B (en) * | 2022-02-24 | 2024-02-23 | 北京百度网讯科技有限公司 | Abnormal data processing method, device, electronic equipment and storage medium |
| CN114897516A (en) * | 2022-07-12 | 2022-08-12 | 山东乐习信息科技有限公司 | Method for overall process data management of special equipment |
| CN117312295A (en) * | 2023-11-29 | 2023-12-29 | 中国电力科学研究院有限公司 | Method, device, medium and equipment for managing power grid material spot check data |
| CN117312295B (en) * | 2023-11-29 | 2024-03-12 | 中国电力科学研究院有限公司 | Method, device, medium and equipment for managing power grid material spot check data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113469857A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111343161B (en) | Abnormal information processing node analysis method, abnormal information processing node analysis device, abnormal information processing node analysis medium and electronic equipment | |
| US20120102361A1 (en) | Heuristic policy analysis | |
| CN113489713B (en) | Network attack detection method, device, equipment and storage medium | |
| CN110751451B (en) | Laboratory big data management system | |
| TWI734466B (en) | Risk assessment method and device for leakage of privacy data | |
| US11968162B1 (en) | Message content cleansing | |
| CN109241084B (en) | Data query method, terminal equipment and medium | |
| CN110688536A (en) | Label prediction method, device, equipment and storage medium | |
| CN110471912B (en) | Employee attribute information verification method and device and terminal equipment | |
| CN111209153B (en) | Abnormity detection processing method and device and electronic equipment | |
| CN113806370A (en) | Environmental data supervision method, device, equipment and storage medium based on big data | |
| CN114595765A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114443437A (en) | Alarm root cause output method, apparatus, device, medium, and program product | |
| CN109145609B (en) | Data processing method and device | |
| CN110866700A (en) | Method and device for determining enterprise employee information disclosure source | |
| CN110795308A (en) | Server inspection method, device, equipment and storage medium | |
| CN115758389A (en) | Vulnerability processing result checking method and device, electronic equipment and storage medium | |
| CN115016929A (en) | Data processing method, device, equipment and storage medium | |
| CN114785616A (en) | Data risk detection method and device, computer equipment and storage medium | |
| CN115083030A (en) | Service inspection method and device and electronic equipment | |
| KR20140054913A (en) | Apparatus and method for processing data error for distributed system | |
| CN112687030A (en) | Vehicle condition information processing method and device | |
| CN111782967A (en) | Information processing method, information processing device, electronic equipment and computer readable storage medium | |
| CN112989403A (en) | Method, device and equipment for detecting database destruction and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211001 |