CN114564469A - Method and system for processing collected data - Google Patents
Method and system for processing collected data Download PDFInfo
- Publication number
- CN114564469A CN114564469A CN202210162584.7A CN202210162584A CN114564469A CN 114564469 A CN114564469 A CN 114564469A CN 202210162584 A CN202210162584 A CN 202210162584A CN 114564469 A CN114564469 A CN 114564469A
- Authority
- CN
- China
- Prior art keywords
- data
- abnormal
- acquired data
- parameters
- judgment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Abstract
The invention discloses a method and a system for processing collected data. The processing method of the collected data comprises the following steps: carrying out preliminary abnormity judgment on the acquired data of the parameters according to a preset judgment rule; if the initial abnormality is judged to be normal, configuring a corresponding threshold value for the parameter according to the change rule of the acquired data of the parameter; and carrying out secondary abnormity judgment on the acquired data of the parameters based on the threshold values corresponding to the parameters. The invention can quickly detect obvious abnormal data and can ensure the accuracy of the acquired data.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method and a system for judging and processing abnormity of acquired data.
Background
With the advent of the big data era, data acquisition is a basic operation of all devices, abnormal data may exist in acquired data of any parameter of the devices, and in some cases, the abnormal data may cause an inaccurate analysis result of the acquired data, so that the abnormal data in the acquired data needs to be accurately judged and processed.
Taking the existing energy management system as an example, data plays an important role, and errors in collected data (i.e., collected data of parameters) will cause inaccurate statistical data of the whole system, so that users cannot correctly know energy data of equipment or the whole plant.
Therefore, how to quickly, effectively and accurately determine the abnormal data is an urgent technical problem to be solved in the industry.
Disclosure of Invention
The invention provides a method and a system for processing acquired data, and aims to solve the technical problems that in the prior art, the judgment is not accurate or not fast enough due to an abnormal data judgment method.
The processing method of the acquired data provided by the invention comprises the following steps:
carrying out preliminary abnormity judgment on the acquired data of the parameters according to a preset judgment rule;
if the initial abnormality is judged to be normal, configuring a corresponding threshold value for the parameter according to the change rule of the acquired data of the parameter;
and carrying out anomaly judgment again on the acquired data of the parameters based on the threshold values corresponding to the parameters.
Further, the determination rule is preset based on classifying the type of the parameter or the type of the device to which the parameter belongs, or the determination rule is preset according to the characteristic of the parameter or the characteristic of the device to which the parameter belongs.
Further, the corresponding threshold is configured according to the change rule of the acquired data of the parameters and is configured to be a constant threshold when the data has no trend.
Further, the corresponding threshold is configured according to the change rule of the acquired data of the parameters, and when the acquired data has trend, the corresponding threshold is dynamically configured according to the trend of the acquired data.
Further, still include: if the abnormal condition is judged to be normal again, corresponding difference operation is carried out on the acquired data of the parameters, and if the obtained difference sequence is stable, the threshold value corresponding to the parameters is updated;
and judging upgrading abnormity of the acquired data of the parameters based on the updated threshold values of the parameters.
Further, the difference operation is a first order difference operation.
Further, still include: and prompting the acquired data which is judged to be abnormal to a user, and deleting the abnormal acquired data after the user determines.
The processing system for the acquired data, which is provided by the invention, adopts the processing method for the acquired data of the technical scheme to process the acquired data of the parameters, and comprises the following steps:
the conventional abnormal value judgment module is used for carrying out the preliminary abnormal judgment on the acquired data of the parameters;
and the special abnormal value judgment module is used for carrying out the repeated abnormal judgment on the collected data of the parameters or carrying out other abnormal judgment after carrying out the repeated judgment on the collected data of the parameters.
Further, the other anomaly determination includes an escalation anomaly determination.
Further, still include: and the abnormal data processing module is used for prompting the acquired data which is judged to be abnormal to a user and deleting the abnormal data after the user determines the acquired data.
The invention divides the detection of data abnormity into a plurality of stages, quickly detects obvious abnormal data through preliminary abnormity judgment, and further ensures the accuracy of the acquired data through secondary abnormity judgment and other abnormity judgment. The method can quickly and accurately detect the acquired abnormal data, and avoids data statistics and data analysis errors caused by the acquired abnormal data, thereby ensuring the reliability of system data and simultaneously ensuring the cleanliness of the database table.
Drawings
The invention is described in detail below with reference to embodiments and the attached drawings, wherein:
FIG. 1 is a flow chart of a method according to an embodiment of the present invention.
FIG. 2 is a flow chart of an embodiment of a preliminary anomaly determination of the present invention.
FIG. 3 is a flow chart of one embodiment of a re-anomaly determination of the present invention.
FIG. 4 is a block diagram of an embodiment of the system of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects to be solved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.
Thus, a feature indicated in this specification will serve to explain one of the features of one embodiment of the invention, and does not imply that every embodiment of the invention must have the stated feature. Further, it should be noted that this specification describes many features. Although some features may be combined to show a possible system design, these features may also be used in other combinations not explicitly described. Thus, the combinations illustrated are not intended to be limiting unless otherwise specified.
As shown in fig. 1 to 3, the processing method of the collected data of the present invention determines an abnormality quickly by performing progressive abnormality determination in stages, and performs accurate determination after the quick determination.
In one embodiment, the method performs preliminary abnormal judgment on the collected data of the parameters according to a preset judgment rule. The determination rule may be preset based on classifying the type of the parameter or according to the type of the device to which the parameter belongs, or the determination rule may be preset according to the characteristic of the parameter or the characteristic of the device to which the parameter belongs. For example, when the parameter is the electricity consumption of the commercial power, the acquired data of the parameter cannot be negative, and a judgment rule can be preset at the moment, so that some obvious abnormal data can be quickly screened out, and the obvious abnormal data can be prevented from increasing subsequent workload.
If the acquired data is judged to be normal after the initial abnormity judgment in the steps, configuring a corresponding threshold value for the parameter according to the change rule of the acquired data of the parameter, and carrying out abnormity judgment again on the acquired data of the parameter based on the threshold value corresponding to the parameter.
In one embodiment, the threshold corresponding to the change rule configuration of the acquired data of the parameter is configured to be a constant threshold when the data is non-trending, and for the constant threshold, for example, the constant threshold configured at the time T is constant for the historical data, and at the time T +1, the constant threshold is updated by the newly added data. In another embodiment, the corresponding threshold is configured according to the change rule of the acquired data of the parameters, and when the acquired data has trend, the corresponding threshold is dynamically configured according to the trend of the acquired data. For dynamic thresholds, a moving average and a moving standard deviation are calculated based on a fixed moving window, and upper and lower monitored boundaries are given based on the moving average and the moving standard deviation, for example, during the time period T1-T2, the collected data shows a rapidly rising change trend, and then the time period T1-T2 can be divided into a plurality of intervals, each interval has a corresponding threshold, and the thresholds are changed according to time change. In the time period T3-T4, if the acquired data tends to be stable, a constant threshold may be configured at this time, and the constant threshold may be a constant value or a threshold range in which the upper and lower limits are constant.
If the acquired data is still normal after being judged to be abnormal again in the steps, corresponding differential operation is carried out on the acquired data of the parameters, if the obtained differential sequence is stable (if the differential sequence is in a stable range), a corresponding differential threshold value is configured for the differential sequence, after the acquired data is judged to be normal, the differential sequence is calculated, and then whether the acquired data is abnormal or not is judged based on the differential threshold value so as to ensure the accuracy of the acquired data.
In one embodiment, the difference operation is a first order difference operation, and if the difference sequence is stable, a constant threshold value can be configured for the difference sequence, so that the abnormal condition of the collected data can be judged.
In the above process, when there is the acquired data determined to be abnormal, the acquired data determined to be abnormal is presented to the user, and then the user can finally determine whether the corresponding abnormal data needs to be retained, and if the user determines that the corresponding abnormal data does not need to be retained, the user deletes the abnormal acquired data after determining.
As shown in fig. 4, the processing system for collecting data of the present invention at least includes: a general abnormal value judging module and a special abnormal value judging module. The processing system for the acquired data adopts the processing method for the acquired data in the technical scheme to process the acquired data of the parameters.
The conventional abnormal value judgment module is used for carrying out the preliminary abnormal judgment on the collected data of the parameters.
The special abnormal value judgment module is used for carrying out the above secondary abnormal judgment on the collected data of the parameters or carrying out other abnormal judgment after carrying out the above secondary judgment on the collected data of the parameters. Such as upgrade exception determination.
In a further embodiment, the processing system for collecting data of the present invention further comprises an exception data processing module. The abnormal data processing module is used for prompting the collected data which are judged to be abnormal to a user and deleting the abnormal data after the user determines the abnormal data.
The embodiment shown in fig. 4 is specifically applied to an energy device, and the energy device data acquisition module is used for acquiring the acquired data. Then, the conventional abnormal value judging module is used for quickly judging abnormal data, and then the special abnormal value is used for accurately judging the abnormal data, and the abnormal data obtained by the two modules (namely the conventional abnormal value judging module and the special abnormal value judging module) are processed correspondingly by the abnormal data processing module.
The energy equipment data acquisition module acquires data of bottom equipment in the process, can report the corresponding acquired data to the server, and stores the data in the database. The judgment rules preset in the conventional abnormal value judgment module may form a knowledge base system, and all the judgment rules are stored by the knowledge base system. The judgment rules are dynamically added into the knowledge base, and the corresponding rules can be manually added into the knowledge base through a page by a user or can be obtained according to other information such as historical alarm record conclusions and the like and dynamically added into the knowledge base. And the acquired data is preferentially matched with the knowledge base, and abnormal data is detected according to the judgment rule in the knowledge base. Each collected and reported data has corresponding identifier such as equipment type, equipment gateway, equipment address or equipment ID, and corresponding rule in the knowledge base can be selected through the data type or the equipment ID
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. A method for processing collected data, comprising:
carrying out preliminary abnormity judgment on the acquired data of the parameters according to a preset judgment rule;
if the initial abnormality is judged to be normal, configuring a corresponding threshold value for the parameter according to the change rule of the acquired data of the parameter;
and carrying out anomaly judgment again on the acquired data of the parameters based on the threshold values corresponding to the parameters.
2. The collected data processing method according to claim 1, wherein the determination rule is preset based on classification of a type of the parameter or according to a type of a device to which the parameter belongs, or the determination rule is preset according to a characteristic of the parameter or a characteristic of a device to which the parameter belongs.
3. The data processing method as claimed in claim 1, wherein the threshold value corresponding to the change rule configuration of the acquired data according to the parameter is configured as a constant threshold value when the data is non-trending.
4. The method according to claim 1, wherein the threshold value is dynamically configured according to the trend of the collected data when the collected data has a trend.
5. The acquired data processing method according to any one of claims 1 to 4, further comprising:
if the abnormal condition is judged to be normal again, corresponding difference operation is carried out on the acquired data of the parameters, and if the obtained difference sequence is stable, the threshold value corresponding to the parameters is updated;
and judging upgrading abnormity of the acquired data of the parameters based on the updated threshold values of the parameters.
6. The method of processing collected data as set forth in claim 5, wherein the difference operation is a first order difference operation.
7. The method for processing the collected data according to claim 5, further comprising: and prompting the acquired data which is judged to be abnormal to a user, and deleting the abnormal acquired data after the user determines.
8. A processing system for processing collected data of a parameter by the method for processing collected data according to any one of claims 1 to 7, comprising:
the conventional abnormal value judgment module is used for carrying out the preliminary abnormal judgment on the acquired data of the parameters;
and the special abnormal value judgment module is used for carrying out the repeated abnormal judgment on the collected data of the parameters or carrying out other abnormal judgment after carrying out the repeated judgment on the collected data of the parameters.
9. The data collection processing system of claim 8, wherein the other exception determinations comprise escalation exception determinations.
10. The data acquisition processing system as in claim 8, further comprising: and the abnormal data processing module is used for prompting the acquired data which is judged to be abnormal to a user and deleting the abnormal data after the user determines the acquired data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210162584.7A CN114564469A (en) | 2022-02-22 | 2022-02-22 | Method and system for processing collected data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210162584.7A CN114564469A (en) | 2022-02-22 | 2022-02-22 | Method and system for processing collected data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114564469A true CN114564469A (en) | 2022-05-31 |
Family
ID=81714750
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210162584.7A Pending CN114564469A (en) | 2022-02-22 | 2022-02-22 | Method and system for processing collected data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114564469A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116819318A (en) * | 2023-07-04 | 2023-09-29 | 赫义博自动化科技(江苏)有限公司 | Motor fault detection method and system |
-
2022
- 2022-02-22 CN CN202210162584.7A patent/CN114564469A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116819318A (en) * | 2023-07-04 | 2023-09-29 | 赫义博自动化科技(江苏)有限公司 | Motor fault detection method and system |
| CN116819318B (en) * | 2023-07-04 | 2024-01-12 | 赫义博自动化科技(江苏)有限公司 | Motor fault detection method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102117730B (en) | Method for processing parameter data of machine station in manufacturing process of semiconductor and device thereof | |
| CN111126824B (en) | Multi-index correlation model training method and multi-index anomaly analysis method | |
| US9547545B2 (en) | Apparatus and program for detecting abnormality of a system | |
| TWI459011B (en) | Method and system for determing status of machine and computer readable storage medium for storing the method | |
| CN115576738B (en) | Method and system for realizing equipment fault determination based on chip analysis | |
| CN102801548B (en) | A kind of method of intelligent early-warning, device and information system | |
| CN114564469A (en) | Method and system for processing collected data | |
| CN112612680A (en) | Message warning method, system, computer equipment and storage medium | |
| CN108399115B (en) | Operation and maintenance operation detection method and device and electronic equipment | |
| CN113077065A (en) | Method, device and equipment for processing faults of vehicle production line and storage medium | |
| CN113810792B (en) | Edge data acquisition and analysis system based on cloud computing | |
| CN114610553A (en) | Intelligent equipment detection system and detection method thereof | |
| CN114189348A (en) | Asset identification method suitable for industrial control network environment | |
| CN108063764B (en) | Network traffic processing method and device | |
| CN113010394A (en) | Machine room fault detection method for data center | |
| CN109062801B (en) | Automatic conversion system for software interface | |
| CN111176950A (en) | Method and equipment for monitoring network card of server cluster | |
| CN115295016A (en) | Equipment running state monitoring method, device, equipment and storage medium | |
| CN114500178B (en) | Self-operation intelligent Internet of things gateway | |
| CN113626236B (en) | Fault diagnosis method, device, equipment and medium for distributed file system | |
| JP2001217169A (en) | Data variation monitoring method and monitoring device | |
| JPWO2018122889A1 (en) | Anomaly detection method, system and program | |
| CN111488272A (en) | Automatic testing method and device for complex transaction files | |
| CN112714288A (en) | Intelligent monitoring method and device and monitoring equipment | |
| CN115858505A (en) | Data processing method for identifying abnormal state of frequency converter |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |