CN113225221B - Effective data quality management method for industrial internet industry - Google Patents
Effective data quality management method for industrial internet industry Download PDFInfo
- Publication number
- CN113225221B CN113225221B CN202110370784.7A CN202110370784A CN113225221B CN 113225221 B CN113225221 B CN 113225221B CN 202110370784 A CN202110370784 A CN 202110370784A CN 113225221 B CN113225221 B CN 113225221B
- Authority
- CN
- China
- Prior art keywords
- data
- executing
- collected
- judging whether
- gateway
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
Abstract
The invention provides an effective data quality management method for the industrial internet industry, which comprises the following steps: s1, inputting all gateway MAC addresses, and calculating each data quality judgment index; s2, judging whether point location data of each gateway is completely collected or not, and if all the point location data are collected, executing the step S3; otherwise, informing related personnel to check the problems in the acquisition flow and solve the problems; s3, judging whether the key fields are abnormal or not, and if not, executing the step S4; otherwise, informing related personnel to check the problems in the acquisition flow and solve the problems; and S4, judging that the data acquisition and the data quality are normal. The method is simple and practical, is beneficial to data quality check of data acquired by the data processing platform, and ensures data quality reliability of data acquisition in the industrial internet industry. In addition, the method can play a role in early warning, for example, the gateway can timely find and early warn corresponding personnel to solve problems and prevent data from being lost for a long time when the data is abnormally collected.
Description
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of industrial Internet/Internet of things, in particular to an effective data quality management method for the industrial Internet industry.
[ background of the invention ]
In the existing industrial internet/internet of things industry, data are acquired through different types of industrial gateways according to certain data source formats, types, frequencies and the like; the collected data are transmitted and summarized to a data processing platform through an Internet of things protocol; the data processing platform extracts, converts and loads various data, analyzes and processes the data and stores the data into a target database; the quality of the acquired data is not checked, and the quality of the acquired data cannot be verified.
However, the above technical solutions have certain drawbacks:
1) the data processing platform does not carry out data quality verification on the acquired data, whether the data are complete or not, whether the data meet consistency or not and whether the data are normally acquired or not are judged, and manual inquiry and determination are needed when one gateway is added, so that time and labor are wasted.
2) If the industrial internet platform does not carry out data quality verification, the reliability of the data quality cannot be ensured.
[ summary of the invention ]
The invention aims to solve the problems in the prior art and provides an effective data quality management method for the industrial internet industry, which can be used for carrying out data quality verification on collected data and ensuring data quality reliability.
In order to achieve the purpose, the invention provides an effective data quality management method for the industrial internet industry, which sequentially comprises the following steps:
s1, inputting all gateway MAC addresses, and calculating each data quality judgment index;
s2, judging whether point location data of each gateway is completely collected or not, and if all the point location data are collected, executing the step S3; otherwise, informing related personnel to check the problems in the acquisition flow and solve the problems;
s3, judging whether the key fields are abnormal or not, and if not, executing the step S4; otherwise, informing related personnel to check the problems in the acquisition flow and solve the problems;
and S4, judging that the data acquisition and the data quality are normal.
Preferably, the method specifically comprises the following steps:
s01, inputting all gateway MAC addresses;
s02, calculating the number of the configured point positions of each gateway;
s03, calculating the number of the collected point positions of each gateway;
s04, calculating the number of null values of each key field;
s05, calculating the repeated number of each key field;
s06, judging whether the number of the configured point locations is equal to that of the collected point locations, if so, representing that all point location data are collected, and executing a step S07; otherwise, representing that part of point location data is not collected, and executing the step S010;
s07, judging whether a null value exists in the related key field, if not, executing the step S08; otherwise, executing step S010;
s08, judging whether a key field has a repeated value, if not, executing the step S09; otherwise, executing step S010;
s09, judging whether the data acquisition frequency is normal or not, if not, executing the step S010, otherwise, executing the step S012;
s010, informing related personnel to check and check the problems in the acquisition process and solving the problems;
s011, judging whether the problem is solved or not, and if the problem is solved, returning to execute the step S01; otherwise, re-executing step S010;
and S012, normal data acquisition and normal data quality.
The invention has the beneficial effects that:
1) the method is simple and practical, is beneficial to data quality check of data acquired by the data processing platform, and ensures data quality reliability of data acquisition in the industrial internet industry.
2) The method can play a role in early warning, for example, the gateway can timely find and early warn corresponding personnel to solve problems and prevent data from being lost for a long time when the gateway acquires abnormal data.
The features and advantages of the present invention will be described in detail by embodiments in conjunction with the accompanying drawings.
[ description of the drawings ]
Fig. 1 is a flowchart of an effective data quality management method for the industrial internet industry according to the present invention.
[ detailed description ] embodiments
The invention relates to an effective data quality management method for the industrial internet industry, which is provided by the weak data quality of the industrial internet. The data quality not only relates to the accuracy of a data application presentation result, but also shows the rationality of the whole flow of data acquisition, processing, cleaning and the like. This method is described in detail below.
The invention mainly manages the data quality in the aspect of three dimensional indexes, namely a consistency index, a normative index and an accuracy index, wherein the specific indexes are realized in a program.
A method for managing the quality of effective data in the industrial Internet industry sequentially comprises the following steps:
s1, collecting all gateway MAC addresses;
s2, judging whether point location data of each gateway is completely collected or not, and if all the point location data are collected, executing the step S3; otherwise, informing related personnel to check the problems in the acquisition flow and solve the problems;
s3, judging whether the key fields are abnormal or not, and if not, executing the step S4; otherwise, informing related personnel to check the problems in the acquisition flow and solve the problems;
and S4, judging that the data acquisition and the data quality are normal.
Referring to fig. 1, the method specifically includes the following steps:
1) acquiring all gateway MAC addresses, wherein the gateway addresses have unique identifiers and can represent the uniqueness of each gateway;
2) calculating the configured point location number a1 of each gateway, wherein a1 represents the configured point location number of each gateway;
3) calculating the number a2 of the point locations collected by each gateway, wherein a2 represents the number of the point locations collected by each gateway;
4) calculating the number b of null values of each key field;
5) calculating the repeated number c of each key field;
6) and judging whether the configured point location is equal to the collected point location. If a1 is a2, all the points are collected, and step 7 is executed; if a1 ≠ a2, indicating that the partial point data was not collected, then step 10 is performed.
7) And judging whether each gateway key field has a null value. If b is 0, representing that no null value exists in the key field, executing step 8; if b ≠ 0, it represents a data quality anomaly and step 10 is performed.
8) And judging whether each gateway key field has a repeated value. If c is 0, representing that there is no duplicate value case, then step 9 is executed; if c ≠ 0, it indicates data quality anomaly and step 10 is performed.
9) And judging whether the data acquisition frequency is normal or not. If not, executing step 10; if so, step 12 is performed.
10) Informing relevant personnel to check and check the problems in the acquisition flow and solving the problems;
11) it is determined whether the problem has been resolved. If the problem has been solved, perform step 1; if not, step 10 is re-executed.
12) The data acquisition is normal, and the data quality is normal.
The above embodiments are illustrative of the present invention, and are not intended to limit the present invention, and any simple modifications of the present invention are within the scope of the present invention.
Claims (1)
1. A method for managing the quality of effective data in the industrial Internet industry is characterized in that: the method sequentially comprises the following steps:
s1, inputting all gateway MAC addresses, and calculating each data quality judgment index;
s2, judging whether point location data of each gateway is completely collected or not, and if all the point location data are collected, executing the step S3; otherwise, informing related personnel to check the problems in the acquisition flow and solve the problems;
s3, judging whether the key fields are abnormal or not, and if not, executing the step S4; otherwise, informing related personnel to check the problems in the acquisition flow and solve the problems;
s4, judging that data acquisition and data quality are normal;
the method specifically comprises the following steps:
s01, inputting all gateway MAC addresses;
s02, calculating the number of the configured point positions of each gateway;
s03, calculating the number of the collected point positions of each gateway;
s04, calculating the number of null values of each key field;
s05, calculating the repeated number of each key field;
s06, judging whether the number of the configured point locations is equal to that of the collected point locations, if so, representing that all point location data are collected, and executing a step S07; otherwise, representing that part of point location data is not collected, and executing the step S010;
s07, judging whether a null value exists in the related key field, if not, executing the step S08; otherwise, executing step S010;
s08, judging whether a key field has a repeated value, if not, executing the step S09; otherwise, executing step S010;
s09, judging whether the data acquisition frequency is normal or not, if not, executing the step S010, otherwise, executing the step S012;
s010, informing related personnel to check and check the problems in the acquisition process and solving the problems;
s011, judging whether the problem is solved or not, and if the problem is solved, returning to execute the step S01; otherwise, re-executing step S010;
and S012, normal data acquisition and normal data quality.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110370784.7A CN113225221B (en) | 2021-04-07 | 2021-04-07 | Effective data quality management method for industrial internet industry |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110370784.7A CN113225221B (en) | 2021-04-07 | 2021-04-07 | Effective data quality management method for industrial internet industry |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113225221A CN113225221A (en) | 2021-08-06 |
| CN113225221B true CN113225221B (en) | 2022-08-05 |
Family
ID=77086473
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110370784.7A Active CN113225221B (en) | 2021-04-07 | 2021-04-07 | Effective data quality management method for industrial internet industry |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113225221B (en) |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100574224C (en) * | 2007-03-07 | 2009-12-23 | 中控科技集团有限公司 | The detection method of Industrial Ethernet data monitoring and device |
| US20180091390A1 (en) * | 2016-09-27 | 2018-03-29 | Ca, Inc. | Data validation across monitoring systems |
| CN110495138B (en) * | 2017-05-31 | 2023-09-29 | 西门子股份公司 | Industrial control system and monitoring method for network security thereof |
| CN108282026A (en) * | 2017-12-27 | 2018-07-13 | 河南平高电气股份有限公司 | A kind of high-tension switch gear novel maintenance system |
| CN109656913B (en) * | 2018-12-20 | 2022-12-27 | 江苏昂内斯电力科技股份有限公司 | Data acquisition abnormity recruitment method based on Internet of things |
| CN109857732A (en) * | 2019-01-31 | 2019-06-07 | 山东省电子信息产品检验院 | A kind of industry internet platform monitoring data transmission switching method and system |
| CN109981402A (en) * | 2019-03-05 | 2019-07-05 | 山东浪潮云信息技术有限公司 | The real-time detection and appraisal procedure and system of a kind of data collection effect |
| CN110336703A (en) * | 2019-07-12 | 2019-10-15 | 河海大学常州校区 | Industrial big data based on edge calculations monitors system |
| CN110941677B (en) * | 2019-11-27 | 2023-04-18 | 上海西码智能科技股份有限公司 | Gateway device, control method and Internet of things control system based on edge calculation |
| CN111047431A (en) * | 2019-12-11 | 2020-04-21 | 深圳微众信用科技股份有限公司 | Credit service processing device, method and equipment based on big data |
| CN112235159B (en) * | 2020-10-13 | 2022-05-10 | 中移（杭州）信息技术有限公司 | Gateway quality portrait generation method, system, network equipment and storage medium |
| CN112506097A (en) * | 2020-11-27 | 2021-03-16 | 江苏科技大学 | Jig frame remote monitoring system and method based on industrial internet |
-
2021
- 2021-04-07 CN CN202110370784.7A patent/CN113225221B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113225221A (en) | 2021-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP2008139995A5 (en) | ||
| CN113225221B (en) | Effective data quality management method for industrial internet industry | |
| CN111782456B (en) | Anomaly detection method, device, computer equipment and storage medium | |
| CN111049882B (en) | Cache state processing system, method, device and computer readable storage medium | |
| CN113806343B (en) | Evaluation method and system for Internet of vehicles data quality | |
| CN113612567B (en) | Alignment method and device for data collected by multiple sensors of equipment and electronic equipment | |
| CN117061170B (en) | Intelligent manufacturing industry big data analysis method based on feature selection | |
| CN117312290A (en) | Method for improving heterogeneous system data quality | |
| CN111754334A (en) | Vehicle mortgage risk early warning method and device | |
| KR20180130630A (en) | Vulnerability diagnosing and managing system and method of information system using automatic diagnosis tool | |
| CN109688236B (en) | Sinkhole domain name processing method and server | |
| CN114866546B (en) | PaaS-based one-stop management system for monitoring platform | |
| CN112491584B (en) | Service operation safety condition judgment method and device, electronic medium and storage medium | |
| CN112990711B (en) | Aluminum alloy template construction monitoring method and system based on site construction | |
| CN112291085B (en) | Fault positioning method, device, equipment and medium | |
| CN112687030A (en) | Vehicle condition information processing method and device | |
| CN113986990A (en) | Data resource acquisition and labeling method and device based on block chain data mining | |
| CN113360359A (en) | Index abnormal data tracing method, device, equipment and storage medium | |
| CN112819071A (en) | Fault information clustering method and device, computer equipment and storage medium | |
| CN113572628A (en) | Data association method and device, computing equipment and computer storage medium | |
| CN112579559A (en) | Key value pair management verification method, device, equipment and storage medium | |
| CN112148459B (en) | Processing method, device, readable medium and equipment for node association data | |
| CN110968862B (en) | Data anomaly detection method and terminal | |
| CN111209130B (en) | Fault processing method, system, equipment and medium based on MySQL master-slave replication cluster | |
| CN117171740A (en) | Method and device for determining health degree of open source assembly |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |