CN115062348A - Electric power data preprocessing system and method - Google Patents
Electric power data preprocessing system and method Download PDFInfo
- Publication number
- CN115062348A CN115062348A CN202210995607.2A CN202210995607A CN115062348A CN 115062348 A CN115062348 A CN 115062348A CN 202210995607 A CN202210995607 A CN 202210995607A CN 115062348 A CN115062348 A CN 115062348A
- Authority
- CN
- China
- Prior art keywords
- data
- privacy
- power data
- electric power
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000007781 pre-processing Methods 0.000 title claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 42
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/177—Editing, e.g. inserting or deleting of tables; using ruled lines
- G06F40/18—Editing, e.g. inserting or deleting of tables; using ruled lines of spreadsheets
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/186—Templates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Bioethics (AREA)
- Economics (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Human Resources & Organizations (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Medical Informatics (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- Computer Security & Cryptography (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses a power data preprocessing system and method, mainly relates to the technical field of data preprocessing, and is used for solving the problems that existing user privacy data are easy to leak, a multi-source power data processing process is complex and the like. The method comprises the following steps: the type processing module is used for determining the private data position and the private data type corresponding to the electric power data table; the privacy removing module is used for determining privacy data processing rules corresponding to the electric power data table; determining a privacy data processing algorithm corresponding to the electric power data table; to obtain a de-privacy power data table; and the data unification module is used for converting the format of the privacy-removed power data into the format of the header data of the preset template. According to the method, the private data are encrypted and the data format is unified.
Description
Technical Field
The present application relates to the field of data preprocessing technologies, and in particular, to a system and a method for preprocessing electric power data.
Background
The power data refers to data generated in the power production process, and because the power data has the characteristics of large data volume, multiple data types, high privacy degree of user data and the like, how to efficiently and deeply process the power data to further extract valuable information and complete monitoring of the power data is a challenging problem.
At the present stage, the preprocessing method of the power data is mainly to perform manual processing on the power data by operation and maintenance staff, or detect whether the power data is abnormal through a neural network, so as to remove the abnormal power data.
However, since the power data relates to more user privacy data, such as information of a power user name identification card, the existing method is very easy to cause data leakage. In addition, the number of devices for generating the power data is large, and the devices correspond to the power data in various formats, so that the neural network cannot be perfectly adapted to the data in each format, and the multi-source power data processing process is complex.
Disclosure of Invention
In view of the above-mentioned deficiencies of the prior art, the present invention provides a power data monitoring system and method to solve the above-mentioned technical problems.
In a first aspect, the present application provides a power data preprocessing system, the system comprising: the type processing module is used for reading header data of the electric power data tables uploaded by each electric power device, and further determining the private data position and the private data type corresponding to each electric power data table; the privacy removing module is used for determining a privacy data processing rule corresponding to the electric power data table according to the privacy data type; determining a privacy data processing algorithm corresponding to the electric power data table based on the privacy data position and the privacy data processing rule; to obtain a de-privacy power data table; the data unifying module is used for determining whether each data cell in the header data exists in a template cell of the preset template header data; when the template cell consistent with the data cell does not exist, adding the data cell into the preset template header data; and converting the format of the privacy-removed power data into the format with the preset template header data.
Further, the type processing module further comprises a privacy verification unit; the privacy verification unit is used for extracting keywords from the header data and determining the privacy data type corresponding to the privacy data in the electric power data table based on the keywords and the keyword-type database; the privacy data type comprises an operable type and an inoperable type.
Further, the privacy removal module further comprises: an operational privacy unit; the operable privacy unit is used for acquiring first electric power data of which the privacy data type is an operable type in the electric power data table; randomly extracting any privacy encryption rule from a preset privacy encryption rule base, and encrypting the first power data; and marking the preset encryption number of the privacy encryption rule into the header data of the first power data.
Further, the privacy elimination module further comprises: a non-calculable privacy unit; the non-calculation privacy unit is used for acquiring second electric power data of which the privacy data type is a non-calculation type in the electric power data table; and performing mask processing on the second power data through a preset mask algorithm.
In a second aspect, the present application provides a power data preprocessing method, including: reading header data of the electric power data tables uploaded by each electric power device, and further determining the private data position and the private data type corresponding to each electric power data table; determining a privacy data processing rule corresponding to the electric power data table according to the privacy data type; determining a privacy data processing algorithm corresponding to the electric power data table based on the privacy data position and the privacy data processing rule; to obtain a de-privacy power data table; determining whether each data cell in the header data exists in a template cell of preset template header data; when the template cell consistent with the data cell does not exist, adding the data cell into the preset template header data; and converting the format of the privacy-removed power data into a format with preset template header data.
Further, determining a private data position and a private data type corresponding to the power data table specifically includes: extracting keywords from the header data, and determining the privacy data type corresponding to the privacy data in the electric power data table based on the keywords and the keyword-type database; the privacy data type comprises an operable type and an inoperable type.
Further, prior to obtaining the de-privacy power data table, the method further comprises: acquiring first power data of which the privacy data type is an operable type in a power data table; randomly extracting any privacy encryption rule from a preset privacy encryption rule base, and encrypting the first power data; and marking the preset encryption number of the privacy encryption rule into the header data of the first power data.
Further, prior to obtaining the de-privacy power data table, the method further comprises: acquiring second electric power data with the type of the private data being an inoperable type in the electric power data table; and performing mask processing on the second power data through a preset mask algorithm.
As can be appreciated by those skilled in the art, the present invention has at least the following beneficial effects:
the classification of the private data is realized through the type processing module; the privacy data is subjected to privacy removal operation through the privacy removal module, and the risk that the privacy is revealed due to the fact that the follow-up data are revealed is avoided. Through the data unification module, the format unification of the multi-source power data is realized.
Drawings
Some embodiments of the disclosure are described below with reference to the accompanying drawings, in which:
fig. 1 is a schematic diagram of an internal structure of an electric power data preprocessing system according to an embodiment of the present application.
Fig. 2 is a flowchart of a power data preprocessing method according to an embodiment of the present disclosure.
Detailed Description
It should be understood by those skilled in the art that the embodiments described below are only preferred embodiments of the present disclosure, and do not mean that the present disclosure can be implemented only by the preferred embodiments, which are merely for explaining the technical principles of the present disclosure and are not intended to limit the scope of the present disclosure. All other embodiments that can be derived by one of ordinary skill in the art from the preferred embodiments provided by the disclosure and that fall within the scope of the disclosure are intended to be encompassed by the present disclosure without any inventive step.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Fig. 1 is a power data preprocessing system according to an embodiment of the present disclosure. As shown in fig. 1, the system provided in the embodiment of the present application mainly includes a type processing module, a privacy removing module, and a unifying module.
After the power data preprocessing is started, the received power data is firstly sent to the type processing module. The type processing module is any feasible device or device capable of positioning the private data and classifying the private data.
Specifically, the type processing module reads header data of the electric power data table uploaded by each electric power device, and further determines a private data position and a private data type corresponding to the electric power data table.
It should be noted that the header data is composed of a plurality of table cells, and each table cell is used to indicate the data type of the column in which the table cell is located. For example: when the table cell is "name", the column is used to show name data in the power data.
The specific process of "determining the private data location and the private data type corresponding to the power data table" may be: extracting keywords from header data through a privacy verification unit in the type processing module, and determining a privacy data type corresponding to the privacy data in the electric power data table based on the keywords and a keyword-type database; the privacy data type comprises an operable type and an inoperable type. It should be noted that the keyword-type database is used to store the corresponding relationship between the keywords and the data types, and the specific content can be obtained through many experiments by those skilled in the art.
After determining the power data of the private data type, the power data is de-private by the de-privacy module. Specifically, the privacy removing module determines a privacy data processing rule corresponding to the electric power data table according to the privacy data type; determining a privacy data processing algorithm corresponding to the electric power data table based on the privacy data position and the privacy data processing rule; to obtain a de-privacy power data table. It should be noted that the private data processing rule is any feasible rule for processing the private data, and those skilled in the art can determine the specific content according to actual needs.
In addition, since some of the private data needs to participate in subsequent calculations, the private data is divided into operable private data and inoperable private data. After privacy of the operable privacy data is removed, the encryption rule is marked so as to facilitate decryption operation during subsequent calculation.
As an example one, the privacy removing module acquires first power data of which the privacy data type is an operable type in the power data table through the operable privacy unit; randomly extracting any privacy encryption rule from a preset privacy encryption rule base, and encrypting the first power data; and marking the preset encryption number of the privacy encryption rule into the header data of the first power data. It should be noted that the preset privacy encryption rule base is used for storing a plurality of existing privacy encryption rules.
As an example place, the privacy removing module acquires second electric power data of which the privacy data type is an inoperable type in the electric power data table through the inoperable privacy unit; and performing mask processing on the second power data through a preset mask algorithm.
The data unifying module is used for determining whether each data cell in the header data exists in a template cell of the preset template header data; when the template cell consistent with the data cell does not exist, adding the data cell into the preset template header data; and converting the format of the privacy-removed power data into a format with preset template header data.
In addition, an embodiment of the present application further provides a power data preprocessing method, as shown in fig. 2, the method provided in the embodiment of the present application mainly includes the following steps:
step 210, reading header data of the electric power data table uploaded by each electric power device, and further determining a private data position and a private data type corresponding to the electric power data table.
Determining a private data position and a private data type corresponding to the electric power data table, specifically comprising: extracting keywords from the header data, and determining the privacy data type corresponding to the privacy data in the electric power data table based on the keywords and the keyword-type database; the privacy data type comprises an operable type and an inoperable type.
Step 220, determining a privacy data processing rule corresponding to the electric power data table according to the privacy data type; determining a privacy data processing algorithm corresponding to the electric power data table based on the privacy data position and the privacy data processing rule; to obtain a de-privacy power data table.
As an example one, before obtaining the de-privacy power data table, the method further comprises: acquiring first electric power data with the type of the private data being an operable type in an electric power data table; randomly extracting any privacy encryption rule from a preset privacy encryption rule base, and encrypting the first power data; and marking the preset encryption number of the privacy encryption rule into the header data of the first power data.
As an example ground, prior to obtaining the de-privacy electric power data table, the method further includes: acquiring second electric power data with the type of the private data being an inoperable type in the electric power data table; and performing mask processing on the second power data through a preset mask algorithm.
Step 230, determining whether each data cell in the header data exists in a template cell of preset template header data; when the template cell consistent with the data cell does not exist, adding the data cell into the preset template header data; and converting the format of the privacy-removed power data into a format with preset template header data.
So far, the technical solutions of the present disclosure have been described in connection with the foregoing embodiments, but it is easily understood by those skilled in the art that the scope of the present disclosure is not limited to only these specific embodiments. The technical solutions in the above embodiments can be split and combined, and equivalent changes or substitutions can be made on related technical features by those skilled in the art without departing from the technical principles of the present disclosure, and any changes, equivalents, improvements, and the like made within the technical concept and/or technical principles of the present disclosure will fall within the protection scope of the present disclosure.
Claims (8)
1. A power data preprocessing system, the system comprising:
the type processing module is used for reading header data of the electric power data tables uploaded by each electric power device, and further determining the private data position and the private data type corresponding to each electric power data table;
the privacy removing module is used for determining a privacy data processing rule corresponding to the electric power data table according to the privacy data type; determining a privacy data processing algorithm corresponding to the electric power data table based on the privacy data position and the privacy data processing rule; to obtain a de-privacy power data table;
the data unifying module is used for determining whether each data cell in the header data exists in a template cell of preset template header data; when the template cell consistent with the data cell does not exist, adding the data cell into preset template header data; and converting the format of the privacy-removed power data into a format with preset template header data.
2. The power data preprocessing system of claim 1 wherein the type processing module further comprises a privacy check unit;
the privacy verification unit is used for extracting keywords from header data and determining the privacy data type corresponding to the privacy data in the electric power data table based on the keywords and the keyword-type database; wherein the private data types include an operable type and an inoperable type.
3. The power data pre-processing system of claim 2, wherein the de-privacy module further comprises: an operational privacy unit;
the operable privacy unit is used for acquiring first electric power data of which the privacy data type is an operable type in the electric power data table; randomly extracting any privacy encryption rule from a preset privacy encryption rule base, and encrypting the first power data; and marking the preset encryption number of the privacy encryption rule into header data of the first electric power data.
4. The power data preprocessing system of claim 2, wherein the de-privacy module further comprises: a non-calculable privacy unit;
the non-calculation privacy unit is used for acquiring second electric power data of which the privacy data type is a non-calculation type in the electric power data table; and performing mask processing on the second power data through a preset mask algorithm.
5. A method of pre-processing power data, the method comprising:
reading header data of the electric power data tables uploaded by each electric power device, and further determining the private data position and the private data type corresponding to each electric power data table;
determining a privacy data processing rule corresponding to the electric power data table according to the privacy data type; determining a privacy data processing algorithm corresponding to the electric power data table based on the privacy data position and the privacy data processing rule; to obtain a de-privacy power data table;
determining whether each data cell in the header data exists in a template cell of preset template header data; when the template cell consistent with the data cell does not exist, adding the data cell into preset template header data; and converting the format of the privacy-removed power data into a format with preset template header data.
6. The power data preprocessing method according to claim 5, wherein determining the private data position and the private data type corresponding to the power data table specifically comprises:
extracting keywords from header data, and determining a privacy data type corresponding to the privacy data in the electric power data table based on the keywords and a keyword-type database; wherein the private data types include an operable type and an inoperable type.
7. The power data preprocessing method of claim 6, wherein prior to obtaining the de-privacy power data table, the method further comprises:
acquiring first electric power data with the type of the private data being an operable type in an electric power data table; randomly extracting any privacy encryption rule from a preset privacy encryption rule base, and encrypting the first power data; and marking the preset encryption number of the privacy encryption rule into header data of the first power data.
8. The power data preprocessing method of claim 6, wherein prior to obtaining the de-privacy power data table, the method further comprises:
acquiring second electric power data with the type of the private data being an inoperable type in the electric power data table; and performing mask processing on the second power data through a preset mask algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210995607.2A CN115062348A (en) | 2022-08-19 | 2022-08-19 | Electric power data preprocessing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210995607.2A CN115062348A (en) | 2022-08-19 | 2022-08-19 | Electric power data preprocessing system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115062348A true CN115062348A (en) | 2022-09-16 |
Family
ID=83207636
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210995607.2A Pending CN115062348A (en) | 2022-08-19 | 2022-08-19 | Electric power data preprocessing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115062348A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115460018A (en) * | 2022-11-09 | 2022-12-09 | 国网山东省电力公司阳信县供电公司 | Electric power data preprocessing method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120005720A1 (en) * | 2010-07-01 | 2012-01-05 | International Business Machines Corporation | Categorization Of Privacy Data And Data Flow Detection With Rules Engine To Detect Privacy Breaches |
| CN110795761A (en) * | 2019-10-29 | 2020-02-14 | 国网山东省电力公司信息通信公司 | Dynamic desensitization method for sensitive data of ubiquitous power Internet of things |
| CN112861169A (en) * | 2021-01-14 | 2021-05-28 | 支付宝(杭州)信息技术有限公司 | Data processing method, device and equipment based on privacy protection |
| CN114218318A (en) * | 2022-02-21 | 2022-03-22 | 国网山东省电力公司乳山市供电公司 | Data processing system and method for electric power big data |
-
2022
- 2022-08-19 CN CN202210995607.2A patent/CN115062348A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120005720A1 (en) * | 2010-07-01 | 2012-01-05 | International Business Machines Corporation | Categorization Of Privacy Data And Data Flow Detection With Rules Engine To Detect Privacy Breaches |
| CN110795761A (en) * | 2019-10-29 | 2020-02-14 | 国网山东省电力公司信息通信公司 | Dynamic desensitization method for sensitive data of ubiquitous power Internet of things |
| CN112861169A (en) * | 2021-01-14 | 2021-05-28 | 支付宝(杭州)信息技术有限公司 | Data processing method, device and equipment based on privacy protection |
| CN114218318A (en) * | 2022-02-21 | 2022-03-22 | 国网山东省电力公司乳山市供电公司 | Data processing system and method for electric power big data |
Non-Patent Citations (4)
| Title |
|---|
| 彭婧等: "电力数据脱敏安全防护体系", 《计算机应用》 * |
| 朱克等: "电力用户信息脱敏研究", 《电脑知识与技术》 * |
| 胡荣磊等: "一种大数据环境下医疗隐私保护方案设计与实现", 《信息网络安全》 * |
| 黄凌宇等: "电力大数据应用建设中的重要数据保护", 《2018智能电网信息化建设研讨会论文集》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115460018A (en) * | 2022-11-09 | 2022-12-09 | 国网山东省电力公司阳信县供电公司 | Electric power data preprocessing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107239666B (en) | Method and system for desensitizing medical image data | |
| Zhao et al. | Robust hashing for image authentication using Zernike moments and local features | |
| CN104486075B (en) | A kind of verification method of intelligent substation ICD model file digital signature | |
| CN102684927B (en) | A kind of method and apparatus of patrolling and examining the network equipment | |
| CN106951796B (en) | Desensitization method and device for data privacy protection | |
| CN115062348A (en) | Electric power data preprocessing system and method | |
| CN107240022A (en) | Insurance information processing method, device and system | |
| CN110752930A (en) | Electronic evidence security device | |
| US20180122498A1 (en) | Methods for securely processing information having handwritten data | |
| CN110135684B (en) | Capability assessment method, capability assessment device and terminal equipment | |
| CN104021217A (en) | System and method for extracting fragment file and deleted file of mobile phone | |
| CN111597418B (en) | Big data-based archive data storage system | |
| CN109753961A (en) | A kind of substation's spacer units unlocking method and system based on image recognition | |
| CN114997843A (en) | Enterprise operation management information storage system | |
| CN114444105A (en) | Intelligent audit data reporting safety method | |
| CN114528591A (en) | Data management method, device, server and storage medium | |
| CN114707166A (en) | Data storage and reading method, device, equipment and storage medium | |
| CN113469717B (en) | Animal identity pedigree tracing system and method based on blockchain and iris recognition | |
| CN112329757A (en) | Method, device and system for desensitizing acquisition of bill information | |
| CN111008387A (en) | Anti-counterfeiting tracing system and method for printed document based on digital signature and document DNA | |
| CN110968275A (en) | The anti-counterfeiting tracing system and method for the printed document | |
| GB2546567A (en) | Method of associating a person with a digital object | |
| CN112765641B (en) | Efficient desensitization method and device | |
| CN114064893A (en) | Abnormal data auditing method, device, equipment and storage medium | |
| WO2018095737A1 (en) | Method for generating an access for an entity to identification data of a person, associated generation device and authentication method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220916 |
|
| RJ01 | Rejection of invention patent application after publication |