CN113190542A - Big data cleaning and denoising method and system for power grid and computer storage medium - Google Patents
Big data cleaning and denoising method and system for power grid and computer storage medium Download PDFInfo
- Publication number
- CN113190542A CN113190542A CN202110545316.9A CN202110545316A CN113190542A CN 113190542 A CN113190542 A CN 113190542A CN 202110545316 A CN202110545316 A CN 202110545316A CN 113190542 A CN113190542 A CN 113190542A
- Authority
- CN
- China
- Prior art keywords
- detection data
- power
- data
- power grid
- abnormal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003860 storage Methods 0.000 title claims abstract description 35
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000004140 cleaning Methods 0.000 title claims abstract description 24
- 238000001514 detection method Methods 0.000 claims abstract description 186
- 230000002159 abnormal effect Effects 0.000 claims abstract description 67
- 238000007405 data analysis Methods 0.000 claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000012423 maintenance Methods 0.000 claims description 17
- 238000012216 screening Methods 0.000 claims description 16
- 238000012217 deletion Methods 0.000 claims description 9
- 230000037430 deletion Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims 1
- 238000004458 analytical method Methods 0.000 abstract description 8
- 238000007667 floating Methods 0.000 description 6
- 238000013024 troubleshooting Methods 0.000 description 5
- 230000009191 jumping Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010248 power generation Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
- Y02E40/00—Technologies for an efficient electrical power generation, transmission or distribution
- Y02E40/70—Smart grids as climate change mitigation technology in the energy generation sector
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Economics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Marketing (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Software Systems (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Quality & Reliability (AREA)
- Supply And Distribution Of Alternating Current (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application relates to the technical field of data processing, in particular to a big data cleaning and denoising method, a system and a computer storage medium for a power grid. Abnormal data with empty numerical values in the power grid data can be deleted before being stored, power detection data items missing in the stored power grid data can be filled with corresponding filling values, analysis time of the power grid data is reduced, data analysis results are not affected, meanwhile, the abnormal data are stored in an abnormal marking mode, and therefore a worker can analyze the abnormal power grid data conveniently, and the problem that detection and analysis efficiency of the power grid data is low is improved.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a method and a system for cleaning and denoising big data for a power grid and a computer storage medium.
Background
With the continuous development of economy and the continuous progress of related technologies in China, the coverage rate of the smart power grid is continuously increased at a higher speed. The data source of the power grid industry mainly comes from each link of power generation, power transmission, power transformation, power distribution and power utilization of power production and electric energy use, and the electric energy metering data of each link is counted, collected and processed. Because the data volume of the whole network power grid is very big, the phenomenon that the same time node detects many times can appear to the electric power detection data of power equipment, and only once data is normal, and the abnormal data is direct to be stored, can increase the detection time of later stage staff to the power grid data for the detection analysis efficiency to the power grid data is lower.
Disclosure of Invention
In order to solve the problem of low detection and analysis efficiency of power grid data, the application provides a method, a system and a computer storage medium for cleaning and denoising big data for a power grid.
In a first aspect, the application provides a big data cleaning and denoising method for a power grid, which adopts the following technical scheme:
a big data cleaning and denoising method for a power grid comprises the steps of obtaining power grid data, eliminating abnormal data items in the power grid data, storing the abnormal data items in an abnormal data table, storing the power grid data with the abnormal data items eliminated in a first detection data table, filling missing abnormal data in the first detection data table through a preset filling model, and filling the filled data into missing positions in the first detection data table;
the preset filling model comprises:
acquiring the missing type of the missing power detection data item;
if the deletion type is random deletion or complete random deletion, the filling model calls a first model for filling, wherein the first model is
In the formula ay1Detecting the previous time node information of the corresponding time node information of the data item for the missing power, ax1The time node information is the next time node information of the corresponding time node information of the power detection data item which is missed, n is the number of the selected adjacent time node information of the corresponding time node information of the power detection data item which is missed and n is an integer, A is the number of the adjacent power detection data items which are calculated;
if the loss type is non-random loss, the filling model calls a second model to fill, wherein the second model comprises a plurality of experience pools, the experience pools respectively correspond to each data item in the power grid data, a plurality of memory slots are arranged in the experience pools, and the memory slots are used for recording the acquisition result of the data item;
obtaining the front Y of the experience pool corresponding to the missing power detection data itemiCollecting results of the wheels and the shapes of a plurality of memory grooves in the experience pool, wherein the value range of i is more than or equal to 1;
if the missing power detection data item corresponds to Y before the experience pooliIf the acquisition results of the wheels are all null values, the missing power detection data items are removed, and filling is not performed;
if the missing power detection data item corresponds to Y before the experience pooliWhen the acquisition result of the wheel has an acquisition value, acquiring the shape of a memory slot corresponding to the acquisition result, and comparing the shape of the memory slot with the shape of the memory slot corresponding to each data item in the current power grid data to acquire the most similar data item for similar filling;
further, the grid data includes:
the power detection data items comprise names, numerical values, time node information and power equipment numbers of power equipment.
Further, the exception data item includes:
a power detection data item whose value is empty.
Further, the method also comprises the following steps:
and forming a plurality of corresponding data analysis graphs according to the first detection data table, wherein any data analysis graph corresponds to the power detection data item and is associated with corresponding power equipment in a hyperlink mode.
Further, the method also comprises the following steps:
and if the power detection data items of the continuous time node information and the abnormal marks exist in the first detection data table, the identification information of the power equipment corresponding to the power detection data items of the continuous time node information and the abnormal marks and the corresponding time node information are stored to form a warning report.
Further, the method also comprises the following steps:
matching and detecting time node information and abnormal marks of power detection data items with abnormal marks in each first detection data table and corresponding power detection data items in a preset maintenance storage table, wherein the maintenance storage table comprises power detection data items of different time nodes of corresponding power equipment before the first detection data table;
and if the power detection data item with the abnormal mark in the first detection data table is matched with the time node information of the power detection data item with the abnormal mark in the corresponding maintenance storage table, storing the identification information of the power equipment corresponding to the matched power detection data item in the first detection data table and the corresponding time node information to form a maintenance report.
Further, the method also comprises the following steps:
screening the power detection data items with the abnormal marks in each first detection data table to form a screening data table, wherein the screening data table comprises identification information and time node information of the power equipment corresponding to the power detection data items with the abnormal marks;
and matching and screening the data items in the abnormal data table according to the data items in the screening data table, and deleting all matched data of the data items in the abnormal data table to form a checking data table.
In a second aspect, the application provides a big data cleaning and denoising system for a power grid, which adopts the following technical scheme:
the acquisition module is used for acquiring power grid data;
the storage module is used for storing a preset classification principle, a preset abnormal data table, a preset first detection data table and a preset filling model;
a first processing module, configured to perform the following processing:
deleting the power detection data item with the empty corresponding numerical value in the power grid data from the corresponding power grid data, and storing the power detection data item with the empty corresponding numerical value in a preset abnormal data table stored in a storage module;
updating and storing the deleted power grid data in a corresponding preset first detection data table;
whether a first detection data table has missing power detection data items or not is detected according to the sequence of time node information, if the first detection data table has the missing power detection data items, the missing power detection data items of the first detection data table are filled according to a preset filling model stored in a storage module, and the power detection data items filled in the first detection data table are subjected to abnormity marking.
In a third aspect, the present application provides an intelligent terminal, which adopts the following technical scheme:
an intelligent terminal comprising a memory and a processor, said memory having stored thereon a computer program that can be loaded by the processor and that executes the method according to the first aspect.
By adopting the technical scheme, after the power grid data are obtained, the power grid data are classified, cleaned and denoised, abnormal data stored in the database are reduced, and the utilization rate of the storage space of the database is improved.
In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solutions:
a computer readable storage medium comprising a computer program stored thereon which is loadable by a processor and adapted to carry out the method of the first aspect.
By adopting the technical scheme, the computer readable storage medium can store corresponding programs, can clean and denoise the big data of the power grid and then store the big data in the database, and is beneficial to improving the problem of poor detection and analysis efficiency of the power grid data.
In summary, the present application includes at least one of the following beneficial technical effects:
the power equipment with the abnormal power detection data and the time node with the abnormal power detection data of the same time node in the two continuous time periods are stored in the maintenance report, so that the detection condition of the power equipment in the power grid data can be analyzed and processed by workers, and the detection and analysis efficiency of the power grid data can be improved. Abnormal data with empty numerical values in the power grid data can be deleted before being stored, power detection data items missing in the stored power grid data can be filled with corresponding filling values, analysis time of the power grid data is reduced, data analysis results are not affected, meanwhile, the abnormal data are stored in an abnormal marking mode, and therefore a worker can analyze the abnormal power grid data conveniently, and the problem that detection and analysis efficiency of the power grid data is low is improved.
Drawings
Fig. 1 is a schematic flow chart of a method for cleaning and denoising big data for a power grid according to an embodiment of the present application.
Fig. 2 is a block diagram of a structure of a big data cleaning and denoising system for a power grid according to an embodiment of the present application.
Description of reference numerals: 1. an acquisition module; 2. a storage module; 3. a first processing module.
Detailed Description
The present application is described in further detail below with reference to figures 1-2.
The embodiment of the application discloses a method and a system for cleaning and denoising big data for a power grid and a computer storage medium. Referring to fig. 1, a big data cleaning and denoising method for a power grid includes:
s1: and acquiring power grid data.
Specifically, the grid data includes identification information of electrical devices in a plurality of areas, and names, values, and current time node information of corresponding electrical detection data items, where the electrical devices are devices such as generators and transformers. Because each power device plays a different role in the operation process, the power detection data items corresponding to each power device are different, for example, the power detection data items of the generator are power generation amount, current and the like, and the power detection data items of the transformer are input voltage, output voltage and the like.
S2: and deleting the power detection data item with the empty corresponding numerical value in the power grid data from the corresponding power grid data, and storing the power detection data item with the empty corresponding numerical value in a corresponding preset abnormal data table.
Specifically, the abnormal data table includes names of power detection data items and corresponding time node information, where values of current time nodes in a preset time period in a corresponding area are empty, where the preset time period is one day or one hour, the time node information is time nodes averagely divided in the preset time period, a time interval between two adjacent pieces of time node information is 5 minutes or 10 minutes, and the like, it should be noted that other settings may be performed according to actual needs for selection of the time period and the time node information, and this is not specifically limited in this application.
S3: and updating and storing the deleted power grid data in a corresponding preset first detection data table.
Specifically, the first detection data table is used to store power detection data items of the power devices in the corresponding area within a preset time period, where the first detection data table in S3 includes names, values, and time node information of the power detection data items of a plurality of time nodes of the power devices in the corresponding area before the current time node, where each first detection data table corresponds to one preset time period, each first detection data table has a plurality of sub data tables, and each sub data table stores power detection data items of the corresponding power devices.
S4: detecting whether each first detection data table has missing power detection data items according to the sequence of the time node information, and if detecting that one or a plurality of first detection data tables have missing power detection data items, jumping to S5; if it is not detected that there is a missing power detection data item in one or more of the first detection data tables, the process proceeds to S6.
S5: and filling the power detection data items missing in the first detection data table according to a preset filling model, and performing abnormity marking on the power detection data items filled in the first detection data table so as to analyze the power grid big data by workers.
Specifically, the abnormal marking is performed by marking the filled power detection data items as warning colors, and the numerical value of the power detection data item corresponding to each power device has a certain variation range.
Acquiring the missing type of the missing power detection data item;
if the deletion type is random deletion or complete random deletion, the filling model calls a first model for filling, wherein the first model is
ay1Detecting the previous time node information of the corresponding time node information of the data item for the missing power, ax1The time node information is the next time node information of the corresponding time node information of the power detection data item which is missed, n is the number of the selected adjacent time node information of the corresponding time node information of the power detection data item which is missed and n is an integer, and A is the number of the adjacent power detection data items which are calculated. When the time node information corresponding to the missing power detection data item is the end point of the preset time period, the value of A is n, and the numerical value corresponding to the power detection data item without the time node information is defined as O; when it is at homeAnd if the time node information corresponding to the missing power detection data item is the time node information before or after the end point of the preset time period, the value of A is 2 n-1. It should be noted that the manner of the exception flag and the value of n may be set according to actual needs, and this is not specifically limited in this application.
If the loss type is non-random loss, the filling model calls a second model to fill, wherein the second model comprises a plurality of experience pools, the experience pools respectively correspond to each data item in the power grid data, a plurality of memory slots are arranged in each experience pool, and the memory slots are used for recording the acquisition result of the data item;
obtaining the front Y of the experience pool corresponding to the missing power detection data itemiCollecting results of the wheels and the shapes of a plurality of memory grooves in the experience pool, wherein the value range of i is more than or equal to 1;
in specific implementation, whether a missing power detection data item appears in an acquisition result is firstly acquired, wherein the acquisition result comprises a plurality of acquired power grid data.
If the missing power detection data item corresponds to Y before the experience pooliIf the acquisition results of the wheels are all null values, the missing power detection data items are removed, and filling is not performed;
in specific implementation, when no missing power detection data item appears in the acquisition result, the missing power detection data item is judged to be interference data or error data, and then the interference data or the error data is eliminated.
If the missing power detection data item corresponds to Y before the experience pooliAnd when the acquisition result of the wheel has an acquisition value, acquiring the shape of the memory slot corresponding to the acquisition result, and comparing the shape of the memory slot corresponding to each data item in the current power grid data to acquire the most similar data item for similar filling.
In specific implementation, when the missing power detection data item appears in the acquisition result, the missing power detection data item is judged to be valid acquisition data, and at this time, the missing power detection data item is similarly filled according to the shape of the memory slot corresponding to the missing power detection data item appearing in the acquisition result.
Specifically, the comparison process includes obtaining a value of i corresponding to the acquisition result, and when i is greater than 1, multiplying i by a fuzzy coefficient to obtain a floating range value, wherein the fuzzy coefficient is specifically
KQ is the sum of the electric power detection data item of the main power grid and the electric power detection data item in the first data table, and through a fuzzy coefficient, the longer the acquisition time interval is, the more fuzzy the corresponding shape of the memory slot is, so that the range of the electric power detection data item after similar filling is larger, and the later-stage manual adjustment is facilitated.
The floating range value is different because the selected i is different every time, but the floating range value is increased along with the increase of i;
and when i is larger than 1, adding the shape of the memory slot corresponding to the acquisition result with the floating range value to obtain the shape of the floating memory slot, acquiring the shape of the memory slot corresponding to each data item in the power grid data and the shape of the memory slot corresponding to each data item in the power grid data, and selecting the data item in the power grid data with the same shape as the floating memory slot as the filling data.
And when i is equal to 1, comparing the shape of the memory slot corresponding to the acquisition result with the shape of the memory slot corresponding to each data item in the power grid data, and selecting the data item in the power grid data with the same shape as the memory slot as filling data.
Correspondingly, when the value of i is larger, the reliability of the corresponding similar filling result is lower, a credibility threshold is set according to the importance of the collected power grid data in specific implementation, and when the reliability is lower than the credibility threshold, the filling value machine is marked and is subjected to manual examination.
S6: and forming a plurality of corresponding data analysis graphs according to each first detection data table.
Each data analysis graph comprises names, numerical values and time node information of power detection data items of a plurality of time nodes of the corresponding power equipment before the current time node.
S7: and associating each data analysis graph with the identification information of the corresponding power equipment in the corresponding first detection data table in a hyperlink mode, so that a worker can click the identification information of the power equipment to jump to the corresponding data analysis graph.
In order to facilitate data analysis on the grid big data, after S7, referring to fig. 2, the method further includes:
s11: detecting whether a plurality of power detection data items with continuous time node information and abnormal marks exist in the sub-data table of each first detection data table according to the sequence of the time node information, and if detecting that a plurality of power detection data items with continuous time node information and abnormal marks exist in the first detection data table, jumping to S12; if it is not detected that there are consecutive plural time node information power detection data items having an abnormality flag in the first detection data table, it jumps to S13.
S12: and storing the identification information of the electric power equipment corresponding to the electric power detection data items with the plurality of continuous time node information and abnormal marks and the corresponding time node information to form an alarm report, so that a worker can maintain the relevant electric power equipment according to the alarm report.
Specifically, the alarm report includes identification information of the corresponding power device and start time node information of a plurality of consecutive time node information, where if the power device has abnormal data of the plurality of consecutive time node information, the power device has a device abnormal condition.
S13: matching and detecting the power detection data item with the abnormal mark in each first detection data table and the corresponding power detection data item in the corresponding maintenance storage table by using the time node information and the abnormal mark, and if the power detection data item with the abnormal mark in the first detection data table is matched with the time node information of the power detection data item with the abnormal mark in the corresponding maintenance storage table, jumping to S14; if the power detection data item with the abnormal mark in the first detection data table does not match the time node information of the power detection data item with the abnormal mark in the corresponding maintenance storage table, the process goes to S15.
The maintenance storage table comprises names, values, time node information and power equipment numbers of power detection data items of power equipment in a time period before the current time period in the corresponding region.
S14: and storing the name of the electric power equipment corresponding to the matched electric power detection data item in the first detection data table and the corresponding time node information to form a maintenance report, so that the staff can reset and maintain the electric power equipment in the maintenance report.
If the node information of the power equipment in the maintenance report has an empty numerical value in the same time period in two time periods, the power detection data of the power equipment is acquired or corresponding data is uploaded to cause an abnormality.
S15: and screening the power detection data items with the abnormal marks in each first detection data table to form a screening data table.
The screening data table comprises identification information and time node information of the electric power equipment corresponding to the electric power detection data item with the abnormal mark.
S16: and matching and screening the identification information, the electric power data item name and the time node information of the electric power equipment on the data items in the abnormal data table according to the data items in the screening data table, and deleting the data items matched with the data items in the screening data table in the abnormal data table to form a troubleshooting data table so as to allow a worker to reset and maintain the electric power equipment in the troubleshooting data table.
Specifically, the identifier information of the electrical equipment and the name and time node information of the corresponding electrical detection data item contained in the troubleshooting data table are included, wherein the electrical data items corresponding to the electrical equipment are repeatedly measured at a certain time node and occupy the storage resource of the database, so that a worker performs troubleshooting and modification setting according to the troubleshooting data table.
Based on the above method, the embodiment of the present application further discloses a big data cleaning and denoising system for a power grid, and referring to fig. 2, the big data cleaning and denoising system for the power grid includes:
the acquisition module 1 is used for acquiring power grid data;
the storage module 2 is used for storing a preset classification principle, a preset abnormal data table, a preset first detection data table, a preset filling model and a preset maintenance storage table;
a first processing module 3, configured to perform the following processing:
deleting the power detection data item with the empty corresponding numerical value in the power grid data from the corresponding power grid data, and storing the power detection data item with the empty corresponding numerical value in a preset abnormal data table stored in a storage module 2;
updating and storing the deleted power grid data in a corresponding preset first detection data table;
whether a first detection data table has missing power detection data items or not is detected according to the sequence of time node information, if the first detection data table has the missing power detection data items, the missing power detection data items of the first detection data table are filled according to a preset filling model stored in a storage module 2, and the power detection data items filled in the first detection data table are subjected to abnormity marking.
The embodiment of the application also discloses an intelligent terminal, which comprises a memory and a processor, wherein the memory is stored with a computer program which can be loaded by the processor and can execute the big data cleaning and denoising method for the power grid.
The embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program that can be loaded by a processor and execute the above big data cleaning and denoising method for a power grid, and the computer-readable storage medium includes, for example: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above examples are only used to illustrate the technical solutions of the present application, and do not limit the scope of protection of the application. It is to be understood that the embodiments described are only some of the embodiments of the present application and not all of them. Based on these embodiments, all other embodiments obtained by a person of ordinary skill in the art without any creative effort belong to the protection scope of the present application.
Claims (10)
1. A big data cleaning and denoising method for a power grid comprises the steps of obtaining power grid data, eliminating abnormal data items in the power grid data, storing the abnormal data items in an abnormal data table, and storing the power grid data with the abnormal data items eliminated in a first detection data table, wherein the abnormal data missing in the first detection data table are filled through a preset filling model, and the filled data are filled in the missing position in the first detection data table;
the preset filling model comprises:
acquiring the missing type of the missing power detection data item;
if the deletion type is random deletion or complete random deletion, the filling model calls a first model for filling, wherein the first model is
In the formula ay1Detecting the previous time node information of the corresponding time node information of the data item for the missing power, ax1The time node information is the next time node information of the corresponding time node information of the power detection data item which is missed, n is the number of the selected adjacent time node information of the corresponding time node information of the power detection data item which is missed and n is an integer, A is the number of the adjacent power detection data items which are calculated;
if the loss type is non-random loss, the filling model calls a second model to fill, wherein the second model comprises a plurality of experience pools, the experience pools respectively correspond to each data item in the power grid data, a plurality of memory slots are arranged in the experience pools, and the memory slots are used for recording the acquisition result of the data item;
obtaining the front Y of the experience pool corresponding to the missing power detection data itemiWheel acquisition and warpThe shape of a plurality of memory grooves in the test pool, wherein the value range of i is more than or equal to 1;
if the missing power detection data item corresponds to Y before the experience pooliIf the acquisition results of the wheels are all null values, the missing power detection data items are removed, and filling is not performed;
if the missing power detection data item corresponds to Y before the experience pooliWhen the acquisition result of the wheel has an acquisition value, acquiring the shape of a memory slot corresponding to the acquisition result, and comparing the shape of the memory slot with the shape of the memory slot corresponding to each data item in the current power grid data to acquire the most similar data item for similar filling;
2. the big data cleaning and denoising method for the power grid according to claim 1, wherein the power grid data comprises:
the power detection data items comprise names, numerical values, time node information and power equipment numbers of power equipment.
3. The big data cleaning and denoising method for the power grid as claimed in claim 2, wherein the abnormal data items comprise:
a power detection data item whose value is empty.
4. The big data cleaning and denoising method for the power grid according to claim 1, further comprising:
and forming a plurality of corresponding data analysis graphs according to the first detection data table, wherein any data analysis graph corresponds to the power detection data item and is associated with corresponding power equipment in a hyperlink mode.
5. The big data cleaning and denoising method for the power grid according to claim 4, further comprising:
and if the power detection data items of the continuous time node information and the abnormal marks exist in the first detection data table, the identification information of the power equipment corresponding to the power detection data items of the continuous time node information and the abnormal marks and the corresponding time node information are stored to form a warning report.
6. The big data cleaning and denoising method for the power grid according to claim 5, further comprising:
matching and detecting time node information and abnormal marks of power detection data items with abnormal marks in each first detection data table and corresponding power detection data items in a preset maintenance storage table, wherein the maintenance storage table comprises power detection data items of different time nodes of corresponding power equipment before the first detection data table;
and if the power detection data item with the abnormal mark in the first detection data table is matched with the time node information of the power detection data item with the abnormal mark in the corresponding maintenance storage table, storing the identification information of the power equipment corresponding to the matched power detection data item in the first detection data table and the corresponding time node information to form a maintenance report.
7. The big data cleaning and denoising method for the power grid according to claim 1, further comprising:
screening the power detection data items with the abnormal marks in each first detection data table to form a screening data table, wherein the screening data table comprises identification information and time node information of the power equipment corresponding to the power detection data items with the abnormal marks;
and matching and screening the data items in the abnormal data table according to the data items in the screening data table, and deleting all matched data of the data items in the abnormal data table to form a checking data table.
8. A big data cleaning and denoising system for a power grid is characterized by comprising,
the acquisition module (1) is used for acquiring power grid data;
the storage module (2) is used for storing a preset classification principle, a preset abnormal data table, a preset first detection data table and a preset filling model;
a first processing module (3) for performing the following processing:
deleting the power detection data item with the empty corresponding numerical value in the power grid data from the corresponding power grid data, and storing the power detection data item with the empty corresponding numerical value in a preset abnormal data table stored in a storage module (2);
updating and storing the deleted power grid data in a corresponding preset first detection data table;
whether a first detection data table has missing power detection data items or not is detected according to the sequence of time node information, if the first detection data table has the missing power detection data items, the missing power detection data items of the first detection data table are filled according to a preset filling model stored in a storage module (2), and the power detection data items filled in the first detection data table are subjected to abnormity marking.
9. An intelligent terminal, comprising a memory and a processor, the memory having stored thereon a computer program that can be loaded by the processor and that executes the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which can be loaded by a processor and which executes the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110545316.9A CN113190542B (en) | 2021-05-19 | 2021-05-19 | Big data cleaning and denoising method and system for power grid and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110545316.9A CN113190542B (en) | 2021-05-19 | 2021-05-19 | Big data cleaning and denoising method and system for power grid and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113190542A true CN113190542A (en) | 2021-07-30 |
| CN113190542B CN113190542B (en) | 2023-02-24 |
Family
ID=76982480
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110545316.9A Active CN113190542B (en) | 2021-05-19 | 2021-05-19 | Big data cleaning and denoising method and system for power grid and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113190542B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116708514A (en) * | 2023-08-02 | 2023-09-05 | 深圳龙电华鑫控股集团股份有限公司 | Electric energy meter data acquisition method and system based on Internet of things |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150100530A1 (en) * | 2013-10-08 | 2015-04-09 | Google Inc. | Methods and apparatus for reinforcement learning |
| US20170140269A1 (en) * | 2015-11-12 | 2017-05-18 | Google Inc. | Training neural networks using a prioritized experience memory |
| CN109766334A (en) * | 2019-01-07 | 2019-05-17 | 国网湖南省电力有限公司 | Processing method and system for electrical equipment online supervision abnormal data |
| CN110825798A (en) * | 2019-10-29 | 2020-02-21 | 深圳供电局有限公司 | Electric power application data maintenance method and device |
| CN111797078A (en) * | 2019-04-09 | 2020-10-20 | Oppo广东移动通信有限公司 | Data cleaning method, model training method, device, storage medium and equipment |
| CN111797857A (en) * | 2019-04-09 | 2020-10-20 | Oppo广东移动通信有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN112164454A (en) * | 2020-10-10 | 2021-01-01 | 联仁健康医疗大数据科技股份有限公司 | Diagnosis prediction method and device and electronic equipment |
-
2021
- 2021-05-19 CN CN202110545316.9A patent/CN113190542B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150100530A1 (en) * | 2013-10-08 | 2015-04-09 | Google Inc. | Methods and apparatus for reinforcement learning |
| US20170140269A1 (en) * | 2015-11-12 | 2017-05-18 | Google Inc. | Training neural networks using a prioritized experience memory |
| CN109766334A (en) * | 2019-01-07 | 2019-05-17 | 国网湖南省电力有限公司 | Processing method and system for electrical equipment online supervision abnormal data |
| CN111797078A (en) * | 2019-04-09 | 2020-10-20 | Oppo广东移动通信有限公司 | Data cleaning method, model training method, device, storage medium and equipment |
| CN111797857A (en) * | 2019-04-09 | 2020-10-20 | Oppo广东移动通信有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN110825798A (en) * | 2019-10-29 | 2020-02-21 | 深圳供电局有限公司 | Electric power application data maintenance method and device |
| CN112164454A (en) * | 2020-10-10 | 2021-01-01 | 联仁健康医疗大数据科技股份有限公司 | Diagnosis prediction method and device and electronic equipment |
Non-Patent Citations (4)
| Title |
|---|
| JING SHI ET AL: "Concept Learning through Deep Reinforcement Learning with Memory-Augmented Neural Networks", 《ARXIV:1811.06145V1 [CS.LG] 15 NOV 2018》 * |
| MENG XU ET AL: "Play games using Reinforcement Learning and Artificial Neural Networks with Experience Replay", 《2018 IEEE/ACIS 17TH INTERNATIONAL CONFERENCE ON COMPUTER AND INFORMATION SCIENCE (ICIS)》 * |
| 万里鹏 等: "深度强化学习理论及其应用综述", 《模式识别与人工智能》 * |
| 汪晨 等: "记忆增强型深度强化学习研究综述", 《小型微型计算机系统》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116708514A (en) * | 2023-08-02 | 2023-09-05 | 深圳龙电华鑫控股集团股份有限公司 | Electric energy meter data acquisition method and system based on Internet of things |
| CN116708514B (en) * | 2023-08-02 | 2023-10-31 | 深圳龙电华鑫控股集团股份有限公司 | Electric energy meter data acquisition method and system based on Internet of things |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113190542B (en) | 2023-02-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108376982A (en) | Load recognition methods and the device of phase sequence | |
| CN110175167A (en) | A kind of data cleaning method and system suitable for low-voltage platform area electricity consumption data | |
| CN108802535B (en) | Screening method, main interference source identification method and device, server and storage medium | |
| CN110571925B (en) | Method for analyzing power quality by using data of power distribution network monitoring terminal | |
| CN109033174A (en) | A kind of power quality data cleaning method and device | |
| CN108053095B (en) | Power quality disturbance event feature extraction method and system | |
| CN115271263B (en) | Power equipment defect early warning method, system and medium based on improved association rule | |
| CN112214481A (en) | Data cleaning method for power quality data | |
| CN113190542B (en) | Big data cleaning and denoising method and system for power grid and computer storage medium | |
| CN113125903A (en) | Line loss anomaly detection method, device, equipment and computer-readable storage medium | |
| CN111881124A (en) | Data processing method and system based on state estimation of improved algorithm | |
| CN111489053A (en) | FP-growth algorithm-based power grid anomaly reason determination method and system | |
| CN115796708A (en) | Intelligent quality inspection method, system and medium for big data for engineering construction | |
| CN116401532A (en) | Method and system for recognizing frequency instability of power system after disturbance | |
| CN114168583A (en) | Electric quantity data cleaning method and system based on regular automatic encoder | |
| CN112595918A (en) | Low-voltage meter reading fault detection method and device | |
| CN115494382B (en) | Method, system, equipment and storage medium for judging upper and lower level relation of intelligent switch | |
| CN111125078A (en) | Defect data correction method for relay protection device | |
| CN115759250A (en) | Attribution analysis method, attribution analysis device, electronic equipment and storage medium | |
| CN114665610B (en) | Capacitor monitoring method, system and equipment based on reactive power acquisition | |
| CN112526440A (en) | Method and system for monitoring running error of electric energy meter based on accuracy grade of mutual inductor | |
| CN115907308B (en) | Electric power material provider evaluation method and device based on user portrait | |
| CN113541600B (en) | Method, system, equipment and storage medium for judging branch fault of photovoltaic power station | |
| CN114330527B (en) | Building ammeter distribution identification method, system, device and storage medium | |
| CN116307860B (en) | Photovoltaic power station monitoring and alarming method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |