CN116861290A - Unstructured data detection classification method, model training method and device - Google Patents
Unstructured data detection classification method, model training method and device Download PDFInfo
- Publication number
- CN116861290A CN116861290A CN202310799513.2A CN202310799513A CN116861290A CN 116861290 A CN116861290 A CN 116861290A CN 202310799513 A CN202310799513 A CN 202310799513A CN 116861290 A CN116861290 A CN 116861290A
- Authority
- CN
- China
- Prior art keywords
- local
- generator
- unstructured data
- training
- unstructured
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 title claims abstract description 126
- 238000001514 detection method Methods 0.000 title claims abstract description 110
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000013145 classification model Methods 0.000 claims abstract description 61
- 238000004590 computer program Methods 0.000 claims abstract description 26
- 238000003860 storage Methods 0.000 claims abstract description 12
- 230000002776 aggregation Effects 0.000 claims description 12
- 238000004220 aggregation Methods 0.000 claims description 12
- 230000004931 aggregating effect Effects 0.000 claims description 5
- 238000004458 analytical method Methods 0.000 abstract description 4
- 238000010191 image analysis Methods 0.000 abstract description 4
- 238000012544 monitoring process Methods 0.000 description 16
- 239000002131 composite material Substances 0.000 description 11
- 238000012545 processing Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The application relates to an unstructured data detection classification model training method, a device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an initial unstructured dataset; performing federal learning iteration according to the local generator training set and the local discriminant training set to obtain a local generator updating information set and a local discriminant updating information set; acquiring a generator parameter set and a discriminator parameter set according to the local generator update information set and the local discriminator update information set; repeatedly iterating to obtain a generator parameter set and a discriminator parameter set until reaching a target standard; an unstructured data detection classification model is obtained based on the generator parameter set and the arbiter parameter set. When the analysis and modeling problems of the power unstructured image data are solved, distributed data are fully utilized, data privacy is protected, and accurate image analysis and modeling capability is provided, so that management and safety of a power system are improved.
Description
Technical Field
The present application relates to the field of data processing technology, and in particular, to an unstructured data detection classification model training method, a model training method, an apparatus, a computer device, a storage medium, and a computer program product.
Background
Less unstructured data is involved in conventional power data because conventional power data is primarily concerned with aspects such as the operating status and supply of demand of the power system. However, with the development of digitization and intelligence in the power industry, unstructured data also began to appear in certain application scenarios. For example, cameras monitoring power equipment or power lines may generate image data for detecting equipment status, fault detection, safety monitoring, and the like. Thus, in certain power applications, there may also be power data types associated with the image.
Because of the existence and application of unstructured data (such as image data) in power applications, particularly in power industry applications after digital and intelligent upgrading, a distributed learning paradigm, i.e. a federal learning method based on the unstructured data of power, which can protect data security and privacy, becomes one of the pain points to be solved in technology and application.
Disclosure of Invention
Based on this, there is a need to provide an unstructured data detection classification model training method, a model training method, an apparatus, a computer device, a storage medium and a computer program product that can fully utilize distributed data, protect data privacy, and provide accurate image analysis and modeling capabilities.
In a first aspect, the application provides a training method for an unstructured data detection classification model. The method comprises the following steps:
acquiring an initial unstructured dataset; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set;
performing federal learning iteration according to the local generator training set and the local discriminant training set to obtain a local generator updating information set and a local discriminant updating information set;
acquiring a generator parameter set and a discriminator parameter set according to the local generator update information set and the local discriminator update information set;
repeatedly iterating to obtain a generator parameter set and a discriminator parameter set until reaching a target standard;
an unstructured data detection classification model is obtained based on the generator parameter set and the arbiter parameter set.
In one embodiment, acquiring the initial unstructured dataset comprises:
Acquiring an initial unstructured data set according to an initial participant;
determining a participant according to the initial participant; the participants retain the local data of the initial unstructured dataset.
In one embodiment, performing federal learning iterations from a local generator training set and a local arbiter training set, the obtaining a local generator update information set and a local arbiter update information set includes:
selecting a training party among the participants;
instructing a training party to train a local generator model according to the local generator training set, and acquiring a local generator update information set;
the instruction training party trains the local discriminant model according to the local discriminant training set, and acquires the local discriminant updating information set.
In one embodiment, obtaining the generator parameter set and the arbiter parameter set from the local generator update information set and the local arbiter update information set comprises:
aggregating the local generator update information set and the local arbiter update information set to a central server;
and acquiring a generator parameter set and a discriminator parameter set according to the aggregation rule.
In a second aspect, the present application also provides an unstructured data detection classification method using the unstructured data detection classification model as provided in the first aspect. The method comprises the following steps:
Obtaining an unstructured dataset;
and calling an unstructured data detection classification model, and inputting the unstructured data set into the unstructured data detection classification model to obtain an unstructured data detection classification result.
In a third aspect, the application further provides an unstructured data detection classification model training device. The device comprises:
the initial unstructured data set acquisition module is used for acquiring an initial unstructured data set; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set;
the federal learning iteration module is used for performing federal learning iteration according to the local generator training set and the local discriminant training set to obtain a local generator updating information set and a local discriminant updating information set;
the parameter set acquisition module is used for acquiring a generator parameter set and a discriminator parameter set according to the local generator update information set and the local discriminator update information set;
the repeated iteration module is used for repeatedly and iteratively acquiring a generator parameter set and a discriminator parameter set until the target standard is reached;
the model generation module is used for acquiring an unstructured data detection classification model based on the generator parameter set and the discriminator parameter set.
In a fourth aspect, the present application also provides an unstructured data detection classification device, which includes:
the unstructured data set acquisition module is used for acquiring an unstructured data set;
the model calling module is used for calling the unstructured data detection classification model, inputting the unstructured data set into the unstructured data detection classification model, and obtaining an unstructured data detection classification result.
In a fifth aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of the unstructured data detection classification model training method when executing the computer program; or, the step of realizing the unstructured data detection classification method is realized.
In a sixth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium has stored thereon a computer program which, when executed by a processor, performs the steps of the unstructured data detection classification model training method described above; or, the step of realizing the unstructured data detection classification method is realized.
In a seventh aspect, the present application also provides a computer program product. The computer program product comprises a computer program which is executed by a processor to realize the steps of the unstructured data detection classification model training method; or, the step of realizing the unstructured data detection classification method is realized.
The unstructured data detection classification model training method, the device, the computer equipment, the storage medium and the computer program product are used for acquiring an initial unstructured data set; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set; performing federal learning iteration according to the local generator training set and the local discriminant training set to obtain a local generator updating information set and a local discriminant updating information set; acquiring a generator parameter set and a discriminator parameter set according to the local generator update information set and the local discriminator update information set; repeatedly iterating to obtain a generator parameter set and a discriminator parameter set until reaching a target standard; the unstructured data detection classification model is obtained based on the generator parameter set and the discriminator parameter set, unstructured data of all the participants are fully utilized, and model training is carried out in a federal learning mode. Meanwhile, the method protects the data privacy, original image data does not need to be shared to other participants, and the safety of the data is ensured. By acquiring an unstructured dataset; the unstructured data detection classification model is called, the unstructured data set is input into the unstructured data detection classification model, and an unstructured data detection classification result is obtained.
Drawings
FIG. 1 is a flow diagram of an unstructured data detection classification model training method in one embodiment;
FIG. 2 is a flow diagram of an unstructured data detection classification method in one embodiment;
FIG. 3 is a flow chart of an unstructured data detection classification method in another embodiment;
FIG. 4 is a schematic diagram of an unstructured data detection classification model training apparatus in one embodiment;
FIG. 5 is a schematic diagram of an unstructured data detection classification device in one embodiment;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
Power data may cover a number of aspects, common power data types including the following: power load data, power supply data, power market data, grid status data, transformer data, and power quality data, respectively, are described below.
Electrical load data, which is used to record the load demands of an electrical power system at different points in time, is typically expressed in time series, e.g. per hour or per minute. The power supply data includes information of productivity, running state, generating capacity and the like of the generator, and is used for describing the supply condition of the power system. The power market data is used for recording transaction information of the power market, including matching conditions of power price, power demand and supply, transaction amount of the power market and the like. And the power grid state data is used for recording the running state of the power system, such as parameters of voltage, current, frequency and the like, and the state information of each device. Transformer data, including parameters of current, voltage, power, etc. of transformers, is used to monitor and manage transformer equipment in an electrical power system. The power quality data is used for recording power quality conditions in the power system, such as voltage fluctuation, harmonic wave, voltage sudden rise and drop and the like.
Because conventional power data is primarily concerned with the operational status and supply of demand of the power system, unstructured data may be less involved, optionally including image data. However, with the development of digitization and intelligence in the power industry, image data has also begun to appear in some application scenarios. For example, cameras monitoring power equipment or power lines may generate image data that is used to detect equipment status, fault detection, safety monitoring, and the like. Thus, in certain power applications, there may also be power data types associated with the image.
In power applications, particularly in power industry applications after digital and intelligent upgrades, the existence and application of unstructured data (such as image data) makes how to apply a distributed learning paradigm, i.e. a federal learning method based on unstructured data of power, which can protect data security and privacy, one of the problems to be solved.
In the application and business aspects, the federal learning method based on the electric power unstructured data can improve equipment state detection, fault detection and prediction, safety monitoring, violation detection and the like, so that the operation efficiency, safety and reliability of an electric power system are improved. In particular, federal learning methods based on electrical unstructured data have the following effects:
First, device state detection is improved. Image data of a plurality of entities can be jointly used through federal learning, so that the accuracy and the robustness of equipment state detection are improved, and data of different entities can jointly train a model and share knowledge about equipment states, so that more accurate detection and diagnosis are realized. And secondly fault detection and prediction. The federal learning can combine the image data of each entity to perform joint fault detection and prediction, and can discover potential fault modes and abnormal conditions through the image data of a plurality of devices and lines, and take measures in advance to repair and maintain, so that the fault occurrence rate of the power system is reduced. But also for security monitoring and violation detection. Federal learning can cooperatively use image data of various entities for safety monitoring and violation detection of power devices and lines. The method can also be used for distributed safety monitoring and violation detection, and federal learning can cooperatively use image data of each entity for safety monitoring and violation detection of power equipment and circuits. Through distributed image data processing and model training, abnormal behaviors can be identified, illegal operations or illegal conditions can be detected, for example, unauthorized persons enter forbidden areas, equipment is damaged, and the like, so that the safety and monitoring effect of the power system are improved.
In the technical aspect, the federal learning method based on the electric unstructured data can solve the problems of data privacy protection, distributed image processing, data diversity and generalization capability, data ownership, compliance and the like.
Specifically, the problem of data privacy protection can be solved first. The image data of the power device or power line may contain sensitive information such as location, device details, etc. With the traditional centralized method, the original data needs to be stored and processed in a centralized way, and the risk of privacy disclosure exists, while federal learning and privacy calculation provide a method for model training and reasoning while protecting the data privacy. Second, distributed image processing, power systems typically involve a large number of devices and lines, with image data on each device or line having local features and differences. Federal learning allows image processing tasks to be performed on local devices to better detect device status, faults, and security issues using the characteristics of the distributed data. Then, the problem of data diversity and generalization capability is solved, and federal learning can enable each participant to train by using own image data on local equipment, so that the diversity data of different areas and different equipment can be fully utilized, and the generalization capability and adaptability of the model are improved. Finally, the problem of data ownership and compliance, power equipment or power lines are often owned and managed by different entities. In a federal learning manner, each entity can retain ownership of its own data and can participate in model training and application under a compliance framework without the need to directly share the data.
The application provides a federal learning idea applied to unstructured data of a power grid, and combines the ideas of generating an countermeasure network (Generative Adversarial Network) and federal learning, and the method can effectively perform distributed federal modeling and application on unstructured image data of power.
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
In one embodiment, as shown in FIG. 1, an unstructured data detection classification model training method is provided that combines the ideas of generating a countermeasure network and federal learning, allows each participant to train generator and discriminant models on a local device, and incorporates model updates by federal learning. The embodiment is illustrated by the method applied to the terminal, and it is understood that the method can also be applied to the server, and can also be applied to a system comprising the terminal and the server, and implemented through interaction between the terminal and the server.
In this embodiment, the method includes the steps of:
step 102, obtaining an initial unstructured data set; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set.
In one embodiment, the initial unstructured dataset contains images of different types of power device states and fault conditions.
And 104, performing federal learning iteration according to the local generator training set and the local discriminant training set to acquire a local generator updating information set and a local discriminant updating information set.
In one embodiment, each participant initializes a corresponding local generator model for generating a composite image and a local discriminant model for distinguishing between a real image and a composite image prior to performing multiple rounds of federal learning iterations.
Further, a part of the participants are selected from all the participants to participate in the training of the current round, namely the training participants. Training by a training party by using a local generator training set and a local generator model to acquire a local generator updating information set; the training party trains by using the local discriminant training set and the local discriminant model to acquire a local discriminant updating information set.
And 106, acquiring a generator parameter set and a discriminator parameter set according to the local generator update information set and the local discriminator update information set.
In one embodiment, the update information of the local generator model and the local arbiter model is aggregated to a central server, i.e. the local generator update information set and the local arbiter update information set. The central server collects the update information of the generator model and the discriminant model of all training parties, calculates and obtains a generator parameter set and a discriminant parameter set according to a certain aggregation rule (such as weighted average), and updates the initial unstructured data detection classification model.
And step 108, repeatedly iterating to obtain a generator parameter set and a discriminator parameter set until the target standard is reached.
In one embodiment, the target criteria includes a predetermined number of federal learning iterations or convergence conditions.
Step 110, obtaining an unstructured data detection classification model based on the generator parameter set and the arbiter parameter set.
In one embodiment, at the end of federal learning, the final model is combined, i.e., the average generator parameters and average arbiter parameters of the various participants are combined into the final federally generated countermeasure network model. The average generator parameters are generator parameter sets, the average discriminator parameters are discriminator parameter sets, and the final federal generation countermeasure network model is an unstructured data detection classification model.
In the unstructured data detection classification model training method, an initial unstructured data set is obtained; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set. Performing federal learning iteration according to the local generator training set and the local discriminator training set, obtaining a local generator updating information set and a local discriminator updating information set, and obtaining a generator parameter set and a discriminator parameter set according to the local generator updating information set and the local discriminator updating information set; repeatedly iterating to obtain a generator parameter set and a discriminator parameter set until reaching a target standard; an unstructured data detection classification model is obtained based on the generator parameter set and the arbiter parameter set. The method fully utilizes the electric power image data in unstructured data of each participant, performs model training in a federal learning mode, and can be used for detecting, classifying and identifying faults of the electric power image. Meanwhile, the method protects the data privacy, original image data does not need to be shared to other participants, and the safety of the data is ensured.
In one embodiment, obtaining the initial unstructured dataset comprises:
Acquiring an initial unstructured data set according to an initial participant; determining a participant according to the initial participant; the participants retain the local data of the initial unstructured dataset.
In one embodiment, during the data preparation phase, a power company or power device manager participating in federal learning prepares a local power image dataset, wherein the initial unstructured dataset contains images of different types of power device states and fault conditions. Specifically, the electric company or the electric equipment manager participating in federal learning is an initial participant, and the local electric image dataset is an initial unstructured dataset.
Further, in the participant selection phase, the utility company or power equipment manager, i.e., the participant, participating in the federal learning is determined from the initial participants. Each participant retains local data and is not shared with other participants.
In this embodiment, the initial unstructured dataset is obtained by following the initial participant; determining a participant according to the initial participant; the participants retain the local data of the initial unstructured data set, so that the effects of protecting the data privacy and structuring cooperation across tissues can be achieved. Since power unstructured image data typically contains sensitive information such as the location, topology, etc. of the power equipment. By using federal learning and generation of the countermeasure network, participants can retain data locally and only share parameter updates of the model, not share the original image data. This ensures the privacy and security of the data. Meanwhile, the federal generation countermeasure network allows cooperation among different power companies or power equipment managers to jointly establish a powerful image analysis and modeling model. Each participant can learn from the data of the other participants, so that the model has better generalization capability and accuracy.
In one embodiment, performing federal learning iterations from a local generator training set and a local arbiter training set, obtaining a local generator update information set and a local arbiter update information set includes:
selecting a training party among the participants; instructing a training party to train a local generator model according to the local generator training set, and acquiring a local generator update information set; the instruction training party trains the local discriminant model according to the local discriminant training set, and acquires the local discriminant updating information set.
In one embodiment, each participant initializes a corresponding local generator model and local discriminant model prior to conducting a federal learning iteration.
Further, a portion of the participants are selected to participate in the training of the current round, i.e., the training participants.
Specifically, the training party trains a local generator model according to the local generator training set, and acquires a local generator update information set. Wherein, the training objective function of the generator is shown in formula (1):
wherein E is x Representing the desire for a real image, E z Representing the desire for random noise, D (x) represents the output of the arbiter to the real image and G (z) represents the composite image generated by the generator from the random noise z.
Specifically, the training party trains the local discriminant model according to the local discriminant training set, and acquires the local discriminant updating information set. Wherein, the training objective function of the discriminator is shown in formula (2):
wherein E is x Representing the desire for a real image, E z Representing the desire for random noise, D (x) represents the output of the arbiter to the real image and G (z) represents the composite image generated by the generator from the random noise z.
In this embodiment, the training party is selected from the participants; instructing a training party to train a local generator model according to the local generator training set, and acquiring a local generator update information set; the instruction training party trains the local discriminant model according to the local discriminant training set, and acquires the local discriminant updating information set. The training generator is used for generating a vivid composite image so as to deceptively judge the device; the training discriminators are used to correctly distinguish between the real image and the composite image. The goal of federally generating a countermeasure network is to enable the generator to generate a realistic composite image by jointly training the generator and a discriminant model that can accurately distinguish the real image from the composite image. In this way, each participant can share knowledge of the generator and the arbiter while protecting data privacy, improving the performance of image generation and classification. Meanwhile, the federal generation countermeasure network model has good expansibility and expansibility, when a new participant joins federal learning, only the parameter update of the local model is needed to be shared, and the whole model is not needed to be retrained, so that the model can be easily adapted to the new participant and more electric unstructured image data.
In one embodiment, obtaining the generator parameter set and the arbiter parameter set from the local generator update information set and the local arbiter update information set comprises:
aggregating the local generator update information set and the local arbiter update information set to a central server; and acquiring a generator parameter set and a discriminator parameter set according to the aggregation rule.
In one embodiment, the participants aggregate the local generator update information set and the local arbiter update information set to a central server. Further, the central server calculates average parameters of the generator and the discriminator, namely a generator parameter set and a discriminator parameter set, according to the weights of the participants, and updates the initial unstructured data detection classification model according to a certain aggregation rule. Optionally, the aggregation rules include a rule of weighted averaging.
Specifically, updating the initial unstructured data detection classification model is shown in formula (3) and formula (4):
wherein G 'and D' represent generator parameters and arbiter parameters after update, G and D represent generator parameters and arbiter parameters before update, η represents a learning rate,and->Representing the gradient of the generator and arbiter objective functions.
In this embodiment, the local generator update information set and the local arbiter update information set are aggregated to the central server; and acquiring a generator parameter set and a discriminator parameter set according to the aggregation rule. Through collaboration and parameter aggregation, a utility company or device manager can co-train a powerful generation countermeasure network model for detection, classification, and fault identification of power images. Meanwhile, the method protects the data privacy, original image data does not need to be shared to other participants, and the safety of the data is ensured. Since power unstructured image data typically has complex features and patterns of variation, federal generation countermeasure networks can learn the potential distribution of the data and generate realistic composite images. Such a model may better capture implicit information in the image data, improving the analysis and modeling capabilities for power device status, faults, etc.
In one embodiment, as shown in fig. 2, there is provided an unstructured data detection classification method, using an unstructured data detection classification model, the method comprising:
step 202, an unstructured dataset is acquired.
And step 204, calling an unstructured data detection classification model, and inputting the unstructured data set into the unstructured data detection classification model to obtain an unstructured data detection classification result.
In one embodiment, the power image data to be detected, i.e., the unstructured data set, is acquired, and the unstructured data set is preprocessed, including operations such as image denoising, resizing, cropping, and the like, so as to ensure consistency and accuracy of input data. Further, in the image detection and classification stage, a trained federal generation countermeasure network model, namely an unstructured data detection classification model, is used for detecting and classifying the preprocessed unstructured data set. Specifically, an unstructured dataset to be detected is input into a generator model in an unstructured data detection classification model to generate a composite image. The real image and the composite image are then input together into a discriminant model in the unstructured data detection classification model, which can judge the authenticity of the image and give a classification result. Optionally, the detection and classification tasks include detecting power device status, fault detection, safety monitoring, and the like.
Further, according to the classification result of the discriminator model, the state of the power equipment and whether a fault exists can be judged, so that fault detection and identification are completed. The power image may be classified into a normal state or a different type of fault state, such as wire break, short circuit, overload, etc., according to the unstructured data detection classification model.
Further, safety monitoring of the power equipment can be achieved through detection and classification results of the power images. If an abnormal or fault condition is detected, an alarm can be sent out in time and corresponding maintenance and protection measures can be taken to prevent power accidents or loss.
In this embodiment, the unstructured dataset is obtained; and calling an unstructured data detection classification model, and inputting the unstructured data set into the unstructured data detection classification model to obtain an unstructured data detection classification result. In the whole flow, the method can fully utilize the electric power image data of each participant and perform model training in a federal learning mode. The application flow can improve the efficiency and accuracy of power equipment management and safety monitoring, and is beneficial to improving the stability and reliability of a power system. Meanwhile, the federal generation countermeasure network model can analyze and model the power image in real time at a local end, so that real-time monitoring and early warning functions are realized. The application can be used for detecting equipment states, fault detection, safety monitoring and other scenes in the power industry, which utilize image data to carry out digital management and upgrading.
In another embodiment, as shown in FIG. 3, an unstructured data detection classification method is provided.
Step 302, obtaining an initial unstructured data set according to an initial participant; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set.
Step 304, determining a participant according to the initial participant; the participants retain the local data of the initial unstructured dataset.
Step 306, a training party is selected among the participants.
Step 308, instructs the trainer to train the local generator model from the local generator training set to obtain a local generator update information set.
Step 310, instruct the training party to train the local discriminant model according to the local discriminant training set, and obtain the local discriminant update information set.
Step 312, aggregating the local generator update information set and the local arbiter update information set to a central server.
Step 314, obtaining a generator parameter set and a arbiter parameter set according to the aggregation rule.
Step 316, iterating to obtain the generator parameter set and the arbiter parameter set until the target standard is reached.
Step 318, obtaining an unstructured data detection classification model based on the generator parameter set and the arbiter parameter set.
At step 320, an unstructured dataset is acquired.
And step 322, invoking the unstructured data detection classification model, and inputting the unstructured data set into the unstructured data detection classification model to obtain an unstructured data detection classification result.
In the embodiment, the method uses the proposed thought of combining the generation of the countermeasure network and the federal learning, makes full use of the power image data of each participant, and carries out model training in a federal learning mode for detecting, classifying and identifying faults of the power image. Meanwhile, the method protects the data privacy, original image data does not need to be shared to other participants, and the safety of the data is ensured. The application flow can improve the efficiency and accuracy of power equipment management and safety monitoring, and is beneficial to improving the stability and reliability of a power system. The method can fully utilize distributed data, protect data privacy and provide accurate image analysis and modeling capability when solving the analysis and modeling problems of the unstructured image data of the electric power system, thereby improving the management and safety of the electric power system
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides an unstructured data detection classification model training device for realizing the above-mentioned unstructured data detection classification model training method. The implementation scheme of the device for solving the problem is similar to that described in the above method, so the specific limitation in the embodiment of the device for training the unstructured data detection classification model provided below can be referred to as the limitation of the method for training the unstructured data detection classification model hereinabove, and will not be repeated here.
In one embodiment, as shown in fig. 4, there is provided an unstructured data detection classification model training apparatus, comprising: an initial unstructured dataset acquisition module 402, a federal learning iteration module 404, a parameter set acquisition module 406, a iterative iteration module 408, and a model generation module 410, wherein:
an initial unstructured dataset acquisition module 402, configured to acquire an initial unstructured dataset; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set.
The federal learning iteration module 404 is configured to perform federal learning iteration according to the local generator training set and the local arbiter training set, and obtain a local generator update information set and a local arbiter update information set.
The parameter set obtaining module 406 is configured to obtain a generator parameter set and a arbiter parameter set according to the local generator update information set and the local arbiter update information set.
And a iterating module 408, configured to iterate repeatedly to obtain the generator parameter set and the identifier parameter set until the target standard is reached.
The model generation module 410 is configured to obtain an unstructured data detection classification model based on the generator parameter set and the arbiter parameter set.
In one embodiment, the initial unstructured dataset acquisition module 402 further comprises:
an initial unstructured data set preparation module is used for acquiring an initial unstructured data set according to an initial participant.
The participant determining module is used for determining the participant according to the initial participant; the participants retain the local data of the initial unstructured dataset.
In one embodiment, the federal learning iteration module 404 further includes:
and the training party selection module is used for selecting a training party from the participants.
The local generator model training module is used for instructing a training party to train a local generator model according to the local generator training set and obtaining a local generator update information set.
The local discriminant model training module is used for instructing a training party to train a local discriminant model according to the local discriminant training set and obtaining a local discriminant updating information set.
In one embodiment, the parameter set acquisition module 406 further includes:
and the updating information aggregation module is used for aggregating the local generator updating information set and the local discriminator updating information set to the central server.
And the aggregation rule application module is used for acquiring the generator parameter set and the discriminator parameter set according to the aggregation rule.
The modules in the unstructured data detection classification model training device can be fully or partially implemented by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
Based on the same inventive concept, the embodiment of the application also provides an unstructured data detection and classification device for realizing the above-mentioned unstructured data detection and classification method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiments of the device for detecting and classifying unstructured data provided below may be referred to the limitation of the method for detecting and classifying unstructured data hereinabove, and will not be repeated here.
In one embodiment, as shown in fig. 5, an unstructured data detection classification apparatus is provided, which includes an unstructured data set acquisition module 502 and a model invocation module 504:
an unstructured dataset acquisition module 502 for acquiring an unstructured dataset.
The model calling module 504 is configured to call the unstructured data detection classification model, input the unstructured data set into the unstructured data detection classification model, and obtain an unstructured data detection classification result.
The above-described modules in the unstructured data detection classification apparatus may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program, when executed by a processor, implements an unstructured data detection classification model training method and an unstructured data detection classification method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in FIG. 6 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (10)
1. A method for training an unstructured data detection classification model, the method comprising:
acquiring an initial unstructured dataset; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set;
performing federal learning iteration according to the local generator training set and the local discriminant training set to acquire a local generator updating information set and a local discriminant updating information set;
Acquiring a generator parameter set and a discriminator parameter set according to the local generator update information set and the local discriminator update information set;
repeatedly iterating to obtain the generator parameter set and the discriminator parameter set until reaching a target standard;
an unstructured data detection classification model is obtained based on the generator parameter set and the arbiter parameter set.
2. The method of claim 1, wherein the acquiring an initial unstructured dataset comprises:
acquiring an initial unstructured data set according to an initial participant;
determining a participant according to the initial participant; the participants retain local data of the initial unstructured dataset.
3. The method of claim 1, wherein the performing federal learning iterations from the local generator training set and the local arbiter training set to obtain a local generator update information set and a local arbiter update information set comprises:
selecting a training party among the participants;
instructing the training party to train a local generator model according to the local generator training set, and acquiring a local generator update information set;
and the training party is instructed to train the local discriminant model according to the local discriminant training set, and a local discriminant updating information set is obtained.
4. The method of claim 1, wherein said obtaining a set of generator parameters and a set of arbiter parameters from said set of local generator update information and said set of local arbiter update information comprises:
aggregating the local generator update information set and the local arbiter update information set to a central server;
and acquiring a generator parameter set and a discriminator parameter set according to the aggregation rule.
5. A method of unstructured data detection classification, the method using the unstructured data detection classification model provided in claim 1, the method comprising:
obtaining an unstructured dataset;
and calling the unstructured data detection classification model, and inputting the unstructured data set into the unstructured data detection classification model to obtain an unstructured data detection classification result.
6. An unstructured data detection classification model training apparatus, the apparatus comprising:
the initial unstructured data set acquisition module is used for acquiring an initial unstructured data set; wherein the initial unstructured dataset comprises a local generator training set and a local arbiter training set;
The federal learning iteration module is used for carrying out federal learning iteration according to the local generator training set and the local discriminant training set to obtain a local generator updating information set and a local discriminant updating information set;
the parameter set acquisition module is used for acquiring a generator parameter set and a discriminator parameter set according to the local generator update information set and the local discriminator update information set;
the repeated iteration module is used for repeatedly and iteratively acquiring the generator parameter set and the discriminator parameter set until reaching a target standard;
and the model generation module is used for acquiring an unstructured data detection classification model based on the generator parameter set and the discriminator parameter set.
7. An unstructured data detection classification device, the device comprising:
the unstructured data set acquisition module is used for acquiring an unstructured data set;
and the model calling module is used for calling the unstructured data detection classification model, inputting the unstructured data set into the unstructured data detection classification model, and obtaining an unstructured data detection classification result.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 5 when the computer program is executed.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 5.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310799513.2A CN116861290A (en) | 2023-06-30 | 2023-06-30 | Unstructured data detection classification method, model training method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310799513.2A CN116861290A (en) | 2023-06-30 | 2023-06-30 | Unstructured data detection classification method, model training method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116861290A true CN116861290A (en) | 2023-10-10 |
Family
ID=88224519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310799513.2A Pending CN116861290A (en) | 2023-06-30 | 2023-06-30 | Unstructured data detection classification method, model training method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116861290A (en) |
-
2023
- 2023-06-30 CN CN202310799513.2A patent/CN116861290A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Mayer et al. | Exposing fake images with forensic similarity graphs | |
| He et al. | A dependency graph approach for fault detection and localization towards secure smart grid | |
| CN110149327A (en) | Alarm method, device, computer equipment and the storage medium of network security threats | |
| Jain et al. | Rule‐based classification of energy theft and anomalies in consumers load demand profile | |
| CN111401722A (en) | Intelligent decision method and intelligent decision system | |
| Savola et al. | A visualization and modeling tool for security metrics and measurements management | |
| CN113486584B (en) | Method and device for predicting equipment failure, computer equipment and computer readable storage medium | |
| Al‐Naji et al. | Quality index evaluation of videos based on fuzzy interface system | |
| CN114241358A (en) | Equipment state display method, device and equipment based on digital twin transformer substation | |
| CN112364008A (en) | Equipment portrait construction method for intelligent terminal of power internet of things | |
| CN111932200A (en) | Remote bidding evaluation system | |
| CN116861290A (en) | Unstructured data detection classification method, model training method and device | |
| CN115907786A (en) | Supplier portrait construction method based on knowledge graph | |
| CN110516590A (en) | Operation or work standard prompt system based on scene Recognition | |
| Kefeng et al. | CNN Based No‐Reference HDR Image Quality Assessment | |
| CN116112200B (en) | Method, device, computer equipment and storage medium for detecting longitudinal access of power distribution network | |
| Luo et al. | A graph neural network-based digital assessment method for vocational education level of specific regions | |
| CN116866394A (en) | Security system operation method and device and computer equipment | |
| CN115527161B (en) | Abnormal discharge monitoring method, device, computer equipment and storage medium | |
| CN116797029B (en) | IDI full life cycle risk digital management method and system | |
| CN115330559B (en) | Power distribution network elasticity evaluation method and device based on information data time-space coordination | |
| CN113435802B (en) | Power transmission network monitoring planning scheme construction method, device, equipment and storage medium | |
| Bao et al. | Neural network‐based image quality comparator without collecting the human score for training | |
| CN117196306A (en) | Method, apparatus, device, storage medium and program product for determining operation risk | |
| CN118196492A (en) | Defect identification method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |