CN115203292B - Data processing method, device and equipment for industrial equipment - Google Patents
Data processing method, device and equipment for industrial equipment Download PDFInfo
- Publication number
- CN115203292B CN115203292B CN202211118959.6A CN202211118959A CN115203292B CN 115203292 B CN115203292 B CN 115203292B CN 202211118959 A CN202211118959 A CN 202211118959A CN 115203292 B CN115203292 B CN 115203292B
- Authority
- CN
- China
- Prior art keywords
- abnormal data
- node
- resource consumption
- consumption information
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 230000002159 abnormal effect Effects 0.000 claims abstract description 278
- 238000000034 method Methods 0.000 claims abstract description 123
- 238000012549 training Methods 0.000 claims abstract description 63
- 230000008569 process Effects 0.000 claims description 90
- 238000012545 processing Methods 0.000 claims description 26
- 230000010354 integration Effects 0.000 claims description 11
- 230000002123 temporal effect Effects 0.000 claims 1
- 239000003245 coal Substances 0.000 description 32
- 230000009466 transformation Effects 0.000 description 10
- 238000000605 extraction Methods 0.000 description 8
- 238000007781 pre-processing Methods 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 7
- 208000024891 symptom Diseases 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000002547 anomalous effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000005215 recombination Methods 0.000 description 2
- 230000006798 recombination Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 239000011800 void material Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/04—Manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Manufacturing & Machinery (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a data processing method, a device and equipment of industrial equipment, wherein the method comprises the following steps: acquiring abnormal data of the industrial equipment and time information of the abnormal data; the anomaly data is generated during operation of at least one node in a training model of the industrial equipment; determining target resource consumption information of a target node when the target node in the training model generates abnormal data according to the abnormal data and the time information of the abnormal data; and determining an abnormal data resource model according to the abnormal data generated by the target node, the time information of the abnormal data and the target resource consumption information when the abnormal data is generated. The embodiment of the invention realizes the rapid positioning of the abnormal data of the model.
Description
Technical Field
The present invention relates to the field of data information processing technologies, and in particular, to a data processing method, device and apparatus for an industrial device.
Background
In the graph structure relation based on industrial data processing, no matter historical data or real-time data, when a model is trained, the model is often failed because of computer resource consumption; after the training fails, the problem of the training failure is difficult to be checked, and the time period with abnormal positions is more difficult to locate according to the running condition of historical data.
Disclosure of Invention
The invention provides a data processing method, a data processing device and data processing equipment of industrial equipment. The method and the device realize the quick positioning of the abnormal data of the model.
To solve the above technical problem, an embodiment of the present invention provides the following solutions:
a data processing method of an industrial device, comprising:
acquiring abnormal data of industrial equipment and time information of the abnormal data; the abnormal data is generated in the operation process of at least one node in the training model of the industrial equipment;
determining target resource consumption information of a target node in the training model when the target node generates abnormal data according to the abnormal data and the time information of the abnormal data;
and determining an abnormal data resource model according to the abnormal data generated by the target node, the time information of the abnormal data and the target resource consumption information when the abnormal data is generated.
Optionally, the obtaining abnormal data generated in the operation process of at least one node in the training model of the industrial device and the time information of the abnormal data includes:
acquiring abnormal data generated by the at least one node in the operation process through an abnormal resource collector on each node;
and acquiring the time information for generating the abnormal data according to the time stamp information of the abnormal data.
Optionally, when it is determined that a target node in the training model generates abnormal data, the target resource consumption information of the target node includes:
acquiring resource consumption information respectively generated in the running process of at least one node in the training model;
and determining target resource consumption information when the target node generates the abnormal data in the resource consumption information respectively generated in the running process of at least one node.
Optionally, the obtaining resource consumption information respectively generated in the running process of at least one node in the training model includes:
and acquiring process resource consumption information and system resource consumption information which are respectively generated in the running process of the at least one node through the distributed remote object.
Optionally, after obtaining resource consumption information respectively generated in an operation process of at least one node in the training model, the method further includes:
and storing the resource consumption information respectively generated in the running process of the at least one node into the middleware.
Optionally, in resource consumption information respectively generated in the running process of at least one node, determining target resource consumption information when the target node generates the abnormal data includes:
and acquiring target resource consumption information of a target node when the target node in the at least one node generates the abnormal data from the middleware.
Optionally, determining an abnormal data resource model according to the abnormal data generated by the target node, the time information of the abnormal data, and the target resource consumption information when the abnormal data is generated, includes:
sequentially carrying out statistical processing on abnormal data of at least one target node, time information of the abnormal data and target resource consumption information of the target node when the abnormal data are generated through a data integration component to obtain a statistical result;
and forming an abnormal data resource model according to abnormal data in the statistical result, the time information of the abnormal data, the target resource consumption information of the target node when the abnormal data is generated, and the corresponding relation among the abnormal data, the time information of the abnormal data and the target resource consumption information.
The present invention also provides a data processing apparatus of an industrial device, comprising:
the acquisition module is used for acquiring abnormal data of the industrial equipment and time information of the abnormal data; the abnormal data is generated in the operation process of at least one node in the training model of the industrial equipment;
the processing module is used for determining target resource consumption information of a target node in the training model when the target node generates abnormal data according to the abnormal data and the time information of the abnormal data; and determining an abnormal data resource model according to the abnormal data generated by the target node, the time information of the abnormal data and the target resource consumption information when the abnormal data is generated.
The invention also provides an electronic device comprising a processor, a memory and a program or instructions stored on the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the data processing method of the industrial device as described above.
The present invention also provides a readable storage medium on which a program or instructions are stored, which when executed by a processor implement the steps of the data processing method of the industrial device as described above.
The scheme of the invention at least comprises the following beneficial effects:
according to the scheme, the abnormal data of the industrial equipment and the time information of the abnormal data are obtained; the abnormal data is generated in the operation process of at least one node in the training model of the industrial equipment; determining target resource consumption information of a target node in the training model when the target node generates abnormal data according to the abnormal data and the time information of the abnormal data; and determining an abnormal data resource model according to the abnormal data generated by the target node, the time information of the abnormal data and the target resource consumption information when the abnormal data is generated. The method and the device realize the quick positioning of the abnormal data of the model, and can finely count the abnormal data and the resource consumption condition in the historical data time period on the model node.
Drawings
FIG. 1 is a schematic flow chart of a data processing method of an industrial apparatus according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an anomaly collector according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating a process of registering and binding a distributed remote object by a server for resource collection according to an embodiment of the present invention;
fig. 4 is a schematic flow chart of a client for resource acquisition to obtain resource consumption information according to an embodiment of the present invention;
FIG. 5 is a flow chart illustrating a data processing method of an industrial plant according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating an abnormal resource collector according to an embodiment of the present invention;
FIG. 7 is a schematic flow chart of data integration according to an embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating a flow of determining a frequent coal blockage of a coal pulverizer of a thermal power plant according to an embodiment of the present invention;
FIG. 9 is a schematic flow chart illustrating a logic analysis of a process performed by a coal pulverizer in accordance with an exemplary embodiment of the present disclosure;
FIG. 10 is a schematic illustration of a training model for a coal pulverizer in accordance with an exemplary embodiment of the present invention;
FIG. 11 is a schematic illustration of an execution plan for a training model for a coal pulverizer in accordance with an exemplary embodiment of the present invention;
FIG. 12 is a schematic timing flow diagram illustrating the execution of a training model for a coal pulverizer in accordance with an exemplary embodiment of the present invention;
fig. 13 is a schematic structural diagram of a data processing apparatus of an industrial device according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
As shown in fig. 1, the present invention provides a data processing method for an industrial device, including:
and step 13, determining an abnormal data resource model according to the abnormal data generated by the node, the time information of the abnormal data and the target resource consumption information when the abnormal data is generated.
In this embodiment, after performing logical analysis on an operation mechanism of the industrial device, a training model of the industrial device may be obtained, where the training model includes at least one node, each node is configured to process data in an operation process of the training model, and by obtaining abnormal data generated by a target node in the at least one node in the operation process and time information corresponding to the abnormal data, the abnormal data in the operation process of the target node may be quickly located, and then when the abnormal data is generated, target resource consumption information of the target node is obtained, and an abnormal data resource model is determined according to the abnormal data of the target node, the time information of the abnormal data, and the target resource consumption information when the abnormal data is generated; the method and the device realize the rapid positioning of the abnormal data of the training model of the industrial equipment.
In an optional embodiment of the present invention, step 11 includes:
step 111, acquiring abnormal data generated by the at least one node in the operation process through an abnormal resource collector on each node;
and step 112, acquiring the time information for generating the abnormal data according to the time stamp information of the abnormal data.
In this embodiment, the abnormal resource collector is disposed on the node, and is configured to collect abnormal data generated during the operation of the node, when a target node monitored by the abnormal resource collector is abnormal during the operation, the abnormal data is collected, and time information of the abnormal data is located according to timestamp information of the abnormal data, so that when a training model is in a problem, troubleshooting is performed on historical data of each node of the training model, and abnormal data and time information of the node can be located quickly.
In an optional embodiment of the present invention, step 12 includes:
step 121, acquiring resource consumption information respectively generated in the operation process of at least one node in the training model;
and step 122, determining target resource consumption information when the target node generates the abnormal data in the resource consumption information respectively generated in the running process of at least one node.
In this embodiment, each node in the training model of the industrial device is used for data processing, and resource consumption information generated in the node operation process is collected during the node operation process, and the collection is preferably periodically collected;
when the abnormal resource collector monitors abnormal data for a target node, the target resource consumption information can be determined from the resource consumption information generated in the running process of the target node based on the time information corresponding to the abnormal data.
Preferably, in an alternative embodiment of the present invention, step 121 includes:
and 1211, acquiring process resource consumption information and system resource consumption information respectively generated in the running process of the at least one node through the distributed remote object.
In this embodiment, a distributed remote object (resourceinfosservice) is provided by a remote distributed service; the remote distributed service registers before the training model is run, registers the service in a JVM (Java virtual machine) process, and enables RMI (remote method invocation) service;
the remote distributed service comprises a resource acquisition server and a resource acquisition client, wherein the resource acquisition server is used for counting process resource consumption information and system resource consumption information of nodes, so that cross-service communication resource consumption information is realized; the client side of the resource acquisition is used for the server side to realize the distributed object calling, namely the resource consumption information provided by the server side is obtained through the distributed remote object;
it should be noted that the client for resource collection further includes a timer task, and the timer task collects resource consumption information of the server for resource collection according to a preset period.
The following describes remote distributed services:
as shown in fig. 2, the remote distributed service is applied to the resource collector, and includes a resource collection server (RMIServer) and a resource collection client (RMIClient), and trains the training model in the form of jar (Java archive) package, initializes the training model, and then starts and configures the remote distributed service;
the distributed remote object in the server side for resource collection comprises a JMX component (Java management extension component) and an OSHI component (server and application monitoring library component), wherein the JMX component is used for collecting process resource consumption information (such as a memory pool, a thread, a garbage collection grid communication GC, a heap memory, a non-heap memory and the like), and the OSHI component is used for collecting system resource consumption information (such as a global memory, CPU (central processing unit) use and load, a monitoring process occupying CPU and the like); the service end of resource collection is the internal service of the node, before using, the internal service preferably will register, when registering, need to provide URL (Uniform resource Locator) and distributed remote object (resource info service) of RMI protocol (remote method call protocol), implement and interact with the client end of resource collection;
the resource acquisition client side can periodically access the resource acquisition server side through a Timer (Timer) and a Timer task (TimeTask), acquire and count resource consumption information of the resource acquisition server side, and cache the resource consumption information.
Here, a process of the server for resource collection is described:
as shown in fig. 3, before acquiring resource consumption information, a resource acquisition server (RMIServer) designates a port through a Registry (Registry) to create a Registry, where the Registry is used to designate a port to be monitored when the resource acquisition server performs registration;
after the registration is completed, a resource acquisition server (RMIServer) binds a distributed remote object (ResourceInfoService) through a Naming (registry encapsulation), wherein the Naming comprises a remote object registry and a method for acquiring the distributed remote object, the distributed remote object comprises a JmxMonitor (Java management extension monitor) and an OshiMonitor (server and application monitoring library monitor), the JmxMonitor monitors and counts process resource consumption information of a JVM process, and the OshiMonitor monitors system resource consumption information (comprising a global memory, a CPU, a process and the like).
In a specific embodiment, the process of registering and binding distributed remote objects is (following the symbol "//" below followed by annotation content):
publicvoid start(){
try {
ResourceInfoServiceresourceInfoService=new ResourceInfoServiceImpl();
LocateRegistry.createRegistry(port);
// specify the port to listen
Naming.rebind(this.rmiUrl(), resourceInfoService);
// binding distributed remote objects
System.out.println("start server, port is 1800");
} catch (Exception e) {
e.printStackTrace();}}
The process of acquiring resource consumption information is (annotation contents are followed by the following symbol "//"):
public class ResourceInfoServiceImpl extends UnicastRemoteObject implements ResourceInfoService {
publicResourceInfoServiceImpl() throws RemoteException {}
@Override
publicJvmInfocollectJvmInfo() throws RemoteException {
acquiring process resource consumption information from Jmxmonitor by distributed remote object
returnJmxMonitor.getJvmInfo();}
@Override
publicComputerInfocollectComputerInfo() throws RemoteException {
Acquiring system resource consumption information from OshiMonitor by distributed remote object
return OshiMonitor.getComputeInfo();}}。
Here, a processing procedure of the client for resource collection is explained:
as shown in fig. 4, the resource acquisition client is a remote method invocation client (RMIClient), periodically collects resource consumption information of the resource acquisition server through a Timer (Timer), sends a create Timer to the Timer, creates a Task (Task) through the Timer, and returns to the resource acquisition client after the creation is completed; a timer task (TimerTask) is an asynchronous thread and is used for asynchronously collecting resource consumption information;
after the timer task is established, RMIClient traverses the registry, starts the timer task for each registry and runs, acquires resource consumption information of a service end for resource acquisition through a distributed remote object (resource info service), and caches the resource consumption information into a remote dictionary service (redis).
In another specific embodiment, the process of the resource collection client periodically obtaining the resource consumption information from the resource collection server is as follows (the following symbol "//" is followed by annotation content):
private static List<String>nameingUrls = new LinkedList<>();
public void collect() {
Timer timer = new Timer();
for (String nameingUrl : nameingUrls) {
timer.schedule(new MonitorTimeTask(nameingUrl), 0, 1000);}}
private static class MonitorTimeTask extends TimerTask {
private final String nameingUrl;
publicMonitorTimeTask(String nameingUrl) {
this.nameingUrl = nameingUrl;}
@Override
public void run() {
v/running timer tasks for each registry
try {
// "rmi://localhost:1800/jmx"
ResourceInfoServiceresourceInfoService = (ResourceInfoService) Naming.lookup(this.nameingUrl);
JvmInfojvmInfo = resourceInfoService.collectJvmInfo();
v/Process resource consumption information Collection by distributed remote objects
ComputerInfocomputerInfo = resourceInfoService.collectComputerInfo();
V. collecting System resource consumption information by distributed remote objects, caching
} catch (Exception ae) {
}}。
In an optional embodiment of the present invention, after step 122, the method further includes:
and 123, storing the resource consumption information respectively generated in the running process of the at least one node into the middleware.
In this embodiment, storing the resource consumption information respectively generated in the node operation process to the middleware is implemented by periodically storing, by the resource acquisition client, the acquired process resource consumption information and system resource consumption information to the middleware according to a timer, where the middleware is preferably a storage database (e.g., redis); and the middleware storing the resource consumption information can be called by the abnormal resource collector.
In an alternative embodiment of the present invention, step 122 includes:
step 1221, obtaining target resource consumption information of the node when the target node in the at least one node generates the abnormal data from the middleware.
In this embodiment, after the abnormal resource collector finds that the node has abnormal data in the monitoring process of the node, and collects the time information of generating the abnormal data, the target resource consumption information of the node in generating the abnormal data is acquired from the middleware according to the abnormal data and the time information, so that the time information and the resource consumption information of generating the abnormal data are uniformly collected and managed.
As shown in FIG. 5, in a specific embodiment, for each node of the industrial model of the graph structure, an abnormal collector (i.e., an abnormal resource collector) is provided, the industrial model is further provided with a remote method call Server RMI _ Server (i.e., a resource collection Server), the resource collection Server periodically collects resource consumption information on the node of the industrial model, and calls a Client RMI _ Client (i.e., a resource collection Client) by a remote method to periodically and asynchronously collect resource consumption information from the RMI _ Server based on a registry (RMI:// localhost: 1800/jmx), and writes the collected resource consumption information into a remote dictionary service redis;
when the abnormal data of the node are acquired, the abnormal collector acquires the time information of the abnormal data, accesses redis, acquires the resource consumption data of the abnormal data, integrates the abnormal data, the time information and the resource consumption data, and stores the abnormal data, the time information and the resource consumption data into the database.
In an alternative embodiment of the present invention, step 13 includes:
In the embodiment, the abnormal resource collector comprises a data integration component and an abnormal writer, the data integration component counts the abnormal data monitored by the abnormal resource collector, the time information of the abnormal data and the target resource consumption information to obtain a statistical result, determines the corresponding relation among the abnormal data, the time information of the abnormal data and the target resource consumption information according to the operator information, the node information and the model information on the node, and integrates the abnormal data, the time information of the abnormal data and the target resource consumption information into an abnormal data resource model based on the corresponding relation;
and the exception writer writes the exception data resource model obtained by integrating the data integration component into the storage database.
It should be noted that the resource consumption data includes process resource consumption information and system resource consumption information;
the process resource consumption information is shown in the following table:
TABLE 1
As can be seen from table 1, the process resource consumption information includes: the process resource consumption information in the node running process of the current training model comprises at least one of the following items:
memory pools (MemoryPool);
a memory (including heap memory memoryussage and non-heap memory nopherapmyussage);
garbage collection network communication (garbagcollector, abbreviated as GC);
thread (ThreadMXBean);
the system resource consumption information is shown in the following table:
TABLE 2
As can be seen from table 2, the system resource consumption information includes at least one of:
CPU usage and load (central processor) of industrial equipment;
global memory usage (GlobalMemory);
the process takes up CPU data (ospprocess).
The anomaly data are shown in the following table:
TABLE 3
As can be seen from table 3, the anomaly data includes at least one of the following anomaly information;
exception data (exception);
an exception type (type);
time information (timestamp) of the abnormal data.
Other data related to the nodes of the anomalous data are shown in the following table:
TABLE 4
As can be seen from table 4, the other data related to the nodes of the anomalous data comprises at least one of the following:
the start time (Endtime) of the anomaly data;
end time of abnormal data (Starttime);
a node operator (operator);
a node (node);
model (model).
As shown in fig. 6, in a specific embodiment, the abnormal resource collector includes an abnormal resource data integration component and an abnormal writer (PostgresSink), the abnormal resource collector is disposed on a node of the industrial model, monitors whether the node generates abnormal data during operation, when the node generates the abnormal data, obtains time information corresponding to the abnormal data, accesses a database storing resource consumption information, obtains target resource consumption information, and integrates the abnormal data, the time information, and the target resource consumption information through the abnormal resource data integration component to obtain an abnormal data resource model; storing the abnormal data resource model into a database through PostgressInk; the database is used for storing an abnormal data resource model of the industrial model;
as shown in fig. 7, the abnormal data, the time information and the target resource consumption information are integrated, i forms a wide table (i.e. an abnormal data resource model), wherein the target resource consumption information includes process resource consumption information and system resource consumption information of the JVM process of the industrial model.
As shown in fig. 8 to 12, in another specific embodiment, due to frequent coal blockage of a coal mill of a certain thermal power plant, according to the actual demand and problem background of the coal mill, the target, the operating condition, and the abnormal phenomenon and mechanism of a training model of the coal mill of the power plant are determined to achieve the target, and the target, the operating condition, and the abnormal phenomenon and mechanism of the coal mill of the power plant are shown in the following table:
TABLE 5
Analyzing a model execution process corresponding to the problem that the coal mill of a certain thermal power plant is blocked and frequently generates coal based on expert experience to obtain a judgment flow chart of the model shown in fig. 8, comprehensively judging whether the current rise is higher than a first preset threshold value, whether the unit power consumption of the coal supply is higher than a second preset threshold value, whether the inlet air volume is lower than a preset standard value and is higher than a third preset threshold value, triggering early warning when the current, the coal supply and the outlet air volume are higher than the corresponding preset threshold values, and not triggering early warning when the current, the coal supply and the outlet air volume are higher than the corresponding preset threshold values; wherein, the values of the current, the coal feeding quantity and the outlet air quantity are all between [0,1 ];
as shown in fig. 9, the execution process is converted into a training model of the industrial equipment through logic analysis, and the execution process comprises four data processing stages: data preprocessing, feature extraction, symptom transformation and studying and judging logic; the data of the coal mill are obtained according to the read coal mill monitoring data, and the obtained coal mill monitoring data comprise: the current (Cerent), the Coal feeding quantity (Coal), the inlet air quantity (InWindSum) and the like, the time window for acquiring the monitoring data of the Coal mill is 30 minutes, and the interval is 1 minute;
performing data preprocessing on the coal mill data, namely performing data downsampling and data abnormal value processing through a preprocessing operator to obtain first data, wherein the data downsampling is to perform downsampling on the coal mill data by using platform service; the abnormal data value processing is to perform deletion (interpolation) and zero value (removal/interpolation) on the coal mill data;
performing feature extraction on the first data through a feature operator (change _ rate) to obtain unit power consumption and change rate of the coal mill as second data;
performing symptom transformation on the second data through a symptom operator (characteristic special symptom), determining a first transformation result that the current in the second data is remarkably or slowly increased within the range of 0 to 1, a second transformation result that the inlet air volume is remarkably or slowly decreased within the range of 0 to 1, and a third transformation result that the unit power consumption is high within the range of 0 to 1, and obtaining third data after symptom transformation based on the first transformation result, the second transformation result and the third transformation result;
carrying out judging logical operation on the third data through a judging operator (weight matrix), namely a 0.3+ b 0.4+ c 0.3, so as to obtain fourth data;
determining a coal blockage risk value (0~1) of the coal mill based on the fourth data, and further giving early warning to the coal mill;
as shown in fig. 10 and 11, further, performing logic analysis based on a model execution process corresponding to a problem of frequent coal mill blockage to form a training model shown in fig. 10, where the training model includes 6 nodes, and 4 nodes (data preprocessing, feature extraction, symptom transformation, and study and judgment logic) are used for data processing; when the training model is in operation, a model execution plan as shown in fig. 11 is formed, and the model execution plan includes:
initializing the resource collector through a server of the resource collector, registering a redisClinet and an abnormal resource collector, and storing resource consumption information collected by the server into a remote dictionary service redis through a client; initializing the execution environment of the model, constructing an input source, initializing each data preprocessing node, extracting the features through the data feature extraction nodes, and initializing an early warning output source; when the nodes of the training model run, the resource consumption information of each node is stored through the remote dictionary service Redis client corresponding to the client, when the abnormal resource collector monitors that abnormal data are generated on the nodes, the target resource consumption information corresponding to the abnormal data can be obtained through the Redis client on the nodes, and meanwhile, the abnormal data, the time information and the target resource consumption information are integrated through the abnormal resource collector to obtain an abnormal data resource model.
As shown in fig. 12, in the process of executing the time sequence processing by the training model, a resource collector is initialized, a server for collecting resources is created in the resource collector, and the server for collecting resources is operated; then initializing a flink execution environment; setting a data input source;
and then initializing the data processing nodes for the two data processing nodes, namely sending data to the data input source, constructing a node processor of the data preprocessing nodes, initializing a redis database and a resource acquisition client by the node processor, initializing an abnormal resource acquisition unit (namely, registering the resource acquisition unit and the abnormal resource acquisition unit through an operator node), setting the data output source, and starting model execution.
Wherein the model execution code segment is (the annotation content follows the symbol "///" below):
v/initialize resource collector
RCCollectorrcCollector = new RCCollector();
// register and initiate RMIServer
RmiServerrmiServer = rcCollector.buildRmiServer("localhost", 1800);
rmiServer.start();
// creating a stream execution Environment
StreamExecutionEnvironmentenv = StreamExecutionEnvironment.getExecutionEnvironment();
// Create kafka data type
DataStreamSource<String>dataStreamSource = env.addSource(new KafkaSource());
// setting periodic Window
DataStream<String>dataStream = dataStreamSource.assignTimestampsAndWatermarks(
WatermarkStrategy.forMonotonousTimestamps());
SingleOutputStreamOperator<List<Point>>mapDataStream = dataStream.map(new TranMapFunction());
// data preprocessing
DataStream<List<Point>>pretreatmentDataStream = mapDataStream.process(
newNodeProcessor(new RedisClient(), new AbnormalCollector()));
// data feature extraction
DataStream<List<Point>>featuresDataStream = pretreatmentDataStream.process(
newNodeProcessor(new RedisClient(), new AbnormalCollector()));
// symptom conversion
DataStream<List<Point>> process = featuresDataStream.process(
newNodeProcessor(new RedisClient(), new AbnormalCollector()));
V/logic study and judgment, alarm output
mapDataStream.addSink(new KafkaSink());
// Start model training
env.execute();
The abnormal data and the resource consumption condition on the historical data time period on the training model node of the coal mill can be precisely counted through the training model execution time sequence.
In addition, the data feature extraction node and the initialization data preprocessing node send data to the data input source and construct a node processor of the data feature extraction node, the node processor initializes a redis database and a resource acquisition client, initializes an abnormal resource collector (namely, an operator node, a registered resource collector and an abnormal resource collector), sets a data output source, and starts model execution. The abnormal data on the data feature extraction nodes can be quickly positioned in the operation process.
According to the embodiment of the invention, the abnormal data of the industrial equipment and the time information of the abnormal data are obtained; the abnormal data is generated in the operation process of at least one node in the training model of the industrial equipment; determining target resource consumption information of a target node in the training model when the target node generates abnormal data according to the abnormal data and the time information of the abnormal data; and determining an abnormal data resource model according to the abnormal data generated by the target node, the time information of the abnormal data and the target resource consumption information when the abnormal data is generated. The method and the device realize the quick positioning of the abnormal data of the model, and can finely count the abnormal data and the resource consumption condition in the historical data time period on the model node.
As shown in fig. 13, the present invention also provides a data processing apparatus 130 of an industrial device, including:
an obtaining module 131, configured to obtain abnormal data of an industrial device and time information of the abnormal data; the abnormal data is generated in the operation process of at least one node in the training model of the industrial equipment;
the processing module 132 is configured to determine, according to the abnormal data and the time information of the abnormal data, target resource consumption information of a target node in the training model when the target node generates the abnormal data; and determining an abnormal data resource model according to the abnormal data generated by the target node, the time information of the abnormal data and the target resource consumption information when the abnormal data is generated.
Optionally, the obtaining abnormal data generated in the operation process of at least one node in the training model of the industrial device and the time information of the abnormal data includes:
acquiring abnormal data generated by the at least one node in the running process through an abnormal resource collector on each node;
and acquiring the time information for generating the abnormal data according to the time stamp information of the abnormal data.
Optionally, when determining that the target node in the training model generates abnormal data, the target resource consumption information of the target node includes:
acquiring resource consumption information respectively generated in the operation process of at least one node in the training model;
and determining target resource consumption information when the target node generates the abnormal data in the resource consumption information respectively generated in the running process of at least one node.
Optionally, the obtaining resource consumption information respectively generated in the running process of at least one node in the training model includes:
and acquiring process resource consumption information and system resource consumption information which are respectively generated in the running process of the at least one node through the distributed remote object.
Optionally, after obtaining resource consumption information respectively generated in an operation process of at least one node in the training model, the method further includes:
and storing the resource consumption information respectively generated in the running process of the at least one node into the middleware.
Optionally, in resource consumption information respectively generated in the running process of at least one node, determining target resource consumption information when the target node generates the abnormal data includes:
and acquiring target resource consumption information of a target node when the target node in the at least one node generates the abnormal data from the middleware.
Optionally, determining an abnormal data resource model according to the abnormal data generated by the target node, the time information of the abnormal data, and the target resource consumption information when the abnormal data is generated, includes:
sequentially carrying out statistical processing on abnormal data of at least one target node, time information of the abnormal data and target resource consumption information of the target node when the abnormal data are generated through a data integration component to obtain a statistical result;
and forming an abnormal data resource model according to abnormal data in the statistical result, the time information of the abnormal data, the target resource consumption information of the target node when the abnormal data is generated, and the corresponding relation among the abnormal data, the time information of the abnormal data and the target resource consumption information.
It should be noted that the apparatus is an apparatus corresponding to the above method, and all the implementations in the above method embodiment are applicable to the embodiment of the apparatus, and the same technical effects can be achieved.
An embodiment of the present invention provides an electronic device, including a processor, a memory, and a program or an instruction stored on the memory and executable on the processor, wherein the program or the instruction, when executed by the processor, implements the steps of the method for determining the abnormal data resource model of the industrial device as described above.
It should be noted that the electronic device is an electronic device corresponding to the method, and all implementation manners in the embodiment of the method are applicable to the embodiment of the electronic device, and the same technical effect can be achieved.
Embodiments of the present invention also provide a readable storage medium storing instructions which, when executed on a computer, cause the computer to perform the steps of the abnormal data resource model determination method for an industrial device as described above.
It should be noted that the readable storage medium is a readable storage medium corresponding to the method, and all implementation manners in the method embodiments are applicable to the embodiment of the readable storage medium, and the same technical effect can be achieved.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.
Furthermore, it is to be noted that in the device and method of the invention, it is obvious that the individual components or steps can be decomposed and/or recombined. These decompositions and/or recombinations are to be regarded as equivalents of the present invention. Also, the steps of performing the series of processes described above may naturally be performed chronologically in the order described, but need not necessarily be performed chronologically, and some steps may be performed in parallel or independently of each other. It will be understood by those skilled in the art that all or any of the steps or elements of the method and apparatus of the present invention may be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or any combination thereof, which can be implemented by those skilled in the art using their basic programming skills after reading the description of the present invention.
Thus, the objects of the invention may also be achieved by running a program or a set of programs on any computing device. The computing device may be a well-known general purpose device. The object of the invention is thus also achieved solely by providing a program product comprising program code for implementing the method or the apparatus. That is, such a program product also constitutes the present invention, and a storage medium storing such a program product also constitutes the present invention. It is to be understood that the storage medium may be any known storage medium or any storage medium developed in the future. It is further noted that in the apparatus and method of the present invention, it is apparent that each component or step can be decomposed and/or recombined. These decompositions and/or recombinations are to be regarded as equivalents of the present invention. Also, the steps of executing the series of processes described above may naturally be executed chronologically in the order described, but need not necessarily be executed chronologically. Some steps may be performed in parallel or independently of each other.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (7)
1. A data processing method of an industrial device, comprising:
acquiring abnormal data of industrial equipment and time information of the abnormal data; the anomaly data being of an industrial plant
Generated during the operation of at least one node in the training model;
determining a target node product in the training model according to the abnormal data and the time information of the abnormal data
When abnormal data is generated, target resource consumption information of the target node is generated;
generating abnormal data according to the abnormal data generated by the target node, the time information of the abnormal data and the generated abnormal data
Determining an abnormal data resource model according to the temporal target resource consumption information;
when determining that the target node in the training model generates abnormal data, the target resource consumption information of the target node includes:
acquiring resource consumption information respectively generated in the running process of at least one node in the training model;
determining target resource consumption information when the target node generates the abnormal data in the resource consumption information respectively generated in the running process of at least one node;
the method for acquiring the resource consumption information respectively generated in the operation process of at least one node in the training model comprises the following steps:
acquiring process resource consumption information and system resource consumption information which are respectively generated in the running process of at least one node through a distributed remote object;
determining an abnormal data resource model according to abnormal data generated by the target node, time information of the abnormal data and target resource consumption information when the abnormal data is generated, wherein the determining of the abnormal data resource model comprises the following steps:
sequentially carrying out statistical processing on abnormal data of at least one target node, time information of the abnormal data and target resource consumption information of the target node when the abnormal data are generated through a data integration component to obtain a statistical result;
and forming an abnormal data resource model according to abnormal data in the statistical result, the time information of the abnormal data, the target resource consumption information of the target node when the abnormal data is generated, and the corresponding relation among the abnormal data, the time information of the abnormal data and the target resource consumption information.
2. The data processing method of the industrial equipment according to claim 1, wherein obtaining abnormal data generated during the operation of at least one node in the training model of the industrial equipment and time information of the abnormal data comprises:
acquiring abnormal data generated by the at least one node in the running process through an abnormal resource collector on each node;
and acquiring the time information for generating the abnormal data according to the time stamp information of the abnormal data.
3. The data processing method of industrial equipment according to claim 1, wherein after acquiring resource consumption information respectively generated during operation of at least one node in the training model, the method further comprises:
and storing the resource consumption information respectively generated in the running process of the at least one node into the middleware.
4. The data processing method of the industrial equipment according to claim 3, wherein the determining the target resource consumption information when the target node generates the abnormal data, from the resource consumption information respectively generated during the operation of at least one node, comprises:
and acquiring target resource consumption information of a target node when the target node in the at least one node generates the abnormal data from the middleware.
5. A data processing apparatus of an industrial device, comprising:
the acquisition module is used for acquiring abnormal data of the industrial equipment and time information of the abnormal data; the abnormal data is generated in the operation process of at least one node in the training model of the industrial equipment;
the processing module is used for determining target resource consumption information of a target node in the training model when the target node generates abnormal data according to the abnormal data and the time information of the abnormal data; determining an abnormal data resource model according to abnormal data generated by the target node, time information of the abnormal data and target resource consumption information when the abnormal data is generated;
when determining that the target node in the training model generates abnormal data, the target resource consumption information of the target node includes:
acquiring resource consumption information respectively generated in the operation process of at least one node in the training model;
determining target resource consumption information when the target node generates the abnormal data in the resource consumption information respectively generated in the running process of at least one node;
the method for acquiring resource consumption information respectively generated in the operation process of at least one node in the training model comprises the following steps:
acquiring process resource consumption information and system resource consumption information which are respectively generated in the running process of the at least one node through a distributed remote object;
determining an abnormal data resource model according to abnormal data generated by the target node, time information of the abnormal data and target resource consumption information when the abnormal data is generated, wherein the determining of the abnormal data resource model comprises the following steps:
sequentially carrying out statistical processing on abnormal data of at least one target node, time information of the abnormal data and target resource consumption information of the target node when the abnormal data are generated through a data integration component to obtain a statistical result;
and forming an abnormal data resource model according to abnormal data in the statistical result, the time information of the abnormal data, the target resource consumption information of the target node when the abnormal data is generated, and the corresponding relation among the abnormal data, the time information of the abnormal data and the target resource consumption information.
6. An electronic device comprising a processor, a memory and a program or instructions stored on the memory and executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the data processing method of the industrial device according to any one of claims 1 to 4.
7. A readable storage medium, characterized in that it stores thereon a program or instructions which, when executed by a processor, implement the steps of the data processing method of an industrial plant according to any one of claims 1 to 4.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211118959.6A CN115203292B (en) | 2022-09-15 | 2022-09-15 | Data processing method, device and equipment for industrial equipment |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211118959.6A CN115203292B (en) | 2022-09-15 | 2022-09-15 | Data processing method, device and equipment for industrial equipment |
Publications (2)
Publication Number | Publication Date |
---|---|
CN115203292A CN115203292A (en) | 2022-10-18 |
CN115203292B true CN115203292B (en) | 2022-11-25 |
Family
ID=83573349
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211118959.6A Active CN115203292B (en) | 2022-09-15 | 2022-09-15 | Data processing method, device and equipment for industrial equipment |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115203292B (en) |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108667855A (en) * | 2018-07-19 | 2018-10-16 | 百度在线网络技术(北京)有限公司 | Network traffic anomaly monitor method, apparatus, electronic equipment and storage medium |
WO2019148715A1 (en) * | 2018-02-01 | 2019-08-08 | 平安科技(深圳)有限公司 | Information processing method and apparatus, and computer device and storage medium |
CN110597984A (en) * | 2019-08-12 | 2019-12-20 | 大箴(杭州)科技有限公司 | Method and device for determining abnormal behavior user information, storage medium and terminal |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102238062A (en) * | 2010-04-20 | 2011-11-09 | 中兴通讯股份有限公司 | Exception handling method and system for network processor |
CN104244307B (en) * | 2014-09-18 | 2018-06-01 | 大唐移动通信设备有限公司 | Anomalous event reports, processing method, device, base station and management server |
JP2017022557A (en) * | 2015-07-10 | 2017-01-26 | 株式会社日立製作所 | Abnormality detection system |
CN107276851B (en) * | 2017-06-26 | 2019-12-13 | 中国信息安全测评中心 | Node abnormity detection method and device, network node and console |
EP3673375B1 (en) * | 2017-10-13 | 2022-04-06 | Huawei Technologies Co., Ltd. | System and method for cloud-device collaborative real-time user usage and performance abnormality detection |
CN108009036B (en) * | 2017-11-17 | 2021-10-01 | 亚信科技(中国)有限公司 | Method for positioning operation causing data abnormity and server |
CN112817785A (en) * | 2019-11-15 | 2021-05-18 | 亚信科技(中国)有限公司 | Anomaly detection method and device for micro-service system |
CN111738412A (en) * | 2020-05-28 | 2020-10-02 | 江门职业技术学院 | Big data exception mining method, system and storage medium for incomplete network |
CN112697267A (en) * | 2020-11-11 | 2021-04-23 | 杭州未名信科科技有限公司 | Abnormal vibration detection device for industrial equipment |
CN113064765B (en) * | 2021-04-26 | 2023-09-05 | 杭州海康威视数字技术股份有限公司 | Node exception handling method, device, electronic equipment and machine-readable storage medium |
CN113609210B (en) * | 2021-08-23 | 2022-07-12 | 广州梦源信息科技有限公司 | Big data visualization processing method based on artificial intelligence and visualization service system |
-
2022
- 2022-09-15 CN CN202211118959.6A patent/CN115203292B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019148715A1 (en) * | 2018-02-01 | 2019-08-08 | 平安科技(深圳)有限公司 | Information processing method and apparatus, and computer device and storage medium |
CN108667855A (en) * | 2018-07-19 | 2018-10-16 | 百度在线网络技术(北京)有限公司 | Network traffic anomaly monitor method, apparatus, electronic equipment and storage medium |
CN110597984A (en) * | 2019-08-12 | 2019-12-20 | 大箴(杭州)科技有限公司 | Method and device for determining abnormal behavior user information, storage medium and terminal |
Also Published As
Publication number | Publication date |
---|---|
CN115203292A (en) | 2022-10-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Chunduri et al. | Characterization of MPI usage on a production supercomputer | |
Ren et al. | Google-wide profiling: A continuous profiling infrastructure for data centers | |
CN110083436A (en) | A kind of business datum real-time monitoring system and method based on Java bytecode enhancing technology | |
CN107111527B (en) | Data stream processing language for analytical instrumentation software | |
US10116534B2 (en) | Systems and methods for WebSphere MQ performance metrics analysis | |
JP2021519988A (en) | Alarm log compression methods, devices and systems, and storage media | |
EP2831740B1 (en) | Logical grouping of profile data | |
US20100251263A1 (en) | Monitoring of distributed applications | |
EP2324406A1 (en) | Method and apparatus for monitoring the performance of a power delivery control system | |
CN102999314A (en) | Immediate delay tracker tool | |
EP4020218B1 (en) | Analyzing large-scale data processing jobs | |
CN107357731B (en) | Monitoring, analyzing and processing method for core dump problem generated by process | |
CN107807872A (en) | A kind of power transmission and transformation system method for monitoring operation states | |
Martino et al. | Logdiver: A tool for measuring resilience of extreme-scale systems and applications | |
US20120185441A1 (en) | Efficient data collection mechanism in middleware runtime environment | |
CN111339052A (en) | Unstructured log data processing method and device | |
Šor et al. | Memory leak detection in Plumbr | |
Stefanov et al. | A review of supercomputer performance monitoring systems | |
CN115203292B (en) | Data processing method, device and equipment for industrial equipment | |
Yang et al. | Statistical data reduction for efficient application performance monitoring | |
JP2008108154A (en) | Management system for operation performance information | |
CN110011845B (en) | Log collection method and system | |
JP2004348640A (en) | Method and system for managing network | |
CN114387123B (en) | Data acquisition management method | |
Cirstoiu et al. | Monitoring, accounting and automated decision support for the alice experiment based on the MonALISA framework |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |