CN113190403A - Monitoring system, computer equipment, terminal and medium for operation state of big data platform - Google Patents
Monitoring system, computer equipment, terminal and medium for operation state of big data platform Download PDFInfo
- Publication number
- CN113190403A CN113190403A CN202110434297.2A CN202110434297A CN113190403A CN 113190403 A CN113190403 A CN 113190403A CN 202110434297 A CN202110434297 A CN 202110434297A CN 113190403 A CN113190403 A CN 113190403A
- Authority
- CN
- China
- Prior art keywords
- data platform
- big data
- node
- information
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 100
- 238000006243 chemical reaction Methods 0.000 claims description 54
- 238000010276 construction Methods 0.000 claims description 37
- 230000005540 biological transmission Effects 0.000 claims description 25
- 238000004458 analytical method Methods 0.000 claims description 18
- 238000000034 method Methods 0.000 claims description 16
- 238000004422 calculation algorithm Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 230000007704 transition Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 3
- 238000001514 detection method Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000006872 improvement Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 235000019580 granularity Nutrition 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
Abstract
The invention belongs to the technical field of platform monitoring, and discloses a monitoring system, computer equipment, a terminal and a medium for the running state of a big data platform. The invention realizes the pre-deployment of the monitoring nodes, and is convenient for monitoring the platform operation information; the operation flow of the platform is acquired before the deployment of the monitoring nodes, the deployed monitoring nodes can be in one-to-one correspondence with each stage of the platform operation, the operation information of each stage of the platform can be acquired more accurately, the problem of low accuracy is solved, the operation state of the big data platform can be accurately monitored, and the operation efficiency of the big data platform is also ensured.
Description
Technical Field
The invention belongs to the technical field of platform monitoring, and particularly relates to a monitoring system, computer equipment, a terminal and a medium for the running state of a big data platform.
Background
At present: with the continuous improvement of social informatization degree, massive and real-time data are generated in various service fields. At present, big data analysis is mainly conducted in an unstructured mode, and a single-machine storage space and operational capacity are difficult to meet requirements, so that big data systems based on distributed hadoop clusters, spark clusters, storm clusters and the like are widely applied. The large data cluster system is usually deployed with hundreds of nodes, and the expansion of the node scale not only makes the cluster resource configuration and service deployment maintenance of the large data platform more difficult, but also makes tasks of submitting, scheduling, retrying, cancelling alarms and the like of the computing operation of the large data platform time-consuming and labor-consuming.
The monitoring is an important component of the big data platform, the dynamic and complexity of the big data system bring a lot of difficulties to the monitoring of the running state of the big data platform, how to effectively monitor and pre-warn the cluster software and hardware resources and the operation with different granularities, and timely take measures when a fault occurs are the key to improve the calculation accuracy and timeliness of the big data platform. The existing scheme for monitoring the big data platform is to acquire the running state of the big data platform by monitoring the service condition of each service component of the big data platform, so that the specific service requirement of the big data platform is difficult to monitor, and the accuracy for acquiring the running state is low.
Through the above analysis, the problems and defects of the prior art are as follows: the existing scheme for monitoring the big data platform is to acquire the running state of the big data platform by monitoring the service condition of each service component of the big data platform, so that the specific service requirement of the big data platform is difficult to monitor, and the accuracy for acquiring the running state is low.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a monitoring system, computer equipment, a terminal and a medium for the running state of a large data platform.
The invention is realized in this way, a monitoring system of the running state of the big data platform, the monitoring system of the running state of the big data platform includes:
the system comprises a platform information acquisition module, an information conversion and transmission module, an information receiving and conversion module, an information analysis module, a central control module, an operation flow acquisition module, a monitoring node deployment module, an operation monitoring module, a task type acquisition module and an operation state judgment module;
the platform information acquisition module is connected with the central control module and used for acquiring the large data platform construction information through a platform information acquisition program to obtain the large data platform construction information;
the information conversion and transmission module is connected with the central control module and is used for converting the large data platform construction information through an information conversion and transmission program and transmitting the converted information;
the conversion of the big data platform construction information and the transmission of the conversion information are carried out through the information conversion and transmission program, and the method comprises the following steps:
determining a linear relationship between a physical quantity of the a/D conversion object and the corresponding voltage value;
judging whether the voltage value measured under the current physical quantity has deviation, and if so, carrying out A/D conversion calibration by the main control machine;
after the calibration is finished, converting the big data platform construction information to obtain conversion information;
the conversion of the big data platform construction information to obtain conversion information comprises the following steps:
connecting and setting parameters of a main control machine and testing the operating characteristics of the main control machine; the operation characteristic of the main control computer of the test comprises the following steps: the main control machine is controlled and corrected under the speed mode, so that the stable performance of the speed inner ring is ensured; the designed MFAC control algorithm is built on an upper PC through Simulink of cSPACE; running a compiling module in the MFAC master controller; automatically generating DSP codes by using an MFAC algorithm; downloading the codes to a digital signal processor through a USB interface of an upper PC for operation, generating a voltage output signal through a main control computer, and driving the main control computer to operate;
the MFAC control method is realized by programming:
calculating the output u (k) of the main control computer,
wherein the MFAC master control unit u (k) is partially based on Simulink of cSPACEEstablishing that lambda is more than 0 and is a weight coefficient used for limiting the change of the control input quantity; rho epsilon (0, 1)]The step size factor is additionally added, so that the algorithm has stronger flexibility and generality;
is an estimate of phi (k) at time k; y (k +1) is the desired output signal; u (k), y (k) respectively represent the input and output of the system at time k;
the MFAC master control unit u (k) is built based on Simulink of cSPACE in part, and comprises the following components:
u (k) is subjected to a time delay module to obtain u (k-1); a Sine Wave module in the cSPACE gives a sinusoidal position signal, namely, y (k + 1); the output y (k +1) of the linear motor can be obtained by the grating detection unit, and then y (k) is obtained by the delay module; the lambda and rho values in the MFAC control law algorithm can be directly adjusted online by WM-Write2 and WM-Write3 in cSPACE; the output of the estimator is connected to the In end of the Subsystem, and the output Out is obtained
y (k) is connected to the desired signal y (k +1) in a negative feedback manner, thereby obtaining y (k +1) -y (k); the output is inserted into the Product module to obtain
Accessing the output sum u (k-1) into the Add block, wherein Listof signs in the Add block is set to (+ + -Io); obtaining an output signal u (k) of the MFAC master controller;
according to the formula
Calculating the pseudo partial derivative of the system at the k moment;
wherein, mu is more than 0, eta belongs to (0, 1)]；
The pseudo-partial derivative of the previous time instant of representation;
Δy(k)＝y(k)-y(k-1)；Δu(k-1)＝u(k-1)-u(k-2)；
by introducing pseudo partial derivativesFormula (II)
Obtaining the output u (k) of the main control computer;
transmitting the conversion information;
the information receiving and converting module is connected with the central control module and used for receiving the conversion information through the information receiving and converting program and converting the conversion information into a digital signal to obtain the construction information of the big data platform;
the information analysis module is connected with the central control module and used for analyzing the acquired big data platform construction information through an information analysis program to obtain an information analysis result;
and the central control module is connected with the platform information acquisition module, the information conversion and transmission module, the information receiving and conversion module and the information analysis module and is used for controlling the operation of each connection module through the main control computer and ensuring the normal operation of each module.
Further, the monitoring system for the running state of the big data platform further comprises:
the operation flow acquisition module is connected with the central control module and used for acquiring the operation flow of the big data platform according to the information analysis result through the operation flow acquisition program to obtain the operation flow of the big data platform;
the monitoring node deployment module is connected with the central control module and used for deploying the monitoring nodes of the big data platform through a monitoring node deployment program;
the operation monitoring module is connected with the central control module and used for monitoring the operation of the big data platform through the deployed monitoring nodes to obtain the operation monitoring result of the big data platform;
the task type acquisition module is connected with the central control module and used for acquiring the type of the large data platform operation task according to the acquired large data platform operation monitoring result through the task type acquisition result; the types of the running tasks of the big data platform comprise an offline task and an online task;
and the operation state judgment module is connected with the central control module and used for judging the operation state of the big data platform according to the acquired operation flow of the big data platform, the operation monitoring result of the big data platform and the type of the operation task of the big data platform through the operation state judgment program to obtain the operation state of the big data platform.
Further, the acquiring of the big data platform construction information through the platform information acquiring program to obtain the big data platform construction information includes:
the method comprises the steps that mobile nodes form a network, a data ID in the network defines data of one type, the mobile nodes which can provide the data of the same type in the network form a k-anycast group, the k-anycast group is uniquely identified by the data ID defining the data of the type, and the mobile nodes in the k-anycast group are called backbone nodes;
in a k-anycast group which comprises X backbone nodes and can provide data C, wherein X is more than or equal to 2, backbone nodes BxBy a unique network prefix MxIdentification, the k-anycast group is defined by the set of network prefixes G, as follows:
wherein X is more than or equal to X is more than or equal to 1;
the address of a backbone node or mobile node comprises two parts: a network prefix of i bits and a node ID of j bits; the network prefix comprises a data ID of k bits and a backbone ID of (i-k) bits, the node ID comprises a data ID of k bits and an internal ID of (j-k) bits, and i, j and k are positive integers smaller than 64;
after a backbone node B is started, a temporary address is created, the network prefix of the temporary address is an i-bit random number, and the node ID is a j-bit random number; backbone node BxBroadcasting an address creation message, wherein the source address of the message is a temporary address, and the load is a random number and a data ID c; backbone node BxWaiting for a certain time, judging the backbone node B after receiving the address creation information broadcast by other X-1 backbone nodes in the same k-anycast groupy1And backbone node By2Wherein y1 ≠ y 2;
backbone node BxThe X backbone nodes in the same k-anycast group are sorted according to the priority level in an increasing way, if the backbone node BxThe priority of (2) is p in the ordering value of X backbone nodes, X is more than or equal to p and more than or equal to 1, and backbone node BxThen set its backbone ID to pxSimultaneously constructing an address, wherein the data ID in the network prefix of the address is c, the node ID is zero, and simultaneously constructing an address according to a formula
Constructing a network prefix set G;
backbone node with network prefix of y according to formula
Obtaining an internal ID space [ L (y), U (y)]，X≥y≥1。
Further, the transmitting the conversion information includes:
(1) dividing nodes into different levels according to energy levels, data services and workloads of sensing nodes in a wireless body area network;
(2) after the current sensor node acquires the data, judging the channel state between the current sensor node and the sink node, if the channel state is good, directly sending the data to the sink node by the current sensor node, and finishing data transmission; otherwise, entering (3);
(3) taking the current sensor node as a source node; selecting a relay node leading to a sink node for a source node by utilizing an enhanced learning algorithm;
(4) after the data reaches the selected relay node, judging the channel state between the current relay node and the sink node, if the channel state is good, directly sending the data to the sink node by the current relay node, and finishing data transmission; and if not, taking the current relay node as the source node, and returning to the step (4).
Further, the determining the channel state with the sink node includes: and judging the channel state between the current sensor node or the current relay node and the sink node by using the Markov channel model.
Further, the determining a channel state between the current sensor node or the current relay node and the sink node by using the markov channel model includes:
setting a state transition matrix P of a Markov channel model;
the state transition matrix P of the markov channel model is:
wherein, the state set S ═ {0,1}, 0 represents a bad state, and 1 represents a good state; pijThe probability of the channel being converted from the i state to the j state is represented, and the following conditions are satisfied:
let S0The initial state variable of the channel between the node and the sink node, and after n times, the probability p (n) that the channel is in a good state is:
further, the acquiring the type of the big data platform operation task according to the acquired big data platform operation monitoring result through the task type acquisition result includes:
acquiring an operation flow of a big data platform to obtain a plurality of task types of the big data platform;
acquiring a large data platform operation monitoring result;
analyzing the operation monitoring result of the big data platform in stages according to the acquired task type of the big data platform to obtain a monitoring node corresponding to the stage;
and calling monitoring nodes which are pre-deployed on the big data platform and correspond to the stages to acquire the task execution conditions of the computing tasks at the corresponding stages.
Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, causes the processor to perform the functions of the monitoring system for the operating status of the big data platform.
Another object of the present invention is to provide an information data processing terminal, which performs the function of the monitoring system for the operation status of the big data platform.
Another object of the present invention is to provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to apply the functions of the monitoring system for the operating status of the big data platform.
By combining all the technical schemes, the invention has the advantages and positive effects that: according to the invention, the pre-deployment of the monitoring nodes is realized by acquiring the construction information of the big data operation platform, so that the platform operation information is conveniently monitored; the operation flow of the platform is acquired before the deployment of the monitoring nodes, so that the deployed monitoring nodes and the platform can be in one-to-one correspondence with each operating stage, and the operation information of each stage of the platform can be acquired more accurately; according to the method and the device, the information of each task stage of the big data platform is obtained through the monitoring nodes which are deployed in advance, the problem that the accuracy is low due to the fact that the service components of the big data platform are monitored in the traditional technology is solved, the running state of the big data platform is accurately monitored, and the running efficiency of the big data platform is guaranteed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained from the drawings without creative efforts.
Fig. 1 is a block diagram of a monitoring system for an operation state of a big data platform according to an embodiment of the present invention.
Fig. 2 is a flowchart of a method for monitoring an operating state of a big data platform according to an embodiment of the present invention.
Fig. 3 is a flowchart for performing the conversion of the big data platform construction information and transmitting the converted information through the information conversion and transmission program according to the embodiment of the present invention.
Fig. 4 is a flowchart for transmitting conversion information according to an embodiment of the present invention.
Fig. 5 is a flowchart for acquiring the type of a big data platform operation task according to the acquired big data platform operation monitoring result according to the task type acquisition result provided in the embodiment of the present invention.
In the figure: 1. a platform information acquisition module; 2. the information conversion and transmission module; 3. the information receiving and converting module; 4. an information analysis module; 5. a central control module; 6. an operation flow acquisition module; 7. a monitoring node deployment module; 8. operating the monitoring module; 9. a task type obtaining module; 10. and an operation state judgment module.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides a monitoring system for the operating status of a large data platform, and the following describes the present invention in detail with reference to the accompanying drawings.
As shown in fig. 1, a monitoring system for an operating state of a big data platform provided in an embodiment of the present invention includes:
the platform information acquisition module 1 is connected with the central control module 5 and is used for acquiring the large data platform construction information through a platform information acquisition program to obtain the large data platform construction information;
the information conversion and transmission module 2 is connected with the central control module 5 and is used for converting the large data platform construction information through an information conversion and transmission program and transmitting the converted information;
the information receiving and converting module 3 is connected with the central control module 5 and is used for receiving the conversion information through an information receiving and converting program and converting the conversion information into a digital signal to obtain the construction information of the big data platform;
the information analysis module 4 is connected with the central control module 5 and used for analyzing the acquired big data platform construction information through an information analysis program to obtain an information analysis result;
the central control module 5 is connected with the platform information acquisition module 1, the information conversion and transmission module 2, the information receiving and conversion module 3, the information analysis module 4, the operation flow acquisition module 6, the monitoring node deployment module 7, the operation monitoring module 8, the task type acquisition module 9 and the operation state judgment module 10, and is used for controlling the operation of each connection module through a main control computer and ensuring the normal operation of each module;
the operation flow acquisition module 6 is connected with the central control module 5 and is used for acquiring the operation flow of the big data platform according to the information analysis result through the operation flow acquisition program to obtain the operation flow of the big data platform;
the monitoring node deployment module 7 is connected with the central control module 5 and used for deploying the monitoring nodes of the big data platform through a monitoring node deployment program;
the operation monitoring module 8 is connected with the central control module 5 and used for monitoring the operation of the big data platform through the deployed monitoring nodes to obtain an operation monitoring result of the big data platform;
the task type acquisition module 9 is connected with the central control module 5 and used for acquiring the type of the large data platform operation task according to the acquired large data platform operation monitoring result through the task type acquisition result; the types of the running tasks of the big data platform comprise an offline task and an online task;
and the running state judging module 10 is connected with the central control module 5 and is used for judging the running state of the big data platform according to the obtained running flow of the big data platform, the running monitoring result of the big data platform and the type of the running task of the big data platform through a running state judging program to obtain the running state of the big data platform.
As shown in fig. 2, the method for monitoring the operating state of the big data platform according to the embodiment of the present invention includes the following steps:
s101, acquiring the construction information of the big data platform by using a platform information acquisition program through a platform information acquisition module to obtain the construction information of the big data platform; the information conversion and transmission module is used for converting the large data platform construction information by using an information conversion and transmission program and transmitting the converted information;
s102, receiving conversion information by using an information receiving and converting program through an information receiving and converting module and converting the conversion information into a digital signal to obtain large data platform construction information; analyzing the acquired big data platform construction information by using an information analysis program through an information analysis module to obtain an information analysis result;
s103, controlling the operation of each connecting module by using a main control computer through a central control module to ensure the normal operation of each module; acquiring the operation flow of the big data platform by using the operation flow acquisition program according to the information analysis result through an operation flow acquisition module to obtain the operation flow of the big data platform;
s104, deploying the monitoring nodes of the big data platform by using a monitoring node deployment program through a monitoring node deployment module; monitoring the operation of the big data platform by using the deployed monitoring nodes through the operation monitoring module to obtain an operation monitoring result of the big data platform;
s105, acquiring the type of the large data platform operation task by using the task type acquisition result through the task type acquisition module according to the acquired large data platform operation monitoring result; the types of the running tasks of the big data platform comprise an offline task and an online task;
and S106, judging the running state of the big data platform by the running state judging module according to the obtained running process of the big data platform, the running monitoring result of the big data platform and the type of the running task of the big data platform by using the running state judging program to obtain the running state of the big data platform.
The embodiment of the invention provides a method for acquiring big data platform construction information through a platform information acquisition program to obtain the big data platform construction information, which comprises the following steps:
the method comprises the steps that mobile nodes form a network, a data ID in the network defines data of one type, the mobile nodes which can provide the data of the same type in the network form a k-anycast group, the k-anycast group is uniquely identified by the data ID defining the data of the type, and the mobile nodes in the k-anycast group are called backbone nodes;
in a k-anycast group which comprises X backbone nodes and can provide data C, wherein X is more than or equal to 2, backbone nodes BxBy a unique network prefix MxIdentification, the k-anycast group is defined by the set of network prefixes G, as follows:
wherein X is more than or equal to X is more than or equal to 1;
the address of a backbone node or mobile node comprises two parts: a network prefix of i bits and a node ID of j bits; the network prefix comprises a data ID of k bits and a backbone ID of (i-k) bits, the node ID comprises a data ID of k bits and an internal ID of (j-k) bits, and i, j and k are positive integers smaller than 64;
after a backbone node B is started, a temporary address is created, the network prefix of the temporary address is an i-bit random number, and the node ID is a j-bit random number; backbone node BxBroadcasting an address creation message, wherein the source address of the message is a temporary address, and the load is a random number and a data ID c; backbone node BxWaiting a certain time, and creating addresses broadcasted by other X-1 backbone nodes in the same k-anycast groupAfter the message is established, judging the backbone node By1And backbone node By2Wherein y1 ≠ y 2;
backbone node BxThe X backbone nodes in the same k-anycast group are sorted according to the priority level in an increasing way, if the backbone node BxThe priority of (2) is p in the ordering value of X backbone nodes, X is more than or equal to p and more than or equal to 1, and backbone node BxThen set its backbone ID to pxSimultaneously constructing an address, wherein the data ID in the network prefix of the address is c, the node ID is zero, and simultaneously constructing an address according to a formula
Constructing a network prefix set G;
backbone node with network prefix of y according to formula
Obtaining an internal ID space [ L (y), U (y)]，X≥y≥1。
As shown in fig. 3, the converting of the big data platform construction information and the transmitting of the converted information through the information converting and transmitting program according to the embodiment of the present invention includes:
s201, determining a linear relation between the physical quantity of the A/D conversion object and the corresponding voltage value;
s202, judging whether the voltage value measured under the current physical quantity has deviation, and if the deviation exists, carrying out A/D conversion calibration by the main control machine;
s203, after the calibration is completed, the big data platform construction information is converted to obtain conversion information;
and S204, transmitting the conversion information.
The converting of the big data platform construction information to obtain the conversion information provided by the embodiment of the invention comprises the following steps:
connecting and setting parameters of a main control machine and testing the operating characteristics of the main control machine; the operation characteristic of the main control computer of the test comprises the following steps: the main control machine is controlled and corrected under the speed mode, so that the stable performance of the speed inner ring is ensured; the designed MFAC control algorithm is built on an upper PC through Simulink of cSPACE; running a compiling module in the MFAC master controller; automatically generating DSP codes by using an MFAC algorithm; downloading the codes to a digital signal processor through a USB interface of an upper PC for operation, generating a voltage output signal through a main control computer, and driving the main control computer to operate;
the MFAC control method is realized by programming:
calculating the output u (k) of the main control computer,
the MFAC main control computer u (k) is built partially based on Simulink of cSPACE, and lambda > 0 is a weight coefficient and used for limiting the change of control input quantity; rho epsilon (0, 1)]The step size factor is additionally added, so that the algorithm has stronger flexibility and generality;
is an estimate of phi (k) at time k; y (k +1) is the desired output signal; u (k), y (k) respectively represent the input and output of the system at time k;
the MFAC master control unit u (k) is built based on Simulink of cSPACE in part, and comprises the following components:
u (k) is subjected to a time delay module to obtain u (k-1); a Sine Wave module in the cSPACE gives a sinusoidal position signal, namely, y (k + 1); the output y (k +1) of the linear motor can be obtained by the grating detection unit, and then y (k) is obtained by the delay module; the lambda and rho values in the MFAC control law algorithm can be directly adjusted online by WM-Write2 and WM-Write3 in cSPACE; the output of the estimator is connected to the In end of the Subsystem, and the output Out is obtained
y (k) is connected to the desired signal y (k +1) in a negative feedback manner, thereby obtaining y (k +1) -y (k); the output is inserted into the Product module to obtain
Accessing the output sum u (k-1) into the Add block, wherein Listof signs in the Add block is set to (+ + -Io); obtaining an output signal u (k) of the MFAC master controller;
according to the formula
Calculating the pseudo partial derivative of the system at the k moment;
wherein, mu is more than 0, eta belongs to (0, 1)]；
The pseudo-partial derivative of the previous time instant of representation;
Δy(k)＝y(k)-y(k-1)；Δu(k-1)＝u(k-1)-u(k-2)；
substituting the pseudo partial derivatives into the formula
This results in master output u (k).
As shown in fig. 4, the transmission of the conversion information provided in the embodiment of the present invention includes:
s301, dividing nodes into different levels according to energy levels, data services and workloads of sensing nodes in the wireless body area network;
s302, after the current sensor node collects data, judging the channel state between the current sensor node and the sink node, if the channel state is good, directly sending the data to the sink node by the current sensor node, and finishing data transmission; otherwise, go to step S303;
s303, taking the current sensor node as a source node; selecting a relay node leading to a sink node for a source node by utilizing an enhanced learning algorithm;
s304, after the data reaches the selected relay node, judging the channel state between the current relay node and the sink node, if the channel state is good, directly sending the data to the sink node by the current relay node, and finishing data transmission; otherwise, the current relay node is taken as the source node, and the step S303 is returned.
The method for judging the channel state between the sink nodes provided by the embodiment of the invention comprises the following steps: and judging the channel state between the current sensor node or the current relay node and the sink node by using the Markov channel model.
The method for judging the channel state between the current sensor node or the current relay node and the sink node by using the Markov channel model comprises the following steps:
setting a state transition matrix P of a Markov channel model;
the state transition matrix P of the markov channel model is:
wherein, the state set S ═ {0,1}, 0 represents a bad state, and 1 represents a good state; pijThe probability of the channel being converted from the i state to the j state is represented, and the following conditions are satisfied:
let S0The initial state variable of the channel between the node and the sink node, and after n times, the probability p (n) that the channel is in a good state is:
as shown in fig. 5, the obtaining of the type of the big data platform operation task according to the obtained big data platform operation monitoring result by the task type obtaining result according to the embodiment of the present invention includes:
s401, acquiring an operation process of a big data platform to obtain a plurality of task types of the big data platform;
s402, obtaining a large data platform operation monitoring result;
s403, analyzing the operation monitoring result of the big data platform in stages according to the acquired task type of the big data platform to obtain a monitoring node corresponding to the stage;
and S404, calling monitoring nodes which are pre-deployed on the big data platform and correspond to the stages, and acquiring the task execution condition of the computing task at the corresponding stage.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention, and the scope of the present invention is not limited thereto, and any modification, equivalent replacement, and improvement made by those skilled in the art within the technical scope of the present invention disclosed herein, which is within the spirit and principle of the present invention, should be covered by the present invention.
Claims (10)
1. The monitoring system for the running state of the big data platform is characterized by comprising the following components:
the platform information acquisition module is connected with the central control module and used for acquiring the large data platform construction information through a platform information acquisition program to obtain the large data platform construction information;
the information conversion and transmission module is connected with the central control module and is used for converting the large data platform construction information through an information conversion and transmission program and transmitting the converted information;
the conversion of the big data platform construction information and the transmission of the conversion information are carried out through the information conversion and transmission program, and the method comprises the following steps:
determining a linear relationship between a physical quantity of the a/D conversion object and the corresponding voltage value;
judging whether the voltage value measured under the current physical quantity has deviation, and if so, carrying out A/D conversion calibration by the main control machine;
after the calibration is finished, converting the big data platform construction information to obtain conversion information;
the conversion of the big data platform construction information to obtain conversion information comprises the following steps:
connecting and setting parameters of a main control machine and testing the operating characteristics of the main control machine; the operation characteristic of the main control computer of the test comprises the following steps: the main control machine is controlled and corrected under the speed mode, so that the stable performance of the speed inner ring is ensured; the designed MFAC control algorithm is built on an upper PC through Simulink of cSPACE; running a compiling module in the MFAC master controller; automatically generating DSP codes by using an MFAC algorithm; downloading the codes to a digital signal processor through a USB interface of an upper PC for operation, generating a voltage output signal through a main control computer, and driving the main control computer to operate;
the MFAC control method is realized by programming:
calculating the output u (k) of the main control computer,
the MFAC main control computer u (k) is built partially based on Simulink of cSPACE, and lambda > 0 is a weight coefficient and used for limiting the change of control input quantity; rho epsilon (0, 1)]The step size factor is additionally added, so that the algorithm has stronger flexibility and generality;
is an estimate of phi (k) at time k; y (k +1) is the desired output signal; u (k), y (k) respectively represent the input and output of the system at time k;
the MFAC master control unit u (k) is built based on Simulink of cSPACE in part, and comprises the following components:
u (k) is subjected to a time delay module to obtain u (k-1); a Sine Wave module in the cSPACE gives a sinusoidal position signal, namely, y (k + 1); the output y (k +1) of the linear motor can be obtained by the grating detection unit, and then y (k) is obtained by the delay module; the lambda and rho values in the MFAC control law algorithm can be directly adjusted online by WM-Write2 and WM-Write3 in cSPACE; the output of the estimator is connected to the In end of the Subsystem, and the output Out is obtained
y (k) is connected to the desired signal y (k +1) in a negative feedback manner, thereby obtaining y (k +1) -y (k); the output is inserted into the Product module to obtain
Accessing the output sum u (k-1) into the Add block, wherein Listof signs in the Add block is set to (+ + -Io); obtaining an output signal u (k) of the MFAC master controller;
according to the formula
Calculating the pseudo partial derivative of the system at the k moment;
wherein, mu is more than 0, eta belongs to (0, 1)]；
The pseudo-partial derivative of the previous time instant of representation;
Δy(k)＝y(k)-y(k-1)；Δu(k-1)＝u(k-1)-u(k-2)；
substituting the pseudo partial derivatives into the formula
Obtaining the output u (k) of the main control computer;
transmitting the conversion information;
the information receiving and converting module is connected with the central control module and used for receiving the conversion information through the information receiving and converting program and converting the conversion information into a digital signal to obtain the construction information of the big data platform;
the information analysis module is connected with the central control module and used for analyzing the acquired big data platform construction information through an information analysis program to obtain an information analysis result;
and the central control module is connected with the platform information acquisition module, the information conversion and transmission module, the information receiving and conversion module and the information analysis module and is used for controlling the operation of each connection module through the main control computer and ensuring the normal operation of each module.
2. The big data platform operating condition monitoring system according to claim 1, further comprising:
the operation flow acquisition module is connected with the central control module and used for acquiring the operation flow of the big data platform according to the information analysis result through the operation flow acquisition program to obtain the operation flow of the big data platform;
the monitoring node deployment module is connected with the central control module and used for deploying the monitoring nodes of the big data platform through a monitoring node deployment program;
the operation monitoring module is connected with the central control module and used for monitoring the operation of the big data platform through the deployed monitoring nodes to obtain the operation monitoring result of the big data platform;
the task type acquisition module is connected with the central control module and used for acquiring the type of the large data platform operation task according to the acquired large data platform operation monitoring result through the task type acquisition result; the types of the running tasks of the big data platform comprise an offline task and an online task;
and the operation state judgment module is connected with the central control module and used for judging the operation state of the big data platform according to the acquired operation flow of the big data platform, the operation monitoring result of the big data platform and the type of the operation task of the big data platform through the operation state judgment program to obtain the operation state of the big data platform.
3. The system for monitoring the operating status of the big data platform according to claim 1, wherein the obtaining of the big data platform construction information through the platform information obtaining program to obtain the big data platform construction information comprises:
the method comprises the steps that mobile nodes form a network, a data ID in the network defines data of one type, the mobile nodes which can provide the data of the same type in the network form a k-anycast group, the k-anycast group is uniquely identified by the data ID defining the data of the type, and the mobile nodes in the k-anycast group are called backbone nodes;
in a k-anycast group which comprises X backbone nodes and can provide data C, wherein X is more than or equal to 2, backbone nodes BxBy a unique network prefix MxIdentifying, k-anycast groups consisting of a set of network prefixesAnd G is defined by the formula:
wherein X is more than or equal to X is more than or equal to 1;
the address of a backbone node or mobile node comprises two parts: a network prefix of i bits and a node ID of j bits; the network prefix comprises a data ID of k bits and a backbone ID of (i-k) bits, the node ID comprises a data ID of k bits and an internal ID of (j-k) bits, and i, j and k are positive integers smaller than 64;
after a backbone node B is started, a temporary address is created, the network prefix of the temporary address is an i-bit random number, and the node ID is a j-bit random number; backbone node BxBroadcasting an address creation message, wherein the source address of the message is a temporary address, and the load is a random number and a data ID c; backbone node BxWaiting for a certain time, judging the backbone node B after receiving the address creation information broadcast by other X-1 backbone nodes in the same k-anycast groupy1And backbone node By2Wherein y1 ≠ y 2;
backbone node BxThe X backbone nodes in the same k-anycast group are sorted according to the priority level in an increasing way, if the backbone node BxThe priority of (2) is p in the ordering value of X backbone nodes, X is more than or equal to p and more than or equal to 1, and backbone node BxThen set its backbone ID to pxSimultaneously constructing an address, wherein the data ID in the network prefix of the address is c, the node ID is zero, and simultaneously constructing an address according to a formula
Constructing a network prefix set G;
backbone node with network prefix of y according to formula
Obtaining an internal ID space [ L (y), U (y)]，X≥y≥1。
4. The system for monitoring the operating state of the big data platform according to claim 1, wherein the transmitting of the conversion information comprises:
(1) dividing nodes into different levels according to energy levels, data services and workloads of sensing nodes in a wireless body area network;
(2) after the current sensor node acquires the data, judging the channel state between the current sensor node and the sink node, if the channel state is good, directly sending the data to the sink node by the current sensor node, and finishing data transmission; otherwise, entering (3);
(3) taking the current sensor node as a source node; selecting a relay node leading to a sink node for a source node by utilizing an enhanced learning algorithm;
(4) after the data reaches the selected relay node, judging the channel state between the current relay node and the sink node, if the channel state is good, directly sending the data to the sink node by the current relay node, and finishing data transmission; and if not, taking the current relay node as the source node, and returning to the step (4).
5. The big data platform operation state monitoring system according to claim 4, wherein the determining the channel state with the sink node comprises: and judging the channel state between the current sensor node or the current relay node and the sink node by using the Markov channel model.
6. The big data platform operation state monitoring system according to claim 5, wherein the determining the channel state between the current sensor node or the current relay node and the sink node by using the Markov channel model comprises:
setting a state transition matrix P of a Markov channel model;
the state transition matrix P of the markov channel model is:
wherein, the state set S ═ {0,1}, 0 represents a bad state, and 1 represents a good state; pijThe probability of the channel being converted from the i state to the j state is represented, and the following conditions are satisfied:
let S0The initial state variable of the channel between the node and the sink node, and after n times, the probability p (n) that the channel is in a good state is:
7. the system for monitoring the operating status of the big data platform according to claim 2, wherein the obtaining of the type of the big data platform operation task according to the obtained big data platform operation monitoring result by the task type obtaining result comprises:
acquiring an operation flow of a big data platform to obtain a plurality of task types of the big data platform;
acquiring a large data platform operation monitoring result;
analyzing the operation monitoring result of the big data platform in stages according to the acquired task type of the big data platform to obtain a monitoring node corresponding to the stage;
and calling monitoring nodes which are pre-deployed on the big data platform and correspond to the stages to acquire the task execution conditions of the computing tasks at the corresponding stages.
8. A computer device, characterized in that the computer device comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor, so that the processor executes the function of the monitoring system for the operating state of the big data platform according to any one of claims 1 to 7.
9. An information data processing terminal, characterized in that the information data processing terminal performs the function of the monitoring system for the operating state of the big data platform according to any one of claims 1 to 7.
10. A computer readable storage medium storing instructions which, when executed on a computer, cause the computer to apply the functions of the big data platform operational status monitoring system as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110434297.2A CN113190403A (en) | 2021-04-22 | 2021-04-22 | Monitoring system, computer equipment, terminal and medium for operation state of big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110434297.2A CN113190403A (en) | 2021-04-22 | 2021-04-22 | Monitoring system, computer equipment, terminal and medium for operation state of big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113190403A true CN113190403A (en) | 2021-07-30 |
Family
ID=76978054
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110434297.2A Pending CN113190403A (en) | 2021-04-22 | 2021-04-22 | Monitoring system, computer equipment, terminal and medium for operation state of big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113190403A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113835967A (en) * | 2021-09-28 | 2021-12-24 | 北京京东拓先科技有限公司 | Monitoring method, monitoring device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105871641A (en) * | 2016-06-08 | 2016-08-17 | 常熟理工学院 | Data obtaining method based on cloud |
| CN106686533A (en) * | 2017-01-06 | 2017-05-17 | 东华大学 | Wireless body area network data transmission method based on reinforced learning algorithm |
| CN108319538A (en) * | 2018-02-02 | 2018-07-24 | 世纪龙信息网络有限责任公司 | The monitoring method and system of big data platform operating status |
| CN109462356A (en) * | 2018-12-06 | 2019-03-12 | 北京信息科技大学 | A kind of MFA control method of servo system for linear motor |
| CN111259073A (en) * | 2020-01-08 | 2020-06-09 | 国网福建省电力有限公司 | Intelligent business system running state studying and judging system based on logs, flow and business access |
| CN112685385A (en) * | 2020-12-31 | 2021-04-20 | 广西中科曙光云计算有限公司 | Big data platform for smart city construction |
-
2021
- 2021-04-22 CN CN202110434297.2A patent/CN113190403A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105871641A (en) * | 2016-06-08 | 2016-08-17 | 常熟理工学院 | Data obtaining method based on cloud |
| CN106686533A (en) * | 2017-01-06 | 2017-05-17 | 东华大学 | Wireless body area network data transmission method based on reinforced learning algorithm |
| CN108319538A (en) * | 2018-02-02 | 2018-07-24 | 世纪龙信息网络有限责任公司 | The monitoring method and system of big data platform operating status |
| CN109462356A (en) * | 2018-12-06 | 2019-03-12 | 北京信息科技大学 | A kind of MFA control method of servo system for linear motor |
| CN111259073A (en) * | 2020-01-08 | 2020-06-09 | 国网福建省电力有限公司 | Intelligent business system running state studying and judging system based on logs, flow and business access |
| CN112685385A (en) * | 2020-12-31 | 2021-04-20 | 广西中科曙光云计算有限公司 | Big data platform for smart city construction |
Non-Patent Citations (1)
| Title |
|---|
| 曹荣敏等: "非圆切削刀具进给系统的复合迭代学习控制", 《电气传动》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113835967A (en) * | 2021-09-28 | 2021-12-24 | 北京京东拓先科技有限公司 | Monitoring method, monitoring device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112990330B (en) | User energy abnormal data detection method and device | |
| CN114169536B (en) | Data management and control method and related device | |
| KR20200052403A (en) | Data analysis system based on edge computing and method thereof | |
| CN115756642A (en) | Business process configuration method, device, equipment and storage medium | |
| CN113190403A (en) | Monitoring system, computer equipment, terminal and medium for operation state of big data platform | |
| CN114884997A (en) | Intelligent cable monitoring system for sensor data grading transmission | |
| CN115421950A (en) | Automatic system operation and maintenance management method and system based on machine learning | |
| CN107203464B (en) | Method and device for positioning service problem | |
| CN113225994B (en) | Intelligent air conditioner control method facing data center | |
| CN112948353B (en) | Data analysis method, system and storage medium applied to DAstudio | |
| CN111090401B (en) | Storage device performance prediction method and device | |
| CN113704046A (en) | Fault alarm processing method and device, equipment and storage medium | |
| CN108401233B (en) | Composite event perception method based on maximum weight binary matching | |
| CN114065937A (en) | Equipment fault diagnosis method and device and terminal equipment | |
| CN112733454A (en) | Equipment predictive maintenance method and device based on joint learning | |
| CN115688447B (en) | Cloud simulation system architecture of high-performance power system | |
| CN111861012A (en) | Test task execution time prediction method and optimal execution node selection method | |
| CN116164820A (en) | Correction method and system for gas data of intelligent gas meter | |
| CN113295963B (en) | Cross-domain cascading failure source node judgment method for CPS system of active power distribution network | |
| CN115794659B (en) | Distributed parallel testing method, device, equipment and medium for CFD software | |
| WO2020202168A1 (en) | A computer implemented method for handling occurrence of anomalies and/or faults in a network node | |
| CN116954169A (en) | Information security control method, device, equipment and medium based on DCS control | |
| CN117474292B (en) | 5G transmission-based scheduling system and scheduling method for network-connected unmanned aerial vehicle | |
| CN110999263B (en) | Hierarchical data processing for IOT device clusters | |
| CN112804709B (en) | Method and device for determining overlapping coverage condition between serving cells |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210730 |