CN113784236B - Distributed data acquisition monitoring method, device, equipment and medium - Google Patents
Distributed data acquisition monitoring method, device, equipment and medium Download PDFInfo
- Publication number
- CN113784236B CN113784236B CN202111333293.1A CN202111333293A CN113784236B CN 113784236 B CN113784236 B CN 113784236B CN 202111333293 A CN202111333293 A CN 202111333293A CN 113784236 B CN113784236 B CN 113784236B
- Authority
- CN
- China
- Prior art keywords
- information
- monitoring
- data
- acquisition
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q9/00—Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a distributed data acquisition monitoring method, a device, equipment and a medium, wherein the method comprises the following steps: sending an information acquisition instruction to the sub-servers to acquire basic information, generating monitoring configuration information corresponding to each sub-server according to the data monitoring request and the basic information, and sending the monitoring configuration information to the sub-servers, wherein the sub-servers execute an acquisition program to acquire initial acquisition information and feed the initial acquisition information back to a monitoring terminal, or acquiring data flow information of a switch to construct a mirror image real-time data flow, acquiring the initial acquisition information from the mirror image real-time data flow through the configured acquisition program, performing data conversion on the initial acquisition information to obtain service monitoring information, and then screening according to information screening conditions to obtain target service monitoring information. By the method, the distributed system can be subjected to data acquisition and monitoring in a non-invasive mode based on the configured acquisition program, excessive intrusion into the system during data monitoring is avoided, a large amount of system resources can be avoided being consumed, and the stability of the system is improved.
Description
Technical Field
The invention relates to the technical field of distributed system monitoring, in particular to a distributed data acquisition monitoring method, a distributed data acquisition monitoring device, distributed data acquisition monitoring equipment and a distributed data acquisition monitoring medium.
Background
A large-scale enterprise needs to construct a distributed service processing system to process massive service data, so as to obtain the operating condition of the distributed service processing system in real time, and can monitor the data in the distributed service processing system in real time. Therefore, the prior art method has the problem of strong invasiveness when monitoring data of the distributed system.
Disclosure of Invention
The embodiment of the invention provides a distributed data acquisition monitoring method, a distributed data acquisition monitoring device, equipment and a medium, and aims to solve the problem of strong intrusiveness when a distributed system is monitored by the prior art.
In a first aspect, an embodiment of the present invention provides a distributed data acquisition monitoring method, where the method is applied to a monitoring terminal, and a network connection is established between the monitoring terminal and a cluster server configured in a distributed system to implement transmission of data information, and the method includes:
if the input data monitoring request is received, sending a pre-stored information acquisition instruction to the sub-servers of the cluster server so as to acquire basic information fed back by each sub-server according to the information acquisition instruction;
generating monitoring configuration information corresponding to the basic information of each sub-server according to a preset configuration information generation rule and the data monitoring request, and respectively sending the monitoring configuration information to each sub-server so that each sub-server configures a pre-stored acquisition program according to the monitoring configuration information;
acquiring initial acquisition information from each sub-server and performing data conversion on each initial acquisition information according to a preset conversion rule to obtain service monitoring information;
and screening the service monitoring information according to preset information screening conditions to obtain target service monitoring information meeting the information screening conditions.
In a second aspect, an embodiment of the present invention further provides a distributed data acquisition monitoring method, where the method is applied to a monitoring terminal, and a network connection is established between the monitoring terminal and a switch configured in a distributed system to implement transmission of data information, and the method includes:
if the input data monitoring request is received, sending a pre-stored data stream information acquisition instruction to the switch so as to acquire data stream information fed back by the switch in real time according to the data stream information acquisition instruction;
carrying out mirror image reduction on the data stream information to construct a mirror image real-time data stream;
configuring a pre-stored acquisition program according to the data monitoring request;
acquiring initial acquisition information from the mirror image real-time data stream according to the configured acquisition program;
performing data conversion on the initial acquisition information according to a preset conversion rule to obtain service monitoring information;
and screening the service monitoring information according to preset information screening conditions to obtain target service monitoring information meeting the information screening conditions.
In a third aspect, an embodiment of the present invention provides a distributed data acquisition monitoring apparatus, including:
the basic information acquisition unit is used for sending a prestored information acquisition instruction to the sub-servers of the cluster server to acquire basic information fed back by each sub-server according to the information acquisition instruction if the input data monitoring request is received;
the monitoring configuration information sending unit is used for generating monitoring configuration information corresponding to the basic information of each sub-server according to a preset configuration information generation rule and the data monitoring request and respectively sending the monitoring configuration information to each sub-server so that each sub-server can configure a pre-stored acquisition program according to the monitoring configuration information;
a service monitoring information obtaining unit, configured to obtain initial acquisition information from each of the sub-servers and perform data conversion on each of the initial acquisition information according to a preset conversion rule to obtain service monitoring information;
and the target service monitoring information acquisition unit is used for screening the service monitoring information according to preset information screening conditions to obtain the target service monitoring information meeting the information screening conditions.
In a fourth aspect, an embodiment of the present invention further provides a distributed data acquisition monitoring apparatus, which includes:
the data flow information acquisition unit is used for sending a pre-stored data flow information acquisition instruction to the switch to acquire data flow information fed back by the switch in real time according to the data flow information acquisition instruction if the input data monitoring request is received;
the mirror image real-time data flow construction unit is used for carrying out mirror image reduction on the data flow information to construct a mirror image real-time data flow;
the configuration unit is used for configuring a pre-stored acquisition program according to the data monitoring request;
the initial acquisition information acquisition unit is used for acquiring initial acquisition information from the mirror image real-time data stream according to the configured acquisition program;
a service monitoring information obtaining unit, configured to perform data conversion on the initial acquisition information according to a preset conversion rule to obtain service monitoring information;
and the target service monitoring information acquisition unit is used for screening the service monitoring information according to preset information screening conditions to obtain the target service monitoring information meeting the information screening conditions.
In a fifth aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the distributed data collection monitoring method according to the first aspect or the second aspect when executing the computer program.
In a sixth aspect, the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the distributed data acquisition monitoring method according to the first aspect or the second aspect.
The embodiment of the invention provides a distributed data acquisition monitoring method, a distributed data acquisition monitoring device, distributed data acquisition monitoring equipment and a distributed data acquisition monitoring medium. Sending an information acquisition instruction to the sub-servers to acquire basic information, generating monitoring configuration information corresponding to each sub-server according to the data monitoring request and the basic information, and sending the monitoring configuration information to the sub-servers, wherein the sub-servers execute an acquisition program to acquire initial acquisition information and feed the initial acquisition information back to a monitoring terminal, or acquiring data flow information of a switch to construct a mirror image real-time data flow, acquiring the initial acquisition information from the mirror image real-time data flow through the configured acquisition program, performing data conversion on the initial acquisition information to obtain service monitoring information, and then screening according to information screening conditions to obtain target service monitoring information. By the method, the data acquisition and monitoring can be carried out on the service data processed in the distributed system in a non-invasive mode based on the configured acquisition program, the excessive intrusion into the system during the data monitoring is avoided, a large amount of system resources can be avoided being consumed, and the stability of the system is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a distributed data acquisition monitoring method according to an embodiment of the present invention;
fig. 2 is a schematic view of an application scenario of a distributed data acquisition monitoring method according to an embodiment of the present invention;
fig. 3 is a schematic sub-flow diagram of a distributed data acquisition monitoring method according to an embodiment of the present invention;
fig. 4 is a schematic view of another sub-flow of the distributed data collection monitoring method according to the embodiment of the present invention;
fig. 5 is a schematic view of another sub-flow of the distributed data collection monitoring method according to the embodiment of the present invention;
fig. 6 is another schematic flow chart of a distributed data acquisition monitoring method according to an embodiment of the present invention;
fig. 7 is a schematic view of another application scenario of the distributed data acquisition monitoring method according to the embodiment of the present invention;
fig. 8 is a schematic view of another sub-flow of the distributed data collection monitoring method according to the embodiment of the present invention;
fig. 9 is a schematic block diagram of a distributed data acquisition monitoring apparatus according to an embodiment of the present invention;
FIG. 10 is another schematic block diagram of a distributed data collection monitoring apparatus according to an embodiment of the present invention;
FIG. 11 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic flow chart of a distributed data acquisition monitoring method according to an embodiment of the present invention; fig. 2 is a schematic view of an application scenario of the distributed data acquisition monitoring method according to the embodiment of the present invention. In this embodiment, the distributed data acquisition monitoring method is applied to the monitoring terminal 10, the distributed data acquisition monitoring method is executed by application software installed in the monitoring terminal 10, the cluster server 30 is configured in the distributed system 20, and a network connection is established between the monitoring terminal 10 and the cluster server 30 to realize transmission of data information; the monitoring terminal 10 is a terminal device, such as a desktop computer, a notebook computer, etc., for executing the distributed data acquisition monitoring method to perform data acquisition and monitoring on the distributed system, the monitoring terminal 10 may also exist in the form of an independent monitoring server, the cluster server 30 is a server end for processing mass service data based on distributed construction by an enterprise or a government department, and the cluster server includes a plurality of sub-servers. As shown in FIG. 1, the method includes steps S110 to S140.
S110, if the input data monitoring request is received, sending a pre-stored information acquisition instruction to the sub-servers of the cluster server to acquire basic information fed back by each sub-server according to the information acquisition instruction.
The user of the monitoring terminal can be an enterprise manager, the enterprise manager can input a data monitoring request to the monitoring terminal, the monitoring terminal can send an information acquisition instruction to a sub-server configured in the cluster server, the monitoring terminal stores the information acquisition instruction in advance, the information acquisition instruction is an instruction for acquiring basic information from the sub-server, the information acquisition instruction can be represented in a program code form, the cluster server is composed of a plurality of sub-servers, and each sub-server can execute the instruction and feed the basic information back to the monitoring terminal after receiving the information acquisition instruction. The basic information fed back by the sub-servers comprises information related to the sub-servers, such as network card type, CPU (Central processing Unit) ratio, memory ratio and the like.
And S120, generating monitoring configuration information corresponding to the basic information of each sub-server according to a preset configuration information generation rule and the data monitoring request, and respectively sending the monitoring configuration information to each sub-server so that each sub-server configures a pre-stored acquisition program according to the monitoring configuration information.
After the monitoring terminal acquires the basic information of each sub-server, monitoring configuration information corresponding to each basic information can be generated according to configuration information generation rules and data monitoring requests, the monitoring configuration information corresponding to each sub-server is respectively sent to corresponding sub-servers, the sub-servers can configure a pre-stored acquisition program after receiving the monitoring configuration information corresponding to the sub-servers, the sub-servers pre-store the acquisition program, and the acquisition program comprises corresponding acquisition parameters, so that the sub-servers can configure the acquisition parameters in the acquisition program through the received monitoring configuration information.
In an embodiment, as shown in fig. 3, step S120 includes sub-steps S121, S122, S123 and S124.
And S121, carrying out hierarchical matching on the basic information of each sub-server according to the configuration information generation rule to obtain a monitoring level matched with the basic information.
The basic information of the sub-server can be matched in a grading manner according to the configuration information generation rule, specifically, the configuration information generation rule comprises a plurality of monitoring levels and basic matching conditions corresponding to each monitoring level, whether each basic matching condition is matched with the basic information can be judged, so that the monitoring level corresponding to the basic matching condition matched with the basic information is obtained and is used as the monitoring level corresponding to the basic information, wherein only one basic matching condition is provided for one basic information, the monitoring levels correspond to different acquisition intervals, if the monitoring level is high, the acquisition interval is short, the data information acquired by the sub-server has short time delay, and a network card with a characteristic type needs to be supported; if the monitoring level is low, the acquisition interval time is longer, the time delay of the data information acquired by the sub server is increased, but any type of network card is supported, and the occupation of the acquisition program on system resources is extremely low.
In another embodiment, the configuration information generation rule may include a classification neural network, and the basic information may be hierarchically matched based on the classification neural network, so as to obtain a monitoring level corresponding to each basic information. Specifically, a plurality of items of information included in the basic information may be converted to obtain corresponding quantization values, each item of information is represented in a numerical manner by the quantization values, the plurality of quantization values obtained by conversion are input from an input layer of the classification neural network, the plurality of quantization values are subjected to correlation analysis and calculation by a neural node configured in the classification neural network, a matching value corresponding to each monitoring level is obtained by input from an output layer, and then the monitoring level with the maximum matching value may be obtained as the monitoring level corresponding to the input basic information.
And S122, performing data conversion on the initial filtering condition in the data monitoring request to obtain filtering matching information.
The data monitoring request comprises initial filtering conditions, and in order to facilitate the input of enterprise managers, the initial filtering conditions are all represented in an intuitive writing form, and the data processing of the computer is all in a binary form, so that the initial filtering conditions can be subjected to data conversion through a conversion format configured in the monitoring terminal to obtain corresponding filtering matching information, and the process of performing data conversion on the initial filtering conditions is performed in the monitoring terminal, so that the resource consumption of the sub-servers can be further reduced.
For example, the initial filtering condition includes an IP address and a port number, where the IP address is: IP = 123.136.55.76; then "IP = 123.136.55.76" may be converted into a binary string represented in binary form, wherein for the digits, only the decimal number needs to be converted into a binary number, and for the letters, symbols, etc., information therein may be converted into the corresponding binary number by ASCII encoding, Base64 encoding, or other encoding.
S123, combining the monitoring grade of each sub-server and the filtering matching information to generate monitoring configuration information corresponding to each sub-server; and S124, respectively sending the monitoring configuration information of each sub-server to the corresponding sub-server.
Respectively combining the filtering matching information with the monitoring grade of each sub-server to generate monitoring configuration information corresponding to each sub-server; and respectively sending the monitoring configuration information corresponding to each sub-server to the corresponding sub-server.
The sub-server receives the monitoring configuration information, can configure the collection program stored in the sub-server, and monitors and collects the service data processed by the sub-server through the configured collection program, so that initial collection information matched with the configured parameters in the collection program is obtained from the monitored service data and is sent to the monitoring terminal.
S130, acquiring initial acquisition information from each sub-server, and performing data conversion on each initial acquisition information according to a preset conversion rule to obtain service monitoring information.
The monitoring terminal can acquire initial acquisition information obtained by monitoring and acquiring the service data by the sub-servers and can perform data conversion on the initial acquisition information of each sub-server according to the conversion rule to obtain the service monitoring information.
In one embodiment, as shown in fig. 4, step S130 includes sub-steps S131 and S132.
S131, carrying out data system conversion on each initial acquisition information according to the conversion rule to obtain conversion data information corresponding to each initial acquisition information;
specifically, the initial acquisition information may be subjected to data system conversion according to a conversion rule, where the conversion rule may be based on an ASCII code, a Base64 code, an UTF-8 code, or a GB2312 code, and the initial acquisition information is represented in a binary form, and then the initial acquisition information may be converted by the conversion rule, so that the initial acquisition information is restored to conversion data information represented in an intuitive writing form, that is, a binary string in the initial acquisition information is restored to text content represented in a form of numbers, letters, chinese characters, and the like.
And S132, performing aggregation sequencing on the conversion data information corresponding to each piece of initial acquisition information to obtain corresponding service monitoring information.
And specifically, each piece of information in the conversion data information also comprises processing time, data source labels and the like, multiple pieces of conversion data information can be aggregated, and the aggregated data is sequenced based on the processing time, the data source labels and the like, so that the data information is restored to be in the same original data structure as that in the sub server.
S140, screening the service monitoring information according to preset information screening conditions to obtain target service monitoring information meeting the information screening conditions.
The monitoring terminal can further screen the obtained business monitoring information according to the preconfigured information screening conditions, for example, enterprise managers can input corresponding subscription information, and the monitoring terminal can analyze the subscription information to generate information screening conditions so as to screen target business monitoring information from the business monitoring information according to the information screening conditions, wherein the information screening conditions comprise screening items and numerical value ranges corresponding to the screening items, and the information screening conditions can include one or more screening items.
In an embodiment, as shown in fig. 5, step S140 includes sub-steps S141, S142 and S143.
And S141, acquiring a project monitoring value corresponding to each service monitoring data and the screening project from the service monitoring information.
Specifically, the service monitoring information includes a plurality of pieces of service monitoring data, the screening items may be time, service type identifiers, data source identifiers, and the like, and then the item monitoring value corresponding to each piece of service monitoring data and the screening item may be obtained from the service monitoring information based on the screening items.
S142, judging whether the project monitoring value of each screening project is in the corresponding numerical range; s143, obtaining the service monitoring data which is not in the numerical range according to the judgment result of the project monitoring value and determining the service monitoring data as the target service monitoring information meeting the information screening condition.
And judging whether the item monitoring value of each screening item is in the corresponding numerical range of the corresponding screening item, and acquiring the service monitoring data which is not in the numerical range as the target service monitoring information according to the judgment result, namely if any item monitoring value in certain service monitoring data is not in the corresponding numerical range, taking the service monitoring data as the service monitoring data meeting the information screening condition, and acquiring all the service monitoring data meeting the information screening condition as the target service monitoring information.
For example, for a transaction distributed system, the service monitoring data is generally divided into three types, namely, a consignment order, a consignment transaction and a consignment rejection order, that is, each piece of service monitoring data includes a service type identifier corresponding to one of the three types, and if the screening item is the service type identifier and the corresponding numerical range is the consignment transaction, the service monitoring data whose service type identifier is the consignment order or the consignment rejection order can be correspondingly acquired as the service monitoring data meeting the information screening condition.
In addition, the service monitoring data in the array range can be obtained according to the judgment result of the item value and determined as the corresponding target service monitoring information, if the item monitoring value in a certain service monitoring data is in the corresponding numerical value range, the service monitoring data is used as the service monitoring data meeting the information screening condition, and all the service monitoring data meeting the information screening condition are obtained as the target service monitoring information.
Referring to fig. 6 and 7, fig. 6 is another schematic flow chart of a distributed data acquisition monitoring method according to an embodiment of the present invention; fig. 7 is a schematic view of another application scenario of the distributed data collection monitoring method according to the embodiment of the present invention. In this embodiment, the distributed data acquisition monitoring method is applied to the monitoring terminal 10, the distributed data acquisition monitoring method is executed by application software installed in the monitoring terminal 10, the cluster server 30 and the switch 40 are configured in the distributed system 20, and a network connection is established between the monitoring terminal 10 and the switch 40 to realize transmission of data information; the monitoring terminal 10 is a terminal device, such as a desktop computer, a notebook computer, etc., for executing the distributed data acquisition monitoring method to perform data acquisition and monitoring on the distributed system, the monitoring terminal 10 may also exist in the form of an independent monitoring server, the cluster server 30 is a server side for processing mass service data based on distributed construction by an enterprise or a government department, the cluster server includes a plurality of sub-servers, and the switch 40 is a network communication device configured in the distributed system 20 and used for performing exchange transmission on data input and output by the sub-servers. As shown in FIG. 6, the method includes steps S210 to S260.
S210, if the input data monitoring request is received, sending a pre-stored data flow information acquisition instruction to the switch so as to acquire data flow information fed back by the switch in real time according to the data flow information acquisition instruction.
The monitoring terminal can send a data stream information acquisition instruction to the switch, and after receiving the data stream information acquisition instruction, the switch can feed back the service data which is subjected to switching transmission to the monitoring terminal in real time as data stream information.
S220, carrying out mirror image reduction on the data flow information to construct a mirror image real-time data flow.
The monitoring terminal can utilize the mirror image function of the switch to perform mirror image restoration on the obtained data flow information, namely, the processing process of the switch for performing exchange transmission on the service data is realized by restoration in the monitoring terminal, so that the mirror image real-time data flow is constructed and obtained.
And S230, configuring a pre-stored acquisition program according to the data monitoring request.
And configuring a pre-stored acquisition program in the monitoring terminal according to the received data monitoring request.
In one embodiment, as shown in fig. 8, step S230 includes sub-steps S231 and S232.
And S231, performing data conversion on the initial filtering condition in the data monitoring request to obtain filtering matching information.
The initial filtering condition included in the data monitoring request may be subjected to data conversion to obtain filtering matching information, and the specific processing procedure is the same as the step S122, which is not described herein again.
And S232, performing parameter configuration on the filtering matching parameters in the acquisition program according to the filtering matching information.
And performing parameter configuration on the filtering matching parameters in the acquisition program according to the filtering matching information, wherein the acquisition program does not need to occupy any system resource of the sub-server because the acquisition program runs in the monitoring terminal instead of in the sub-server, namely the acquisition program is configured without referring to basic information of the sub-server, and the filtering matching parameters are directly acquired to perform parameter configuration on the acquisition program.
S240, acquiring initial acquisition information from the mirror image real-time data stream according to the configured acquisition program.
Acquiring corresponding initial acquisition information from the mirror image real-time data stream according to the configured acquisition program, namely, running the configured acquisition program in the monitoring terminal, and monitoring and acquiring the service data contained in the mirror image real-time data stream through the configured acquisition program so as to acquire the initial acquisition information matched with the configured parameters in the acquisition program.
And S250, performing data conversion on the initial acquisition information according to a preset conversion rule to obtain service monitoring information.
The process of performing data conversion on the effluent collecting information according to the conversion rule is the specific processing process of the step S130, that is, the specific processing process of this step is the same as the step S130, and is not described herein again.
S260, screening the service monitoring information according to preset information screening conditions to obtain target service monitoring information meeting the information screening conditions.
The process of screening the service monitoring information according to the information screening condition is the specific processing process of the step S140, that is, the specific processing process of this step is the same as that of the step S140, and is not described herein again.
In the distributed data acquisition monitoring method provided by the embodiment of the invention, an information acquisition instruction is sent to the sub-servers to acquire basic information, monitoring configuration information corresponding to each sub-server is generated according to a data monitoring request and the basic information and is sent to the sub-servers, the sub-servers execute an acquisition program to acquire initial acquisition information and feed back the initial acquisition information to a monitoring terminal, or data flow information of a switch is acquired to construct a mirror image real-time data flow, the initial acquisition information is acquired from the mirror image real-time data flow through the configured acquisition program, the initial acquisition information is subjected to data conversion to obtain service monitoring information, and then screening is carried out according to information screening conditions to obtain target service monitoring information. By the method, the data of the service data processed in the distributed system can be acquired and monitored in a non-invasive mode based on the configured acquisition program, excessive intrusion into the system during data monitoring is avoided, a large amount of system resources can be avoided being consumed, the proportion of occupying the system resources in the data monitoring process is greatly reduced, and the stability of the system is improved.
The embodiment of the present invention further provides a distributed data acquisition monitoring device, where the distributed data acquisition monitoring device may be configured in the monitoring terminal 10, and the distributed data acquisition monitoring device is configured to execute any embodiment of the foregoing distributed data acquisition monitoring method. Specifically, referring to fig. 9, fig. 9 is a schematic block diagram of a distributed data acquisition monitoring apparatus according to an embodiment of the present invention.
As shown in fig. 9, the distributed data collection monitoring apparatus 100 includes: a basic information obtaining unit 110, configured to send a pre-stored information obtaining instruction to the sub-servers of the cluster server if receiving the input data monitoring request, so as to obtain basic information fed back by each sub-server according to the information obtaining instruction; a monitoring configuration information sending unit 120, configured to generate, according to a preset configuration information generation rule and the data monitoring request, monitoring configuration information corresponding to the basic information of each sub-server, and send the monitoring configuration information to each sub-server, so that each sub-server configures a pre-stored acquisition program according to the monitoring configuration information; a service monitoring information obtaining unit 130, configured to obtain initial acquisition information from each of the sub servers, and perform data conversion on each of the initial acquisition information according to a preset conversion rule to obtain service monitoring information; the target service monitoring information obtaining unit 140 is configured to filter the service monitoring information according to a preset information filtering condition, so as to obtain target service monitoring information meeting the information filtering condition.
The embodiment of the present invention further provides a distributed data acquisition monitoring device, where the distributed data acquisition monitoring device may be configured in the monitoring terminal 10, and the distributed data acquisition monitoring device is configured to execute any embodiment of the foregoing distributed data acquisition monitoring method. Specifically, referring to fig. 10, fig. 10 is another schematic block diagram of a distributed data acquisition monitoring apparatus according to an embodiment of the present invention.
As shown in fig. 10, the distributed data collection monitoring apparatus 200 includes: a data flow information obtaining unit 210, configured to send a pre-stored data flow information obtaining instruction to the switch if an input data monitoring request is received, so as to obtain data flow information fed back by the switch in real time according to the data flow information obtaining instruction; a mirror real-time data stream constructing unit 220, configured to perform mirror restoration on the data stream information to construct a mirror real-time data stream; a configuration unit 230, configured to configure a pre-stored acquisition program according to the data monitoring request; an initial acquisition information obtaining unit 240, configured to acquire initial acquisition information from the mirror image real-time data stream according to the configured acquisition program; a service monitoring information obtaining unit 250, configured to perform data conversion on the initial acquisition information according to a preset conversion rule to obtain service monitoring information; and a target service monitoring information obtaining unit 260, configured to filter the service monitoring information according to a preset information filtering condition, so as to obtain target service monitoring information meeting the information filtering condition.
The distributed data acquisition monitoring device provided by the embodiment of the invention applies the distributed data acquisition monitoring method, sends an information acquisition instruction to the sub-servers to acquire basic information, generates monitoring configuration information corresponding to each sub-server according to a data monitoring request and the basic information and sends the monitoring configuration information to the sub-servers, the sub-servers execute an acquisition program to acquire initial acquisition information and feed the initial acquisition information back to a monitoring terminal, or acquires data flow information of a switch to construct a mirror image real-time data flow, acquires the initial acquisition information from the mirror image real-time data flow through the configured acquisition program, performs data conversion on the initial acquisition information to obtain service monitoring information, and then performs screening according to information screening conditions to obtain target service monitoring information. By the method, the data acquisition and monitoring can be carried out on the service data processed in the distributed system in a non-invasive mode based on the configured acquisition program, the excessive intrusion into the system during the data monitoring is avoided, a large amount of system resources can be avoided being consumed, and the stability of the system is improved.
The above-described distributed data acquisition monitoring apparatus may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 11.
Referring to fig. 11, fig. 11 is a schematic block diagram of a computer device according to an embodiment of the present invention. The computer device may be a monitoring terminal 10 for performing a distributed data acquisition monitoring method for data acquisition and monitoring of a distributed system.
Referring to fig. 11, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a storage medium 503 and an internal memory 504.
The storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, may cause the processor 502 to perform a distributed data collection monitoring method, wherein the storage medium 503 may be a volatile storage medium or a non-volatile storage medium.
The processor 502 is used to provide computing and control capabilities that support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the execution of the computer program 5032 in the storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 can be enabled to execute the distributed data collection monitoring method.
The network interface 505 is used for network communication, such as providing transmission of data information. Those skilled in the art will appreciate that the configuration shown in fig. 11 is a block diagram of only a portion of the configuration associated with aspects of the present invention and is not intended to limit the computing device 500 to which aspects of the present invention may be applied, and that a particular computing device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
The processor 502 is configured to run a computer program 5032 stored in the memory to implement the corresponding functions in the distributed data collection monitoring method.
Those skilled in the art will appreciate that the embodiment of a computer device illustrated in FIG. 11 does not constitute a limitation on the specific organization of a computer device, and that in other embodiments a computer device may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components may be used. For example, in some embodiments, the computer device may only include a memory and a processor, and in such embodiments, the structures and functions of the memory and the processor are consistent with those of the embodiment shown in fig. 11, and are not described herein again.
It should be understood that, in the embodiment of the present invention, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In another embodiment of the invention, a computer-readable storage medium is provided. The computer readable storage medium may be a volatile or non-volatile computer readable storage medium. The computer readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps included in the above-described distributed data acquisition monitoring method.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses, devices and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided by the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only a logical division, and there may be other divisions when the actual implementation is performed, or units having the same function may be grouped into one unit, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a computer-readable storage medium, which includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned computer-readable storage media comprise: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (8)
1. A distributed data acquisition monitoring method is characterized in that the method is applied to a monitoring terminal, network connection is established between the monitoring terminal and a cluster server configured in a distributed system to realize data information transmission, and the method comprises the following steps:
if the input data monitoring request is received, sending a pre-stored information acquisition instruction to the sub-servers of the cluster server so as to acquire basic information fed back by each sub-server according to the information acquisition instruction;
generating monitoring configuration information corresponding to the basic information of each sub-server according to a preset configuration information generation rule and the data monitoring request, and respectively sending the monitoring configuration information to each sub-server so that each sub-server configures a pre-stored acquisition program according to the monitoring configuration information;
acquiring initial acquisition information from each sub-server and performing data conversion on each initial acquisition information according to a preset conversion rule to obtain service monitoring information;
screening the service monitoring information according to preset information screening conditions to obtain target service monitoring information meeting the information screening conditions;
the generating of the monitoring configuration information corresponding to the basic information of each sub-server according to the preset configuration information generating rule and the data monitoring request and the sending of the monitoring configuration information to each sub-server respectively comprise:
carrying out hierarchical matching on the basic information of each sub-server according to the configuration information generation rule to obtain a monitoring level matched with the basic information;
performing data conversion on the initial filtering condition in the data monitoring request to obtain filtering matching information;
combining the monitoring grade of each sub-server with the filtering matching information to generate monitoring configuration information corresponding to each sub-server;
and respectively sending the monitoring configuration information of each sub-server to the corresponding sub-server.
2. The distributed data acquisition monitoring method according to claim 1, wherein the acquiring initial acquisition information from each of the sub-servers and performing data conversion on each of the initial acquisition information according to a preset conversion rule to obtain service monitoring information includes:
carrying out data system conversion on each initial acquisition information according to the conversion rule to obtain conversion data information corresponding to each initial acquisition information;
and performing aggregation sequencing on the conversion data information corresponding to each piece of initial acquisition information to obtain corresponding service monitoring information.
3. The distributed data acquisition monitoring method according to claim 1, wherein the information screening conditions include screening items and a numerical range corresponding to each of the screening items, and the screening of the service monitoring information according to preset information screening conditions to obtain target service monitoring information satisfying the information screening conditions includes:
acquiring a project monitoring value corresponding to each service monitoring data and the screening project from the service monitoring information;
judging whether the item monitoring value of each screening item is within a corresponding numerical range;
and acquiring service monitoring data which is not in the numerical range according to the judgment result of the project monitoring value and determining the service monitoring data as target service monitoring information meeting the information screening condition.
4. A distributed data acquisition monitoring method is characterized in that the method is applied to a monitoring terminal, network connection is established between the monitoring terminal and a switch configured in a distributed system to realize data information transmission, and the method comprises the following steps:
if the input data monitoring request is received, sending a pre-stored data stream information acquisition instruction to the switch so as to acquire data stream information fed back by the switch in real time according to the data stream information acquisition instruction;
carrying out mirror image reduction on the data stream information to construct a mirror image real-time data stream;
configuring a pre-stored acquisition program according to the data monitoring request;
acquiring initial acquisition information from the mirror image real-time data stream according to the configured acquisition program;
performing data conversion on the initial acquisition information according to a preset conversion rule to obtain service monitoring information;
screening the service monitoring information according to preset information screening conditions to obtain target service monitoring information meeting the information screening conditions;
the configuring of the pre-stored acquisition program according to the data monitoring request comprises:
performing data conversion on the initial filtering condition in the data monitoring request to obtain filtering matching information;
and performing parameter configuration on the filtering matching parameters in the acquisition program according to the filtering matching information.
5. A distributed data acquisition monitoring device is characterized in that the device is configured in a monitoring terminal, and a network connection is established between the monitoring terminal and a cluster server configured in a distributed system to realize the transmission of data information, and the device comprises:
the basic information acquisition unit is used for sending a prestored information acquisition instruction to the sub-servers of the cluster server to acquire basic information fed back by each sub-server according to the information acquisition instruction if the input data monitoring request is received;
the monitoring configuration information sending unit is used for generating monitoring configuration information corresponding to the basic information of each sub-server according to a preset configuration information generation rule and the data monitoring request and respectively sending the monitoring configuration information to each sub-server so that each sub-server can configure a pre-stored acquisition program according to the monitoring configuration information;
a service monitoring information obtaining unit, configured to obtain initial acquisition information from each of the sub-servers and perform data conversion on each of the initial acquisition information according to a preset conversion rule to obtain service monitoring information;
a target service monitoring information obtaining unit, configured to filter the service monitoring information according to a preset information filtering condition, so as to obtain target service monitoring information that meets the information filtering condition;
the generating of the monitoring configuration information corresponding to the basic information of each sub-server according to the preset configuration information generating rule and the data monitoring request and the sending of the monitoring configuration information to each sub-server respectively comprise:
carrying out hierarchical matching on the basic information of each sub-server according to the configuration information generation rule to obtain a monitoring level matched with the basic information;
performing data conversion on the initial filtering condition in the data monitoring request to obtain filtering matching information;
combining the monitoring grade of each sub-server with the filtering matching information to generate monitoring configuration information corresponding to each sub-server;
and respectively sending the monitoring configuration information of each sub-server to the corresponding sub-server.
6. A distributed data acquisition monitoring device is characterized in that the device is configured in a monitoring terminal, and a network connection is established between the monitoring terminal and a switch configured in a distributed system to realize the transmission of data information, and the device comprises:
the data flow information acquisition unit is used for sending a pre-stored data flow information acquisition instruction to the switch to acquire data flow information fed back by the switch in real time according to the data flow information acquisition instruction if the input data monitoring request is received;
the mirror image real-time data flow construction unit is used for carrying out mirror image reduction on the data flow information to construct a mirror image real-time data flow;
the configuration unit is used for configuring a pre-stored acquisition program according to the data monitoring request;
the initial acquisition information acquisition unit is used for acquiring initial acquisition information from the mirror image real-time data stream according to the configured acquisition program;
a service monitoring information obtaining unit, configured to perform data conversion on the initial acquisition information according to a preset conversion rule to obtain service monitoring information;
a target service monitoring information obtaining unit, configured to filter the service monitoring information according to a preset information filtering condition, so as to obtain target service monitoring information that meets the information filtering condition;
the configuring of the pre-stored acquisition program according to the data monitoring request comprises:
performing data conversion on the initial filtering condition in the data monitoring request to obtain filtering matching information;
and performing parameter configuration on the filtering matching parameters in the acquisition program according to the filtering matching information.
7. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the distributed data acquisition monitoring method according to any one of claims 1 to 3 or 4 when executing the computer program.
8. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the distributed data acquisition monitoring method of any one of claims 1 to 3 or claim 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111333293.1A CN113784236B (en) | 2021-11-11 | 2021-11-11 | Distributed data acquisition monitoring method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111333293.1A CN113784236B (en) | 2021-11-11 | 2021-11-11 | Distributed data acquisition monitoring method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113784236A CN113784236A (en) | 2021-12-10 |
| CN113784236B true CN113784236B (en) | 2022-02-18 |
Family
ID=78956918
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111333293.1A Active CN113784236B (en) | 2021-11-11 | 2021-11-11 | Distributed data acquisition monitoring method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113784236B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115239212B (en) * | 2022-09-22 | 2022-12-27 | 中科三清科技有限公司 | Monitoring method, device and system of air quality mode and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2546789A1 (en) * | 2011-07-15 | 2013-01-16 | Khalifa University of Science, Technology and Research | Method and system for distributed and collaborative monitoring |
| CN103997439A (en) * | 2014-06-04 | 2014-08-20 | 腾讯科技（深圳）有限公司 | Flow monitoring method, device and system |
| CN108769112A (en) * | 2018-04-18 | 2018-11-06 | 江苏物联网研究发展中心 | A kind of non-intrusion type RPC interface monitoring method and system |
| CN110209559A (en) * | 2019-04-26 | 2019-09-06 | 华为技术有限公司 | Non-invasive monitoring method, control equipment and storage medium |
| CN111478862A (en) * | 2020-03-09 | 2020-07-31 | 邦彦技术股份有限公司 | Remote data mirroring system and method |
| CN112468495A (en) * | 2020-11-26 | 2021-03-09 | 上海天旦网络科技发展有限公司 | Degradation monitoring method, system and medium for complete forward secrecy encryption system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110635938B (en) * | 2019-08-19 | 2021-07-16 | 腾讯科技（深圳）有限公司 | Monitoring method, system, equipment and medium |
| CN211856726U (en) * | 2020-03-24 | 2020-11-03 | 陈焕秋 | Non-invasive load monitoring system based on optical coupling isolation circuit |
-
2021
- 2021-11-11 CN CN202111333293.1A patent/CN113784236B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2546789A1 (en) * | 2011-07-15 | 2013-01-16 | Khalifa University of Science, Technology and Research | Method and system for distributed and collaborative monitoring |
| CN103997439A (en) * | 2014-06-04 | 2014-08-20 | 腾讯科技（深圳）有限公司 | Flow monitoring method, device and system |
| CN108769112A (en) * | 2018-04-18 | 2018-11-06 | 江苏物联网研究发展中心 | A kind of non-intrusion type RPC interface monitoring method and system |
| CN110209559A (en) * | 2019-04-26 | 2019-09-06 | 华为技术有限公司 | Non-invasive monitoring method, control equipment and storage medium |
| CN111478862A (en) * | 2020-03-09 | 2020-07-31 | 邦彦技术股份有限公司 | Remote data mirroring system and method |
| CN112468495A (en) * | 2020-11-26 | 2021-03-09 | 上海天旦网络科技发展有限公司 | Degradation monitoring method, system and medium for complete forward secrecy encryption system |
Non-Patent Citations (1)
| Title |
|---|
| 云计算环境下的资源监控平台研究;管华等;《武汉工业学院学报》;20131215(第04期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113784236A (en) | 2021-12-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111290916B (en) | Big data monitoring method, device and equipment and computer readable storage medium | |
| JP2018085117A (en) | Efficient data compression and analysis as service | |
| CN110830577B (en) | Service request call tracking method and device, computer equipment and storage medium | |
| CN107305611B (en) | Method and device for establishing model corresponding to malicious account and method and device for identifying malicious account | |
| US10476768B2 (en) | Diagnostic and recovery signals for disconnected applications in hosted service environment | |
| CN113784236B (en) | Distributed data acquisition monitoring method, device, equipment and medium | |
| CN114116401A (en) | Method, device, equipment and medium for tracking and monitoring all links of cluster message | |
| CN116436772B (en) | Real-time alarm method and device, electronic equipment and storage medium | |
| CN113794899A (en) | Cloud desktop image data transmission method, device, equipment and storage medium | |
| CN114091704B (en) | Alarm suppression method and device | |
| CN111131375B (en) | Interface service acquisition method, device, computer equipment and storage medium | |
| CN113934733A (en) | Problem positioning method, device, system, storage medium and electronic equipment | |
| CN112181832A (en) | Test data generation method and device, computer equipment and storage medium | |
| CN111259045A (en) | Data processing method, device, server and medium | |
| CN111046082A (en) | Data source determination method, device, server and storage medium | |
| KR20180011183A (en) | How to Remove Message Notifications, Systems, and Servers | |
| CN113992668B (en) | Information real-time transmission method, device, equipment and medium based on multiple concurrences | |
| US11601326B1 (en) | Problem detection and categorization for integration flows | |
| CN113315793B (en) | Data transmission method, device, equipment and medium based on intelligent compression | |
| CN112235367B (en) | Method, system, terminal and storage medium for subscribing entity behavior relation message | |
| WO2021248316A1 (en) | Event processing method and apparatus, electronic device, and storage medium | |
| CN113568803B (en) | Method, electronic device and computer program product for monitoring a storage system | |
| CN111835624B (en) | In-application message distribution method and device, computer equipment and storage medium | |
| CN111565311B (en) | Network traffic characteristic generation method and device | |
| CN109214874B (en) | IP product operation data processing method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: Room 2301, building 5, Shenzhen new generation industrial park, 136 Zhongkang Road, Meidu community, Meilin street, Futian District, Shenzhen, Guangdong 518000 Patentee after: Shenzhen Huarui Distributed Technology Co.,Ltd. Address before: Room 2301, building 5, Shenzhen new generation industrial park, 136 Zhongkang Road, Meidu community, Meilin street, Futian District, Shenzhen, Guangdong 518000 Patentee before: SHENZHEN ARCHFORCE FINANCIAL TECHNOLOGY Co.,Ltd. |