CN112000548A - Big data component monitoring method and device and electronic equipment - Google Patents
Big data component monitoring method and device and electronic equipment Download PDFInfo
- Publication number
- CN112000548A CN112000548A CN202010847304.7A CN202010847304A CN112000548A CN 112000548 A CN112000548 A CN 112000548A CN 202010847304 A CN202010847304 A CN 202010847304A CN 112000548 A CN112000548 A CN 112000548A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- data
- big data
- preset
- data component
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45591—Monitoring or debugging support
Abstract
The invention provides a method and a device for monitoring a big data component and electronic equipment, and relates to the technical field of big data; determining the numerical value of a preset monitoring item according to the real-time monitoring data; and monitoring the big data component to be monitored according to the numerical value of the monitoring item. According to the embodiment of the invention, the real-time monitoring data of the large data assembly is obtained through the preset external interface of the large data assembly or the preset external interface of the large data deployment frame, then the real-time monitoring data is analyzed and processed to obtain the numerical value of the monitoring item, and then the assembly is monitored according to the numerical value of the monitoring item.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a method and a device for monitoring a big data assembly and electronic equipment.
Background
At present, the scheme for monitoring the big Data component includes three kinds, the first is to implement monitoring by combining a big Data automation operation and maintenance tool with an external component, wherein, the operation and maintenance tool collects the state information of the big Data component, such as HDP (hot works Data Platform) or CDH (cloud Distribution Hadoop), and implements monitoring alarm notification by the external component. The second monitoring scheme is used for the big data native self-built system, and requires the installation of a third-party open source assembly to monitor the big data assemblies in the self-built system, and each big data assembly needs an independent monitoring system to monitor. The third scheme is used for monitoring the cloud big data assembly, generally, information of the big data assembly is collected through a process in a Docker, the collected information is reported to a unified access service, and finally the collected information is stored in a database.
Overall, the existing big data component monitoring mode needs to be realized by depending on external components, and considering that the cluster deployment in the public cloud big data field is usually in a virtual network, each virtual machine network in the cluster is isolated from each other, so that the big data component information of each virtual machine in the cluster cannot be acquired by setting the external components.
At present, no effective method can be used for monitoring big data components in the public cloud big data field.
Disclosure of Invention
In view of this, the present invention provides a method and an apparatus for monitoring a big data component, and an electronic device, which can monitor the big data component in the public cloud big data field.
In a first aspect, an embodiment of the present invention provides a big data component monitoring method, where the method is applied to a first server, and the first server is communicatively connected to a second server running a big data component to be monitored, and the method includes: acquiring real-time monitoring data of the big data component to be monitored through a preset interface; determining the numerical value of a preset monitoring item according to the real-time monitoring data; and monitoring the big data component to be monitored according to the numerical value of the monitoring item.
In a preferred embodiment of the present invention, the step of obtaining the real-time monitoring data of the big data component to be monitored through the preset interface includes: and reading a preset first external interface of the big data assembly to be monitored, or reading a preset second external interface of a big data deployment frame in the second server, so as to obtain real-time monitoring data of the big data assembly to be monitored.
In a preferred embodiment of the present invention, the step of reading the preset first external interface of the big data component to be monitored includes: and reading a preset first external interface of the big data assembly to be monitored through a preset data acquisition service.
In a preferred embodiment of the present invention, the first external interface includes a Java management extended application program interface, where the Java management extended application program interface is configured to provide the running state data of the big data component to be monitored to the outside.
In a preferred embodiment of the present invention, the step of determining the value of the preset monitoring item according to the real-time monitoring data includes: converting the real-time monitoring data into data in a specified storage format; and analyzing the data after format conversion according to a preset metadata rule to obtain a numerical value of a preset monitoring item.
In a preferred embodiment of the present invention, the step of analyzing the data after format conversion according to a preset metadata rule to obtain a value of a preset monitoring item includes: according to the data type of the data after format conversion, sending the data to a theme corresponding to the data type in a specified message queue; through a preset stream program, data are pulled from the theme matched with the stream program in the message queue; and analyzing the pulled data according to a preset metadata rule to obtain a numerical value of a preset monitoring item.
In a preferred embodiment of the present invention, the step of sending the data to a topic corresponding to the data type in a specified message queue according to the data type of the data after format conversion includes: and sending the data to a theme corresponding to the data type in the appointed message queue through preset proxy service according to the data type of the data after format conversion.
In a preferred embodiment of the present invention, the specified storage format includes a JSON format.
In a preferred embodiment of the present invention, the step of monitoring the big data component to be monitored according to the value of the monitoring item includes: saving the numerical value of the monitoring item to a specified database; reading the value of the monitoring item in the target time period from the specified database according to a preset period; and determining the running state of the monitoring item in the target time period according to the read value of the monitoring item.
In a preferred embodiment of the present invention, the step of reading the value of the monitoring item in the target time period from the designated database according to the preset period includes: and polling the specified database according to a preset period through a preset data query service to obtain the numerical value of the monitoring item in a target time period.
In a preferred embodiment of the present invention, the type of the designated database is a time-series database.
In a preferred embodiment of the present invention, the method further includes: and presenting the running state of the monitoring item in the target time period on a graphical display interface of the first server through a preset visualization program.
In a second aspect, an embodiment of the present invention further provides a big data component monitoring apparatus, which is applied to a first server, the first server being communicatively connected to a second server running a big data component to be monitored, the apparatus including: the real-time monitoring data acquisition module is used for acquiring real-time monitoring data of the big data component to be monitored through a preset interface; the monitoring item value determining module is used for determining the value of a preset monitoring item according to the real-time monitoring data; and the monitoring module is used for monitoring the big data component to be monitored according to the numerical value of the monitoring item.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the monitoring method for the big data component.
In a fourth aspect, the embodiments of the present invention also provide a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to implement the monitoring method for the big data component.
The embodiment of the invention has the following beneficial effects:
according to the monitoring method and device for the big data component and the electronic equipment provided by the embodiment of the invention, the real-time monitoring data of the big data component to be monitored is acquired through the preset interface; determining the numerical value of a preset monitoring item according to the real-time monitoring data; and monitoring the big data component to be monitored according to the numerical value of the monitoring item. In the mode, the real-time monitoring data of the component is acquired through the preset external interface of the big data component or the preset external interface of the big data deployment frame, then the real-time monitoring data is analyzed and processed to obtain the numerical value of the monitoring item, and then the component is monitored according to the numerical value of the monitoring item.
Additional features and advantages of the disclosure will be set forth in the description which follows, or in part may be learned by the practice of the above-described techniques of the disclosure, or may be learned by practice of the disclosure.
In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flowchart of a method for monitoring a big data component according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating another big data component monitoring method according to an embodiment of the present invention;
FIG. 3 is a flow chart illustrating another big data component monitoring method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a technical framework for implementing big data component monitoring according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a monitoring apparatus for big data components according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Icon: 51-a real-time monitoring data acquisition module; 52-monitoring item value determination module; 53-a monitoring module; 61-a processor; 62-a memory; 63-bus; 64-a communication interface.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In consideration of the problems that the existing big data component monitoring mode is realized by depending on an external component, and a public cloud network cannot acquire the big data component information of each virtual machine in a cluster in a mode of setting the external component, so that the big data component in the public cloud big data field cannot be monitored. For the convenience of understanding the embodiment, a detailed description will be given to a monitoring method for a big data component disclosed in the embodiment of the present invention.
Referring to fig. 1, a schematic flow chart of a monitoring method for a big data component according to an embodiment of the present invention is shown, wherein the method is applied to a first server, and the first server is communicatively connected to a second server running a big data component to be monitored, as seen in fig. 1, the method includes the following steps:
step S102: and acquiring real-time monitoring data of the big data component to be monitored through a preset interface.
Here, the component is a simple package of data and methods, where the large data component to be monitored may be a message queue, resource management, data storage, data integration, data calculation, query analysis, data visualization, task scheduling, and the like, and specifically, for example, a large data component such as Hadoop, elastic search, Hbase, ClickHouse, Kafka, and the like. Moreover, the real-time monitoring data is generally common operation and maintenance information of a big data component, such as: read-write times, occupied machine resources (CPU, memory, disk), etc.
In this embodiment, the big data component to be monitored runs on a second server, where the second server may be a virtual server, for example, a server deployed in a public cloud network cluster, or a real host server deployed in other environments. And the second server is in communication connection with a first server, and the first server is used for monitoring the big data component on the second server. Here, the first server may also be a virtual server or a real host server.
In addition, an interface is preset on the first server, and real-time monitoring data of the big data assembly to be monitored in the second server can be obtained through the interface. Here, the interface may be a preset first external interface of the big data component, or may be a preset second external interface of the big data deployment framework in the second server.
Step S104: and determining the numerical value of a preset monitoring item according to the real-time monitoring data.
In actual operation, the data volume of the real-time monitoring data of the big data component to be monitored, which is acquired through the preset interface, is large, the related parameters are often large, and the number of monitoring items actually monitored is usually few, wherein the monitoring items may be some parameter items in the real-time monitoring data, or parameters obtained by processing the parameter items in the real-time monitoring data, for example, the real-time monitoring data is cleaned, merged, and the like, so as to obtain the value of the preset monitoring item.
Step S106: and monitoring the big data component to be monitored according to the numerical value of the monitoring item.
And monitoring the running state of the monitoring item based on the numerical value of the monitoring item acquired in the above step, for example, for a certain monitoring item a, a running curve of the monitoring item a can be drawn based on the numerical value of the monitoring item a to judge whether the monitoring item is in a normal working state. Similarly, if a plurality of monitoring items exist, the plurality of monitoring items are respectively monitored, and whether the working state of the big data assembly is normal or not is judged based on the working state of each monitoring item, so that the monitoring of the big data assembly to be monitored is realized.
Compared with the mode of monitoring by combining an automatic operation and maintenance tool with an external component in the existing big data component monitoring technology, or the mode of monitoring the big data component in a native self-built system by a third-party open source component, or the mode of monitoring by acquiring the information of the big data component through a process in a Docker.
According to the monitoring method of the big data component provided by the embodiment of the invention, the real-time monitoring data of the big data component to be monitored is acquired through the preset interface; determining the numerical value of a preset monitoring item according to the real-time monitoring data; and monitoring the big data component to be monitored according to the numerical value of the monitoring item. In the mode, the real-time monitoring data of the component is acquired through the preset external interface of the big data component or the preset external interface of the big data deployment frame, then the real-time monitoring data is analyzed and processed to obtain the numerical value of the monitoring item, and then the component is monitored according to the numerical value of the monitoring item.
On the basis of the monitoring method for the big data component shown in fig. 1, the present embodiment further provides another monitoring method for the big data component, where the method is applied to a first server, and the first server is in communication connection with a second server running the big data component to be monitored, where the method mainly describes a specific implementation process of step S102 (obtaining real-time monitoring data of the big data component to be monitored through a preset interface) in the foregoing embodiment, and referring to fig. 2, a flowchart of another monitoring method for the big data component is shown, as can be seen from fig. 2, the method includes the following steps:
step S202: and reading a preset first external interface of the big data component to be monitored through a preset data acquisition service to obtain real-time monitoring data of the big data component to be monitored.
In this embodiment, the data collection service is developed based on Prometheus protocol, where Prometheus is a set of open source system monitoring and alarm framework with the following characteristics:
(1) multidimensional data models (Key, Value Key Value pairs based on time series);
(2) flexible query and aggregation language PromQL;
(3) providing local storage and distributed storage;
(4) collecting time series data through a Pull model based on HTTP;
(5) push mode can be implemented using Pushgateway (optional middleware of Prometheus);
(6) the target machine may be discovered through dynamic service discovery or static configuration;
(7) supporting a variety of charts and data dashboards.
In one possible implementation, the exporter service may be developed based on Prometheus protocol, and the exporter service collects real-time monitoring data of the big data component to be monitored. Here, the exporter service is an important component in Prometheus monitoring, and is a data collection service responsible for collecting data indexes.
In at least one possible implementation, the first external interface includes a Java management extended application program interface, for example, the first external interface may be a JMX API interface, and the Java management extended application program interface is configured to externally provide running state data of the big data component to be monitored, where the running state data may be the number of times of reading and writing of the interface, machine resources occupied by the interface, and the like.
Thus, the exporter service is developed in the second server in advance based on the Prometheus protocol, and the JMX API interface of the big data component to be monitored is read according to the exporter service, so that the real-time monitoring data of the big data component to be monitored is obtained. In other possible embodiments, the real-time monitoring data of the big data component to be monitored can be obtained by reading a preset second external interface of the big data deployment frame in the second server.
Step S204: and determining the numerical value of a preset monitoring item according to the real-time monitoring data.
Step S206: and monitoring the big data component to be monitored according to the numerical value of the monitoring item.
Step S204 and step S206 of this embodiment correspond to step S104 to step S106 of the foregoing embodiment, and corresponding portions of the foregoing embodiment may be referred to for description, which is not repeated herein.
In the monitoring method for the big data component provided in this embodiment, a data acquisition service (for example, an exporter service developed based on a Prometheus protocol) is pre-developed on a second server where the big data component to be monitored is located, and a preset first external interface (for example, JMX API interface) of the big data component to be monitored is read based on the data acquisition service, so as to obtain real-time monitoring data of the big data component to be monitored, and obtain a numerical value of a monitoring item based on the real-time monitoring data, thereby performing monitoring. The method can monitor the running state of the big data assembly without depending on an external assembly, and can be applied to any environment with the big data assembly.
On the basis of the monitoring method for the big data component shown in fig. 1, the present embodiment further provides another monitoring method for the big data component, where the method focuses on describing a specific implementation process of step S104 (determining a value of a preset monitoring item according to the real-time monitoring data) in the foregoing embodiment, and as shown in fig. 3, the method is a schematic flow chart of another monitoring method for the big data component, where the method includes the following steps:
step S302: and acquiring real-time monitoring data of the big data component to be monitored through a preset interface.
Here, step S302 in this embodiment corresponds to step S102 in the foregoing embodiment, and corresponding descriptions may refer to corresponding parts of the foregoing embodiment, which are not described herein again.
Step S304: and converting the real-time monitoring data into data in a specified storage format.
In this embodiment, in order to obtain a numerical value of a preset monitoring item according to the obtained real-time monitoring data of the big data component to be monitored, the real-time monitoring data needs to be cleaned, calculated, or analyzed, and for convenience of post-processing, a format of the monitoring data is first converted into a unified specified storage format, for example, the specified storage format may be a JSON format. The JSON (JSON Object Notation) is a lightweight data exchange format, and a simple and clear hierarchical structure makes the JSON become an ideal data exchange language, and the JSON is also easy to analyze and generate by a machine, and effectively improves the network transmission efficiency.
Step S306: and analyzing the data after format conversion according to a preset metadata rule to obtain a numerical value of a preset monitoring item.
In at least one possible embodiment, the data after the format conversion may be parsed through the following steps 11 to 13:
(11) and sending the data to a theme corresponding to the data type in the specified message queue according to the data type of the data after format conversion.
Here, the specified message queue may be Kafkatopic, where Kafka is a big data technology component, essentially a data transfer queue, and topic is the subject of Kafka distinguishing different data types.
In actual operation, according to the data type of the data after format conversion, the data can be sent to a topic corresponding to the data type in a specified message queue through a preset proxy service. In one possible implementation, the proxy service may be a proxy service, where the proxy service runs on a first server, and sends the formatted data to the specified message queue (e.g., Kafkatopic) via the proxy service, and different data types are correspondingly stored in different topics (topic) in the message queue.
(12) And drawing data from the subject matched with the stream program in the message queue through a preset stream program.
Here, the Stream program (Stream) and the topic (topic) in the message queue are in one-to-one correspondence, and generally, a MessageType is carried in each topic data to distinguish the message types. For example, if the data in one topic carries a MessageType of "a" indicating that it belongs to a class a product, the stream program will process the batch of data according to the preconfigured monitoring index of the class a product.
Thus, the consuming program (stream program) subscribes to topic data in the message queue through spark streaming (a distributed stream computing engine), consumes the data once every period of time, and pulls the data from the message queue.
(13) And analyzing the pulled data according to a preset metadata rule to obtain a numerical value of a preset monitoring item.
The metadata is data describing data, and mainly is information describing data attributes, and the metadata rule in this embodiment refers to a rule for processing data based on data attributes. For example, the metadata rule may be to discard data of a certain attribute, or to combine data of several attributes to obtain new data, and so on.
The analysis processing of the data includes cleaning of the data, calculation (operations such as addition, subtraction, multiplication, and division), and the like. Suppose that a style of a metric is A: Tags { "T1": 0 "," T2 ": 1" }, where the metric A has two attributes T1 and T2, but if only the value of the get attribute T1 is configured inside the metadata rule, the attribute of T2 is lost.
Step S308: and monitoring the big data component to be monitored according to the numerical value of the monitoring item.
In at least one possible embodiment, the big data component to be monitored may be monitored through the following steps 22-24:
(22) and saving the numerical value of the monitoring item to a specified database.
In order to draw a visual map according to the data in the designated database more conveniently at a later stage, the values of the monitoring items can be saved into a time sequence database (e.g., Opentsdb), wherein the time sequence database is a time sequence database, which is mainly used for processing data with time tags (changing according to the time sequence, namely time sequencing), namely time sequence data. The time sequence big data can be efficiently stored and rapidly processed, and is an important technology for solving the problem of processing of mass data. And compared with a relational database, the storage space of the relational database is halved, and the query speed is greatly improved.
(23) And reading the value of the monitoring item in the target time period from the specified database according to a preset period.
In actual operation, the specified database can be polled according to a preset period through a preset data query service, so as to obtain the value of the monitoring item in a target time period. For example, corresponding data can be obtained according to the target monitoring item and the target time period through a data Query service Query API, for example, data of the index a before one day, and then all data of the index a in the last day can be obtained through Query of the Query API.
(24) And determining the running state of the monitoring item in the target time period according to the read value of the monitoring item.
In order to more intuitively acquire the running state of the monitoring item in the target time period, in one possible embodiment, the running state of the monitoring item in the target time period may be presented on the graphical display interface of the first server through a preset visualization program. For example, a trend graph can be drawn by the Grafana component on the numerical value of the monitoring item acquired from the specified database in the target time period, and the running state of the monitoring item of the big data component to be monitored is visually presented to the user.
In order to more clearly understand the monitoring method for the big data component provided by this embodiment, this embodiment further provides an application example, see fig. 4, which is a schematic diagram of a technical framework for implementing big data component monitoring, in the implementation shown in fig. 4, an exporter service developed based on a Prometheus protocol first collects real-time monitoring data of a specified big data component service (e.g., HBase, etc.) through a preset external interface (e.g., JMX API, etc.), then converts the collected data into a JSON format, and reports the JSON to a proxy service at a center, the proxy sends the data to a specified Kafka topic, and correspondingly, a stream program consumes each topic data, and cleans, calculates, analyzes, etc. the pulled data of the stream program according to a preset metadata rule, and stores a result set in a batch in an opentsdb database. In addition, the database shown in fig. 4 is further provided with a QueryAPI interface to provide data query services, and the running state of the monitoring item of the big data component to be monitored is conveniently displayed through the data visualization technology Grafana.
In the monitoring method for the big data component provided by this embodiment, the obtained real-time monitoring data is converted into stream data, and then the stream data is cleaned and calculated according to a preset rule by combining with the metadata management service, so as to obtain data of an actual monitoring item, thereby greatly relieving the pressure on data storage in the monitoring of the big data component; in addition, the data of the actual monitoring item is stored in the time sequence database, and the data of the target monitoring item can be flexibly and conveniently inquired by combining the data inquiry service and the visualization technology, and the monitoring state can be more intuitively presented to the user.
Corresponding to the monitoring method for the big data component shown in fig. 1, an embodiment of the present invention further provides a monitoring apparatus for a big data component, referring to fig. 5, which is a schematic structural diagram of a monitoring apparatus for a big data component, wherein the monitoring apparatus is applied to a first server, the first server is communicatively connected to a second server running a big data component to be monitored, as can be seen from fig. 5, the monitoring apparatus includes a real-time monitoring data obtaining module 51, a monitoring item value determining module 52 and a monitoring module 53, which are connected in sequence, where functions of the modules are as follows:
the real-time monitoring data acquisition module 51 is configured to acquire real-time monitoring data of the big data component to be monitored through a preset interface;
a monitoring item value determining module 52, configured to determine a value of a preset monitoring item according to the real-time monitoring data;
and the monitoring module 53 is configured to monitor the big data component to be monitored according to the value of the monitoring item.
According to the monitoring side device of the big data component, provided by the embodiment of the invention, the real-time monitoring data of the big data component to be monitored is acquired through the preset interface; determining the numerical value of a preset monitoring item according to the real-time monitoring data; and monitoring the big data component to be monitored according to the numerical value of the monitoring item. In the device, through the preset external interface of big data subassembly itself or the preset external interface of big data arrangement frame, acquire the real-time supervision data of subassembly, then carry out analytic processing to this real-time supervision data, obtain the numerical value of monitoring item, and then monitor the subassembly according to the numerical value of this monitoring item, this mode need not rely on external component to realize monitoring the running state of big data subassembly promptly, it can be applied to and has better monitoring flexibility in the environment that has big data subassembly in arbitrary arrangement.
In one possible implementation, the real-time monitoring data obtaining module 51 is further configured to: and reading a preset first external interface of the big data assembly to be monitored, or reading a preset second external interface of a big data deployment frame in the second server, so as to obtain real-time monitoring data of the big data assembly to be monitored.
In another possible embodiment, the real-time monitoring data obtaining module 51 is further configured to: and reading a preset first external interface of the big data assembly to be monitored through a preset data acquisition service.
In another possible embodiment, the data collection service is developed based on Prometheus protocol.
In another possible implementation, the first external interface includes a Java management extended application interface, where the Java management extended application interface is configured to externally provide the running state data of the big data component to be monitored.
In another possible implementation, the above monitoring item number determination module 52 is further configured to: converting the real-time monitoring data into data in a specified storage format; and analyzing the data after format conversion according to a preset metadata rule to obtain a numerical value of a preset monitoring item.
In another possible implementation, the above monitoring item number determination module 52 is further configured to: according to the data type of the data after format conversion, sending the data to a theme corresponding to the data type in a specified message queue; through a preset stream program, data are pulled from the theme matched with the stream program in the message queue; and analyzing the pulled data according to a preset metadata rule to obtain a numerical value of a preset monitoring item.
In another possible implementation, the above monitoring item number determination module 52 is further configured to: and sending the data to a theme corresponding to the data type in the appointed message queue through preset proxy service according to the data type of the data after format conversion.
In another possible embodiment, the specified storage format includes a JSON format.
In another possible embodiment, the monitoring module 53 is further configured to: saving the numerical value of the monitoring item to a specified database; reading the value of the monitoring item in the target time period from the specified database according to a preset period; and determining the running state of the monitoring item in the target time period according to the read value of the monitoring item.
In another possible embodiment, the monitoring module 53 is further configured to: and polling the specified database according to a preset period through a preset data query service to obtain the numerical value of the monitoring item in a target time period.
In another possible embodiment, the type of the designated database is a time-series database.
In another possible embodiment, the apparatus further includes a visualization presenting module, configured to present, through a preset visualization program, an operation status of the monitoring item in the target time period on a graphical display interface of the first server.
The implementation principle and the generated technical effect of the monitoring device for the big data component provided by the embodiment of the invention are the same as those of the monitoring method embodiment for the big data component, and for brief description, corresponding contents in the monitoring method embodiment for the big data component can be referred to for the part of the embodiment of the monitoring device for the big data component which is not mentioned.
An embodiment of the present invention further provides an electronic device, as shown in fig. 6, which is a schematic structural diagram of the electronic device, where the electronic device includes a processor 61 and a memory 62, the memory 62 stores machine executable instructions capable of being executed by the processor 61, and the processor 61 executes the machine executable instructions to implement the monitoring method for the big data component.
In the embodiment shown in fig. 6, the electronic device further comprises a bus 63 and a communication interface 64, wherein the processor 61, the communication interface 64 and the memory 62 are connected by the bus.
The Memory 62 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 64 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.
The processor 61 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 61. The Processor 61 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory, and the processor 61 reads the information in the memory 62, and completes the steps of the monitoring method of the big data component of the foregoing embodiment in combination with the hardware thereof.
The embodiment of the present invention further provides a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the monitoring method for the big data component, and specific implementation may refer to the foregoing method embodiment, and is not described herein again.
The monitoring method for a big data component, the monitoring apparatus for a big data component, and the computer program product for an electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the monitoring method for a big data component described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (15)
1. A big data component monitoring method is applied to a first server which is in communication connection with a second server running a big data component to be monitored, and the method comprises the following steps:
acquiring real-time monitoring data of the big data component to be monitored through a preset interface;
determining the numerical value of a preset monitoring item according to the real-time monitoring data;
and monitoring the big data component to be monitored according to the numerical value of the monitoring item.
2. The big data component monitoring method according to claim 1, wherein the step of obtaining real-time monitoring data of the big data component to be monitored through a preset interface comprises:
and reading a preset first external interface of the big data assembly to be monitored, or reading a preset second external interface of a big data deployment frame in the second server, so as to obtain real-time monitoring data of the big data assembly to be monitored.
3. The big data component monitoring method according to claim 2, wherein the step of reading the preset first external interface of the big data component to be monitored comprises:
and reading a preset first external interface of the big data assembly to be monitored through a preset data acquisition service.
4. The big data component monitoring method according to claim 2, wherein the first external interface comprises a Java management extension application program interface, and wherein the Java management extension application program interface is configured to externally provide the running state data of the big data component to be monitored.
5. The big data component monitoring method according to claim 1, wherein the step of determining the value of the preset monitoring item according to the real-time monitoring data comprises:
converting the real-time monitoring data into data in a specified storage format;
and analyzing the data after format conversion according to a preset metadata rule to obtain a numerical value of a preset monitoring item.
6. The big data component monitoring method according to claim 5, wherein the step of parsing the format-converted data according to a preset metadata rule to obtain a value of a preset monitoring item includes:
according to the data type of the data after format conversion, sending the data to a theme corresponding to the data type in a specified message queue;
through a preset stream program, data are pulled from the theme matched with the stream program in the message queue;
and analyzing the pulled data according to a preset metadata rule to obtain a numerical value of a preset monitoring item.
7. The big data component monitoring method according to claim 6, wherein the step of sending the data to a topic corresponding to the data type in a specified message queue according to the data type of the data after format conversion comprises:
and sending the data to a theme corresponding to the data type in a specified message queue through a preset proxy service according to the data type of the data after format conversion.
8. The big data component monitoring method of claim 5, wherein the specified storage format comprises a JSON format.
9. The big data component monitoring method according to claim 1, wherein the step of monitoring the big data component to be monitored according to the value of the monitoring item comprises:
saving the numerical value of the monitoring item to a specified database;
reading the numerical value of the monitoring item in a target time period from the specified database according to a preset period;
and determining the running state of the monitoring item in the target time period according to the read value of the monitoring item.
10. The big data component monitoring method according to claim 9, wherein the step of reading the value of the monitored item in the target time period from the designated database at a preset cycle comprises:
and polling the specified database according to a preset period through a preset data query service to obtain the numerical value of the monitoring item in a target time period.
11. The big data component monitoring method according to claim 9, wherein the type of the designated database is a time series database.
12. The big data component monitoring method of claim 9, further comprising:
and displaying the running state of the monitoring item in the target time period on a graphical display interface of the first server through a preset visualization program.
13. A big data component monitoring apparatus, the apparatus being applied to a first server communicatively connected to a second server running a big data component to be monitored, the apparatus comprising:
the real-time monitoring data acquisition module is used for acquiring real-time monitoring data of the big data component to be monitored through a preset interface;
the monitoring item value determining module is used for determining the value of a preset monitoring item according to the real-time monitoring data;
and the monitoring module is used for monitoring the big data component to be monitored according to the numerical value of the monitoring item.
14. An electronic device comprising a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the big data component monitoring method of any of claims 1 to 12.
15. A computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to implement the big data component monitoring method of any of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010847304.7A CN112000548A (en) | 2020-08-20 | 2020-08-20 | Big data component monitoring method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010847304.7A CN112000548A (en) | 2020-08-20 | 2020-08-20 | Big data component monitoring method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112000548A true CN112000548A (en) | 2020-11-27 |
Family
ID=73472876
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010847304.7A Pending CN112000548A (en) | 2020-08-20 | 2020-08-20 | Big data component monitoring method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112000548A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112835964A (en) * | 2021-01-27 | 2021-05-25 | 上海臣星软件技术有限公司 | Big data index data display method, device and equipment and computer storage medium |
| CN113407607A (en) * | 2021-06-22 | 2021-09-17 | 中国联合网络通信集团有限公司 | Multi-cloud heterogeneous data processing method and device and electronic equipment |
| CN116431688A (en) * | 2022-11-14 | 2023-07-14 | 北京远舢智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
-
2020
- 2020-08-20 CN CN202010847304.7A patent/CN112000548A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112835964A (en) * | 2021-01-27 | 2021-05-25 | 上海臣星软件技术有限公司 | Big data index data display method, device and equipment and computer storage medium |
| CN113407607A (en) * | 2021-06-22 | 2021-09-17 | 中国联合网络通信集团有限公司 | Multi-cloud heterogeneous data processing method and device and electronic equipment |
| CN113407607B (en) * | 2021-06-22 | 2023-06-27 | 中国联合网络通信集团有限公司 | Multi-cloud heterogeneous data processing method and device and electronic equipment |
| CN116431688A (en) * | 2022-11-14 | 2023-07-14 | 北京远舢智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11727039B2 (en) | Low-latency streaming analytics | |
| CN112000548A (en) | Big data component monitoring method and device and electronic equipment | |
| CN107145489B (en) | Information statistics method and device for client application based on cloud platform | |
| CN111294217B (en) | Alarm analysis method, device, system and storage medium | |
| Hauser et al. | Reviewing cloud monitoring: Towards cloud resource profiling | |
| CN101989931A (en) | Operation alarm processing method and device | |
| CN114090366A (en) | Method, device and system for monitoring data | |
| CN112749153A (en) | Industrial network data management system | |
| CN116166505B (en) | Monitoring platform, method, storage medium and equipment for dual-state IT architecture in financial industry | |
| CN111339052A (en) | Unstructured log data processing method and device | |
| Bielefeld | Online performance anomaly detection for large-scale software systems | |
| CN113111261A (en) | Data processing method of cloud platform, cloud platform and panoramic analysis system | |
| JP2004348640A (en) | Method and system for managing network | |
| CN110928750B (en) | Data processing method, device and equipment | |
| CN110633191A (en) | Method and system for monitoring service health degree of software system in real time | |
| CN111427749A (en) | Monitoring tool and method for ironic service in openstack environment | |
| CN105446707B (en) | Data conversion method | |
| CN115222181B (en) | Robot operation state monitoring system and method | |
| CN114580674A (en) | Method for processing equipment operation information and electronic equipment | |
| CN114625763A (en) | Information analysis method and device for database, electronic equipment and readable medium | |
| CN110971483B (en) | Pressure testing method and device and computer system | |
| EP3465562A1 (en) | Data driven invocation of real time wind market forecasting analytics | |
| CN113407412A (en) | Micro-service monitoring method and system | |
| CN108694107A (en) | Backlog monitoring method, device, readable medium and the electronic equipment of message queue | |
| CN112988443A (en) | Method and device for processing business exception |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |