CN110716841A - Monitoring data collection method, device and equipment - Google Patents
Monitoring data collection method, device and equipment Download PDFInfo
- Publication number
- CN110716841A CN110716841A CN201910878194.8A CN201910878194A CN110716841A CN 110716841 A CN110716841 A CN 110716841A CN 201910878194 A CN201910878194 A CN 201910878194A CN 110716841 A CN110716841 A CN 110716841A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- log
- collected
- logs
- distributed system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 461
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000013480 data collection Methods 0.000 title claims abstract description 48
- 238000013500 data storage Methods 0.000 claims abstract description 62
- 230000002159 abnormal effect Effects 0.000 claims abstract description 38
- 238000012546 transfer Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 14
- 238000004891 communication Methods 0.000 claims description 13
- 230000009467 reduction Effects 0.000 abstract description 9
- 230000006870 function Effects 0.000 description 20
- 230000005540 biological transmission Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
Landscapes
- Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a monitoring data collection method, a monitoring data collection device and monitoring data collection equipment. The monitoring data collection method is applied to a log data collector, and when the monitoring log of a distributed system is detected to be updated, the monitoring log updated by a central node of the distributed system is obtained and used as the monitoring log to be collected; the monitoring log updated by the central node is a log obtained by recording the monitoring data of the distributed system by the central node; and sending the monitoring log to be collected to a monitoring data storage device for storage. The scheme can reduce the problem of performance reduction of the central node caused by abnormal collection of monitoring data.
Description
Technical Field
The present invention relates to the field of monitoring data collection technologies, and in particular, to a monitoring data collection method, apparatus, and device.
Background
In order to ensure that when the distributed system is abnormal, the abnormal operation can be analyzed and processed according to the related operation data, the operation condition of the distributed system needs to be monitored, the monitoring data is obtained, and the obtained monitoring data is collected. In the related art, a class library of the distributed publish-subscribe message tool may be packaged and stored in a central node of the distributed system. When the monitoring data needs to be collected, the central node can directly call the publishing function in the class library, and publish the monitoring data to the distributed publishing and subscribing message tool for storage.
However, in the above collection method, the central node invokes the publish function of the distributed publish-subscribe message tool, and there is a high coupling with the distributed publish-subscribe message tool. Therefore, when the distributed publish-subscribe message tool is abnormal, a publish function called by the central node is abnormal in deadlock, and further a deadlock thread occurs in the central node calling the publish function. Especially, when the central node collects monitoring data concurrently and calls a great amount of release functions, a great amount of dead threads appear in the central node, and processing resources of the central node are occupied, so that performance of the central node for processing other services is reduced and even abnormal.
Disclosure of Invention
The embodiment of the invention aims to provide a monitoring data collection method, a monitoring data collection device and monitoring data collection equipment, so as to achieve the effect of reducing performance reduction of a central node caused by data collection abnormity in a distributed system. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a monitoring data collection method,
applied to a log data collector, the method comprising:
when detecting that the monitoring log of the distributed system is updated, acquiring the monitoring log updated by the central node of the distributed system as the monitoring log to be collected; the monitoring log updated by the central node is a log obtained by recording the monitoring data of the distributed system by the central node;
and sending the monitoring log to be collected to a monitoring data storage device for storage.
Optionally, the sending the monitoring log to be collected to a monitoring data storage device for storage includes:
calling a hypertext transfer protocol (HTTP) interface, and detecting whether the calling of the HTTP interface is abnormal;
if the monitoring logs to be collected are abnormal, stopping sending the monitoring logs to be collected, calling a hypertext transfer protocol HTTP interface to send the acquired monitoring logs to be collected to a monitoring data storage device for storage according to the sequence from the monitoring logs to be collected which are stopped to be sent when the HTTP interface is detected to be capable of being called, and otherwise, sending the monitoring logs to be collected to the monitoring data storage device for storage by using the called HTTP interface.
Optionally, before sending the monitoring log to be collected to a monitoring data storage device for storage, the method further includes:
judging whether the number of the acquired monitoring logs to be collected is a specified number;
if the number of the monitoring logs is not the specified number, returning to execute the step of obtaining the updated monitoring logs as the monitoring logs to be collected when the monitoring logs of the distributed system are detected to be updated, otherwise, sending the monitoring logs to be collected to a monitoring data storage device for storage, wherein the step comprises the following steps:
and sending the acquired monitoring logs to be collected to the monitoring data storage device for storage.
Optionally, the log data collector includes:
a log data collector having a plug-in architecture.
Optionally, the monitoring log includes:
and the central node of the distributed system records and stores a log obtained by monitoring data of the distributed system according to a preset network script language object numbered notation JSON format.
In a second aspect, an embodiment of the present invention provides a monitoring data collection apparatus, which is applied to a log data collector, and includes:
the monitoring log acquiring module is used for acquiring the monitoring log updated by the central node of the distributed system when the monitoring log of the distributed system is detected to be updated, and the monitoring log is used as the monitoring log to be collected; the monitoring log updated by the central node is a log obtained by recording the monitoring data of the distributed system by the central node;
and the monitoring log sending module is used for sending the monitoring logs to be collected to a monitoring data storage device for storage.
Optionally, the monitoring log sending module is specifically configured to:
calling a hypertext transfer protocol (HTTP) interface, and detecting whether the calling of the HTTP interface is abnormal;
if the monitoring logs to be collected are abnormal, stopping sending the monitoring logs to be collected, calling a hypertext transfer protocol HTTP interface to send the acquired monitoring logs to be collected to a monitoring data storage device for storage according to the sequence from the monitoring logs to be collected which are stopped to be sent when the HTTP interface is detected to be capable of being called, and otherwise, sending the monitoring logs to be collected to the monitoring data storage device for storage by using the called HTTP interface.
Optionally, the monitoring log sending module is specifically configured to:
before sending the monitoring logs to be collected to a monitoring data storage device for storage, judging whether the number of the acquired monitoring logs to be collected is a specified number; and if the number of the monitoring logs is not the specified number, triggering the monitoring log obtaining module to obtain the updated monitoring logs as the monitoring logs to be collected when the monitoring logs of the distributed system are detected to be updated, otherwise, sending the obtained monitoring logs to be collected to the monitoring data storage device for storage.
Optionally, the log data collector includes:
a log data collector having a plug-in architecture.
Optionally, the monitoring log includes:
and the central node of the distributed system records and stores a log obtained by monitoring data of the distributed system according to a preset network script language object numbered notation JSON format.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the bus; a memory for storing a computer program; and a processor, configured to execute the program stored in the memory, and implement the steps of the monitoring data collection method provided in the first aspect.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, the computer program implements the steps of the monitoring data collection method provided in the first aspect.
In the scheme provided by the embodiment of the invention, the central node of the distributed system records the monitoring data of the distributed system as a monitoring log; and the log data collector can obtain the monitoring log updated by the central node as the monitoring log to be collected when detecting that the monitoring log of the distributed system is updated, and send the monitoring log to be collected to the monitoring data storage device for storage, thereby realizing the collection of the monitoring data of the distributed system. Compared with the monitoring data collection by directly calling the publishing function of the distributed publishing and subscribing message tool by the central node, when the monitoring data is collected, the central node records and stores the monitoring log to be collected as the monitoring log, and the log data collector acquires and sends the monitoring log to be collected so as to collect the monitoring data. Therefore, the functions of the central node and the monitoring data collection can be decoupled, and the problem of performance reduction of the central node caused by abnormal monitoring data collection is solved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
Fig. 1 is a schematic flow chart of a monitoring data collection method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating a monitoring data collection method according to another embodiment of the present invention;
fig. 3 is an exemplary diagram of an application scenario of a monitoring data collection method according to another embodiment of the present invention;
fig. 4 is a schematic structural diagram of a monitoring data collection apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
First, a monitoring data collection method according to an embodiment of the present invention will be described.
The monitoring data collection method provided by the embodiment of the invention can be applied to a log data collector. The log data collector may be varied in a particular application. Illustratively, the log data collector may be a virtual appliance, or, an electronic device. Moreover, the log data collector can be installed on a central node of the distributed system and forms a device with the central node; or may be independent of the central node of the distributed system. The electronic device may specifically include a desktop computer, a portable computer, an internet television, an intelligent mobile terminal, a server, a wearable intelligent terminal, and the like, which are not limited herein, and any electronic device that can implement the embodiment of the present invention belongs to the protection scope of the embodiment of the present invention.
In addition, a distributed system may be various. Illustratively, the distributed system may be an audio-video interconnection system, such as a live system and a social system, among others. Alternatively, the distributed system may be, for example, a shopping system, an internet-related game system, and the like, and is not limited thereto. The central node of the distributed system may specifically include a desktop computer, a portable computer, an internet television, an intelligent mobile terminal, a server, a wearable intelligent terminal, and the like, which is not limited herein. Any distributed system and corresponding central node that needs to collect the monitoring data of the system belong to the protection scope of the embodiment of the invention.
As shown in fig. 1, a flow of a monitoring data collection method according to an embodiment of the present invention may include the following steps:
s101, when detecting that the monitoring log of the distributed system is updated, acquiring the monitoring log updated by the central node of the distributed system as the monitoring log to be collected; the monitoring log updated by the central node is a log obtained by recording the monitoring data of the distributed system by the central node.
In order to avoid that the central node of the distributed system directly calls the publishing function in the class library of the distributed publishing and subscribing message tool and the monitoring data is published, the central node of the distributed system is coupled with the distributed publishing and subscribing tool, and the central node of the distributed system can record the monitoring data of the distributed system as the monitoring log instead of calling the publishing function in the class library of the distributed publishing and subscribing message tool. On the basis, the log data collector can collect the monitoring logs, so that the collection of the monitoring logs is decoupled from the central node, and the problem of performance reduction of the central node caused by abnormal collection of the monitoring data is solved.
The monitoring log may be various. This is explained in more detail below in the form of alternative embodiments. In an optional implementation manner, the monitoring log may specifically include:
and the central node of the distributed system records the monitoring data of the distributed system and stores the obtained text document.
A text document is a widely used document format for storing text information, and can be viewed using most software, such as notepad, browser, and so on. Therefore, the monitoring data is saved as a text document to be conveniently referred to.
In another optional implementation, the monitoring log may specifically include:
and the central node of the distributed system records and stores a log obtained by monitoring data of the distributed system according to a preset simplified notation JSON format of the network script language object.
The web scripting language Object Notation JSON (JavaScript Object Notation) is a lightweight data exchange format. It is based on a subset of ECMAScript (the specification of scripting languages laid down by the european computer association) and uses a text format completely independent of the programming language to store and represent data. The JSON format has a simple and clear hierarchical structure, is easy to analyze and generate by a machine and is easy to read and write. Moreover, the characteristics of easy machine analysis and generation can ensure that the data in the JSON format is convenient to transmit, and the network transmission efficiency of the data can be improved.
Therefore, in this optional embodiment, the central node of the distributed system records and stores the monitoring data as the monitoring log in the JSON format, which is beneficial to improving the generation efficiency of the monitoring log and the transmission efficiency of the subsequent monitoring log. Any format of monitoring log can be used in the present invention, and the present embodiment does not limit this.
In an optional implementation manner, the log data collector may specifically include:
a log data collector having a plug-in architecture.
In a particular application, the log data collector may be an open source data collector fluent. Fluentd is a log data collector designed specifically for processing data streams, with a plug-in architecture. Accordingly, the data collector has relatively high expandability and usability compared with the data collector with a non-plug-in type framework.
In this embodiment, the log data collector may detect that there is an update in the monitoring log of the distributed system, and may be in various ways. For example, the log data collector may monitor whether a new monitoring log exists in the distributed system in real time, and detect that the monitoring log has an update when it is monitored that the new monitoring log exists. Or, for example, the log data collector may query, according to a preset period, whether a new monitoring log exists in the monitoring logs of the distributed system, and detect that the monitoring logs have updates when it is monitored that the new monitoring log exists. Compared with the mode that the central node of the distributed system actively informs the log data collector that the monitoring log is updated, the mode that the log data collector detects that the monitoring log of the distributed system is updated does not need the central node to set a process aiming at log update notification and interaction between the central node and the log data collector, and decoupling between the central node and monitoring data collection is facilitated.
Any way that the log data collector detects that the monitoring log of the distributed system has update can be used in the present invention, and this embodiment does not limit this.
In addition, the log data collector obtains the monitoring log updated by the central node of the distributed system, and the monitoring log to be collected may be in various ways. For example, when the number of the monitoring logs is one, for example, the central node records and stores all monitoring data of the distributed system in one monitoring log, and the log data collector obtains updated contents in the monitoring log as the monitoring log to be collected. Or, for example, when the number of the monitoring logs is multiple, for example, the central node records and stores one monitoring data, a specified number of monitoring data, or monitoring data within a specified generation time range in one monitoring log, the log data collector obtains a monitoring log different from the monitoring log before updating as the monitoring log to be collected.
In addition, when the number of updated monitoring logs is plural, the log data collector may record the acquisition position of the acquired monitoring log in real time, so that the monitoring log is acquired from the acquisition position of the most recently recorded monitoring log when the monitoring log update is detected next time. For example, when a monitoring log update is detected for the first time, the acquired position of the acquired monitoring log ML1 is P1, and when a monitoring log update is detected for the second time, there are the updated monitoring log ML2 stored at the position P2 and the updated monitoring log ML3 stored at the position P3, and at this time, the monitoring log ML2 and the monitoring log ML3 may be acquired one by one or in batch from the position P2.
Any way that the log data collector obtains the monitoring log updated by the central node of the distributed system as the monitoring log to be collected can be used in the present invention, which is not limited in this embodiment.
In addition, the number of central nodes of the distributed system may be varied. Illustratively, the number of the central nodes of the distributed system may be one, or alternatively, a plurality. It can be understood that, when the number of the central nodes of the distributed system is multiple, a corresponding log data collector may be set for each central node, and the corresponding log data collector collects the monitoring logs to be collected, which are stored by the central node, according to the method provided by the embodiment of the present invention. Or, when the number of the central nodes of the distributed system is multiple, a log data collector may be set, and the log data collector collects the monitoring logs to be collected, which are stored in each central node, according to the method provided by the embodiment of the present invention.
And S102, sending the monitoring log to be collected to a monitoring data storage device for storage.
In a specific application, the log data collector may send the monitoring log to be collected to the monitoring data storage device for storage in various ways. For example, when the electronic device equipped with the log data collector or the log data collector itself has a bluetooth transmission function and the monitoring data storage device has a bluetooth transmission function, the log data collector may send the monitoring log to be collected to the monitoring data storage device for storage by using bluetooth. Or, for example, the log data collector may call a hypertext transfer protocol HTTP interface to send the monitoring log to be collected to the monitoring data storage device for storage.
Any way that the log data collector sends the monitoring log to be collected to the monitoring data storage device for storage can be used in the present invention, which is not limited in this embodiment.
Also, the monitoring data storage means may be various. Illustratively, the monitoring data storage device may be a distributed publish-subscribe message tool, for example, a distributed publish-subscribe message tool Kafka, which has relatively high throughput, built-in partitioning, data copy support, and fault tolerance characteristics, and can well cope with large-scale monitoring log storage scenarios. Alternatively, the monitoring data store may be, for example, a Redis database or the like. Any device capable of storing the monitoring log can be used as the monitoring data storage device of the present invention, and this embodiment does not limit this.
In the scheme provided by the embodiment of the invention, the central node of the distributed system records the monitoring data of the distributed system as a monitoring log; and the log data collector can obtain the monitoring log updated by the central node as the monitoring log to be collected when detecting that the monitoring log of the distributed system is updated, and send the monitoring log to be collected to the monitoring data storage device for storage, thereby realizing the collection of the monitoring data of the distributed system. Compared with the monitoring data collection by directly calling the publishing function of the distributed publishing and subscribing message tool by the central node, when the monitoring data is collected, the central node records and stores the monitoring log to be collected as the monitoring log, and the log data collector acquires and sends the monitoring log to be collected so as to collect the monitoring data. Therefore, the functions of the central node and the monitoring data collection can be decoupled, and the problem of performance reduction of the central node caused by abnormal monitoring data collection is solved.
Optionally, before sending the monitoring log to be collected to the monitoring data storage device for storage, the monitoring data collection method provided in the embodiment of the present invention may further include the following steps:
judging whether the number of the acquired monitoring logs to be collected is a specified number;
if the number of the monitoring logs is not the specified number, returning to execute the step of obtaining the updated monitoring logs as the monitoring logs to be collected when the monitoring logs of the distributed system are detected to be updated, otherwise, sending the monitoring logs to be collected to the monitoring data storage device for storage, wherein the step may include:
and sending the acquired monitoring logs to be collected to a monitoring data storage device for storage.
In a specific application, the specified number is a number greater than 1, so as to ensure that the number of the acquired monitoring logs to be collected is multiple. When the number of the acquired monitoring logs to be collected is not the designated number, the step of acquiring the updated monitoring logs as the monitoring logs to be collected when the monitoring logs of the distributed system are detected to be updated can be returned to, so as to acquire the designated number of the monitoring logs to be collected, and realize the batch acquisition of the monitoring logs to be collected.
The acquired monitoring logs to be collected are sent to a monitoring data storage device for storage, and the monitoring logs to be collected can be sent in batch, or the monitoring logs to be collected can be sent one by one.
In this optional embodiment, by obtaining the monitoring logs to be collected in batch, the problems of time waste and performance degradation of the log data reader caused by frequently obtaining the monitoring logs to be collected can be reduced. In addition, when monitoring logs acquired in batch are sent in batch, the problems of time waste and performance reduction of a log data reader caused by frequent sending can be reduced.
As shown in fig. 2, a flow of a monitoring data collection method according to another embodiment of the present invention may include the following steps:
s201, when detecting that the monitoring log of the distributed system is updated, acquiring the monitoring log updated by the central node of the distributed system as the monitoring log to be collected. The monitoring log updated by the central node is a log obtained by recording the monitoring data of the distributed system by the central node.
S201 is the same as S101 in the embodiment of fig. 1, and is not repeated herein, for details, see the description of the embodiment of fig. 1.
S202, calling a hypertext transfer protocol (HTTP) interface, and detecting whether the calling of the HTTP interface is abnormal; if abnormal, step S203 is performed, and if normal, step S204 is performed.
In a specific application, the log data collector can call a hypertext transfer protocol (HTTP) interface, so that the HTTP interface is used for sending the monitoring log to be collected to the monitoring data storage device for storage. Compared with the Bluetooth, the HTTP interface can reduce the hardware limitation in sending the monitoring log and improve the convenience.
On this basis, in order to deal with the collection abnormality of the monitoring log to be collected caused by the call abnormality of the HTTP interface, whether the call to the HTTP interface is abnormal or not may be detected when the HTTP interface is called. If the HTTP interface is abnormal, it indicates that the HTTP interface cannot normally transmit the monitoring log to be collected, so step S203 may be executed. If it is normal, it indicates that the HTTP interface can normally transmit the monitoring log to be collected at this time, and therefore, step S204 may be performed.
The log data collector may detect whether the call to the HTTP interface is abnormal, and may specifically be in various manners. For example, the log data collector may detect a response duration of the HTTP interface to the call, and detect that the call to the HTTP interface is abnormal when the response duration is detected to be greater than a preset duration. Or, for example, the log data collector may detect whether information on a call failure fed back by the task manager is received, and detect a call exception to the HTTP interface when the information on the call failure is received. Any method capable of detecting whether the call to the HTTP interface is abnormal may be used in the present invention, and this embodiment does not limit this.
And S203, stopping sending the monitoring logs to be collected, and calling the hypertext transfer protocol HTTP interface to send the acquired monitoring logs to be collected to the monitoring data storage device for storage according to the sequence from the monitoring logs to be collected, which are stopped to be sent, to the HTTP interface when the HTTP interface is detected to be capable of being called.
In a specific application, the log data collector calls a hypertext transfer protocol HTTP interface to send the acquired monitoring log to be collected to the monitoring data storage device for storage according to an order from the monitoring log to be collected that is stopped from being sent. For example, when the log data collector records the acquisition position of the currently acquired monitoring log in real time and acquires the monitoring log according to the next acquisition position of the latest recorded acquisition position, the log data collector may stop recording the acquisition position of the currently acquired monitoring log while stopping sending the monitoring log to be collected, so as to ensure that the acquisition position of the recorded monitoring log is the acquisition position of the monitoring log acquired last time, and the acquisition position of the monitoring log acquired when the HTTP interface is abnormal is not recorded; when detecting that the HTTP interface can be called, the monitoring log may be obtained from a position next to the position where the monitoring log is obtained from the latest record, and correspondingly, the hypertext transfer protocol HTTP interface may be called to send the obtained monitoring log to be collected to the monitoring data storage device for storage, starting from the monitoring log to be collected that is stopped from being sent. For example, the acquisition position of the monitoring log acquired when the HTTP interface is abnormal is position P2, and is not recorded; thus, when it is detected that the HTTP interface can be called, the acquisition position of the most recently recorded monitoring log is position P1, and the next acquisition position of position P1 is position P2. Therefore, the acquisition of the monitoring log can be performed from the position P2, and accordingly, the acquired monitoring log to be collected can be transmitted to the monitoring data storage device for storage by calling the hypertext transfer protocol HTTP interface from the monitoring log to be collected whose transmission is stopped.
Or, for example, the log data collector may record a reading position of the currently acquired monitoring log when the reading is stopped, and further, when it is detected that the HTTP interface can be called, start to read the monitoring log from the recorded acquiring position, and ensure that the obtained monitoring log to be collected is sent to the monitoring data storage device for storage by calling the hypertext transfer protocol HTTP interface from the monitoring log to be collected that is stopped to be sent. For example, the obtaining position of the monitoring log obtained when the HTTP interface is abnormal is recorded as a position P2; when detecting that the HTTP interface can be called, the monitoring log may be obtained from the position P2, and accordingly, the obtained monitoring log to be collected may be sent to the monitoring data storage device for storage by calling the HTTP interface.
Further, when the log data collector transmits the monitoring logs to be collected in the order from the monitoring log to be collected whose transmission has been stopped, if the number of the monitoring logs to be collected is plural, the monitoring logs to be collected from the monitoring log to be collected whose transmission has been stopped may be transmitted one by one, or the monitoring logs to be collected from the monitoring log to be collected whose transmission has been stopped may be transmitted in a batch.
And S204, sending the monitoring log to be collected to a monitoring data storage device for storage by using the called HTTP interface.
When the log data collector detects that the call to the HTTP interface is abnormal, the log data collector indicates that the HTTP interface can normally transmit the monitoring log to be collected, so that the called HTTP interface can be used for transmitting the monitoring log to be collected to the monitoring data storage device for storage.
In addition, the monitoring data storage device of this embodiment is the same as the monitoring data storage device of the embodiment of fig. 1 of the present invention, and details are not repeated here, and refer to the description of the embodiment of fig. 1 of the present invention.
In the embodiment of fig. 2, by detecting whether the call of the HTTP interface is abnormal, when the call is detected to be abnormal, the sending of the monitoring log may be stopped, and when the HTTP interface is detected to be capable of being called, the hypertext transfer protocol HTTP interface is called to send the acquired monitoring log to be collected to the monitoring data storage device for storage according to the sequence from the monitoring log to be collected that is stopped to be sent. Compared with the method for directly sending the monitoring log updated next time, the method can ensure the continuous sending of the monitoring log with abnormal sending by using the recorded obtaining position of the monitoring log, and reduce the monitoring log loss caused by abnormal sending of the monitoring log due to abnormal calling of the HTTP interface.
For convenience of understanding, an application scenario in which a plurality of central nodes exist in a distributed system is taken as an example below, and the monitoring data collection method provided in the embodiment of fig. 2 of the present invention is exemplarily described. Illustratively, as shown in fig. 3, there are two central nodes in the distributed system, and a log data collector is respectively arranged for each central node: a us central node 3012 and corresponding log data collector 3011 located in the united states, and an asian central node 3022 and corresponding log data collector 3021 located in asia. The american central node 3012 may transmit the monitoring data of the client 3013 of the distributed system using HTTP, and may transmit the monitoring data of the edge server 3014 of the distributed system using RPC (Remote Procedure Call, a computer communication protocol), and then record and store the transmitted monitoring data as a monitoring log. When detecting that the monitoring log stored in the us center node 3012 has an update, the log data collector 3011 obtains the updated monitoring log as the monitoring log to be collected. The log data collector 3011 calls the HTTP interface, and when detecting that the HTTP interface can be called, sends the monitoring log to be collected to the monitoring data storage device by using the HTTP interface for storage, so as to collect monitoring data of the distributed system in the united states. The client of the distributed system may be an electronic device such as a mobile terminal, a portable computer, and the like. Similarly, asian central node 3022, corresponding log data collector 3021, client 3023, and edge server 3024 may perform the same steps for collecting asian monitoring data for a distributed system.
Corresponding to the above method embodiment, an embodiment of the present invention further provides a monitoring data collection device.
As shown in fig. 4, the monitoring data collecting apparatus according to an embodiment of the present invention is applied to a log data collector, and the apparatus may include:
a monitoring log obtaining module 401, configured to obtain, when it is detected that a monitoring log of a distributed system is updated, a monitoring log updated by a central node of the distributed system, where the monitoring log is used as a monitoring log to be collected; the monitoring log updated by the central node is a log obtained by recording the monitoring data of the distributed system by the central node;
a monitoring log sending module 402, configured to send the monitoring log to be collected to a monitoring data storage device for storage.
In the scheme provided by the embodiment of the invention, the central node of the distributed system records the monitoring data of the distributed system as a monitoring log; and the log data collector can obtain the monitoring log updated by the central node as the monitoring log to be collected when detecting that the monitoring log of the distributed system is updated, and send the monitoring log to be collected to the monitoring data storage device for storage, thereby realizing the collection of the monitoring data of the distributed system. Compared with the monitoring data collection by directly calling the publishing function of the distributed publishing and subscribing message tool by the central node, when the monitoring data is collected, the central node records and stores the monitoring log to be collected as the monitoring log, and the log data collector acquires and sends the monitoring log to be collected so as to collect the monitoring data. Therefore, the functions of the central node and the monitoring data collection can be decoupled, and the problem of performance reduction of the central node caused by abnormal monitoring data collection is solved.
Optionally, the monitoring log sending module 402 is specifically configured to:
calling a hypertext transfer protocol (HTTP) interface, and detecting whether the calling of the HTTP interface is abnormal;
if the monitoring logs to be collected are abnormal, stopping sending the monitoring logs to be collected, calling a hypertext transfer protocol HTTP interface to send the acquired monitoring logs to be collected to a monitoring data storage device for storage according to the sequence from the monitoring logs to be collected which are stopped to be sent when the HTTP interface is detected to be capable of being called, and otherwise, sending the monitoring logs to be collected to the monitoring data storage device for storage by using the called HTTP interface.
Optionally, the monitoring log sending module 402 is specifically configured to:
before sending the monitoring logs to be collected to a monitoring data storage device for storage, judging whether the number of the acquired monitoring logs to be collected is a specified number; if the number is not the designated number, triggering the monitoring log obtaining module 401 to execute the monitoring log obtaining process when the monitoring log of the distributed system is detected to be updated, obtaining the updated monitoring log as the monitoring log to be collected, otherwise, sending the obtained monitoring logs to be collected to the monitoring data storage device for storage.
Optionally, the log data collector includes:
a log data collector having a plug-in architecture.
Optionally, the monitoring log includes:
and the central node of the distributed system records and stores a log obtained by monitoring data of the distributed system according to a preset network script language object numbered notation JSON format.
Corresponding to the above embodiment, an embodiment of the present invention further provides an electronic device, as shown in fig. 5, where the electronic device may include:
the system comprises a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory complete mutual communication through the communication bus 504 through the 503;
a memory 503 for storing a computer program;
the processor 501 is configured to implement the steps of any monitoring data collection method applied to the log data collector in the above embodiments when executing the computer program stored in the memory 503.
It is understood that in a specific application, the electronic device in this embodiment is a log data collector.
In the scheme provided by the embodiment of the invention, the central node of the distributed system records the monitoring data of the distributed system as a monitoring log; and the log data collector can obtain the monitoring log updated by the central node as the monitoring log to be collected when detecting that the monitoring log of the distributed system is updated, and send the monitoring log to be collected to the monitoring data storage device for storage, thereby realizing the collection of the monitoring data of the distributed system. Compared with the monitoring data collection by directly calling the publishing function of the distributed publishing and subscribing message tool by the central node, when the monitoring data is collected, the central node records and stores the monitoring log to be collected as the monitoring log, and the log data collector acquires and sends the monitoring log to be collected so as to collect the monitoring data. Therefore, the functions of the central node and the monitoring data collection can be decoupled, and the problem of performance reduction of the central node caused by abnormal monitoring data collection is solved.
The Memory may include a RAM (Random Access Memory) or an NVM (Non-Volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.
An embodiment of the present invention provides a computer-readable storage medium, included in an electronic device, having a computer program stored therein, where the computer program, when executed by a processor, implements any of the steps of the monitoring data collection method applied to a log data collector in the foregoing embodiments.
In the scheme provided by the embodiment of the invention, the central node of the distributed system records the monitoring data of the distributed system as a monitoring log; and the log data collector can obtain the monitoring log updated by the central node as the monitoring log to be collected when detecting that the monitoring log of the distributed system is updated, and send the monitoring log to be collected to the monitoring data storage device for storage, thereby realizing the collection of the monitoring data of the distributed system. Compared with the monitoring data collection by directly calling the publishing function of the distributed publishing and subscribing message tool by the central node, when the monitoring data is collected, the central node records and stores the monitoring log to be collected as the monitoring log, and the log data collector acquires and sends the monitoring log to be collected so as to collect the monitoring data. Therefore, the functions of the central node and the monitoring data collection can be decoupled, and the problem of performance reduction of the central node caused by abnormal monitoring data collection is solved.
In yet another embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the monitoring data collection method applied to a log data collector as described in any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, from a website, computer, server, or data center, over a wired (e.g., coaxial cable, fiber optic, DSL (Digital Subscriber Line), or wireless (e.g., infrared, radio, microwave, etc.) network, to another website, computer, server, or data center, to any available medium that is accessible by a computer or that is a data storage device including one or more integrated servers, data centers, etc. the available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD (Digital Versatile Disc, digital versatile disc)), or a semiconductor medium (e.g.: SSD (Solid state disk)), etc.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the device and electronic apparatus embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for relevant points.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A monitoring data collection method applied to a log data collector, the method comprising:
when detecting that the monitoring log of the distributed system is updated, acquiring the monitoring log updated by the central node of the distributed system as the monitoring log to be collected; the monitoring log updated by the central node is a log obtained by recording the monitoring data of the distributed system by the central node;
and sending the monitoring log to be collected to a monitoring data storage device for storage.
2. The method of claim 1, wherein sending the monitoring log to be collected to a monitoring data storage device for storage comprises:
calling a hypertext transfer protocol (HTTP) interface, and detecting whether the calling of the HTTP interface is abnormal;
if the monitoring logs to be collected are abnormal, stopping sending the monitoring logs to be collected, calling a hypertext transfer protocol HTTP interface to send the acquired monitoring logs to be collected to a monitoring data storage device for storage according to the sequence from the monitoring logs to be collected which are stopped to be sent when the HTTP interface is detected to be capable of being called, and otherwise, sending the monitoring logs to be collected to the monitoring data storage device for storage by using the called HTTP interface.
3. The method of claim 1, wherein before sending the monitoring log to be collected to a monitoring data storage device for storage, the method further comprises:
judging whether the number of the acquired monitoring logs to be collected is a specified number;
if the number of the monitoring logs is not the specified number, returning to execute the step of obtaining the updated monitoring logs as the monitoring logs to be collected when the monitoring logs of the distributed system are detected to be updated, otherwise, sending the monitoring logs to be collected to a monitoring data storage device for storage, wherein the step comprises the following steps:
and sending the acquired monitoring logs to be collected to the monitoring data storage device for storage.
4. The method of any of claims 1 to 3, wherein the log data collector comprises:
a log data collector having a plug-in architecture.
5. The method of any of claims 1 to 3, wherein monitoring the log comprises:
and the central node of the distributed system records and stores a log obtained by monitoring data of the distributed system according to a preset network script language object numbered notation JSON format.
6. A monitoring data collection apparatus, applied to a log data collector, the apparatus comprising:
the monitoring log acquiring module is used for acquiring the monitoring log updated by the central node of the distributed system when the monitoring log of the distributed system is detected to be updated, and the monitoring log is used as the monitoring log to be collected; the monitoring log updated by the central node is a log obtained by recording the monitoring data of the distributed system by the central node;
and the monitoring log sending module is used for sending the monitoring logs to be collected to a monitoring data storage device for storage.
7. The apparatus according to claim 6, wherein the monitoring log sending module is specifically configured to:
calling a hypertext transfer protocol (HTTP) interface, and detecting whether the calling of the HTTP interface is abnormal;
if the monitoring logs to be collected are abnormal, stopping sending the monitoring logs to be collected, calling a hypertext transfer protocol HTTP interface to send the acquired monitoring logs to be collected to a monitoring data storage device for storage according to the sequence from the monitoring logs to be collected which are stopped to be sent when the HTTP interface is detected to be capable of being called, and otherwise, sending the monitoring logs to be collected to the monitoring data storage device for storage by using the called HTTP interface.
8. The apparatus according to claim 6, wherein the monitoring log sending module is specifically configured to:
before sending the monitoring logs to be collected to a monitoring data storage device for storage, judging whether the number of the acquired monitoring logs to be collected is a specified number; and if the number of the monitoring logs is not the specified number, triggering the monitoring log obtaining module to obtain the updated monitoring logs as the monitoring logs to be collected when the monitoring logs of the distributed system are detected to be updated, otherwise, sending the obtained monitoring logs to be collected to the monitoring data storage device for storage.
9. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the bus; the memory is used for storing a computer program; the processor, configured to execute the program stored in the memory, to implement the method steps according to any one of claims 1-5.
10. A computer-readable storage medium, characterized in that a computer program is stored in the storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910878194.8A CN110716841A (en) | 2019-09-17 | 2019-09-17 | Monitoring data collection method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910878194.8A CN110716841A (en) | 2019-09-17 | 2019-09-17 | Monitoring data collection method, device and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110716841A true CN110716841A (en) | 2020-01-21 |
Family
ID=69210545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910878194.8A Pending CN110716841A (en) | 2019-09-17 | 2019-09-17 | Monitoring data collection method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110716841A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113364651A (en) * | 2021-06-28 | 2021-09-07 | 浪潮云信息技术股份公司 | Novel distributed network flow acquisition method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130019180A1 (en) * | 2011-07-11 | 2013-01-17 | International Business Machines Corporation | Log collector in a distributed computing system |
| CN104486107A (en) * | 2014-12-05 | 2015-04-01 | 曙光信息产业(北京)有限公司 | Log collection device and method |
| CN105306261A (en) * | 2015-09-29 | 2016-02-03 | 北京奇艺世纪科技有限公司 | Method, device and system for collecting logs |
| CN107291928A (en) * | 2017-06-29 | 2017-10-24 | 国信优易数据有限公司 | A kind of daily record storage system and method |
| CN109032910A (en) * | 2018-07-24 | 2018-12-18 | 北京百度网讯科技有限公司 | Log collection method, device and storage medium |
-
2019
- 2019-09-17 CN CN201910878194.8A patent/CN110716841A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130019180A1 (en) * | 2011-07-11 | 2013-01-17 | International Business Machines Corporation | Log collector in a distributed computing system |
| CN104486107A (en) * | 2014-12-05 | 2015-04-01 | 曙光信息产业(北京)有限公司 | Log collection device and method |
| CN105306261A (en) * | 2015-09-29 | 2016-02-03 | 北京奇艺世纪科技有限公司 | Method, device and system for collecting logs |
| CN107291928A (en) * | 2017-06-29 | 2017-10-24 | 国信优易数据有限公司 | A kind of daily record storage system and method |
| CN109032910A (en) * | 2018-07-24 | 2018-12-18 | 北京百度网讯科技有限公司 | Log collection method, device and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113364651A (en) * | 2021-06-28 | 2021-09-07 | 浪潮云信息技术股份公司 | Novel distributed network flow acquisition method |
| CN113364651B (en) * | 2021-06-28 | 2023-05-30 | 浪潮云信息技术股份公司 | Distributed network flow collection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10693900B2 (en) | Anomaly detection based on information technology environment topology | |
| US10771479B2 (en) | Configuring modular alert actions and reporting action performance information | |
| US11038905B2 (en) | Identifying attack behavior based on scripting language activity | |
| US20200028760A1 (en) | Automated service-oriented performance management | |
| US11477298B2 (en) | Offline client replay and sync | |
| CN108829560A (en) | Data monitoring method, device, computer equipment and storage medium | |
| US10169434B1 (en) | Tokenized HTTP event collector | |
| US11829381B2 (en) | Data source metric visualizations | |
| US20130081001A1 (en) | Immediate delay tracker tool | |
| US8694831B2 (en) | Automatic bug reporting tool | |
| CN112199263A (en) | Method, device, equipment and medium for recording page | |
| US20230214229A1 (en) | Multi-tenant java agent instrumentation system | |
| CN109039787A (en) | log processing method, device and big data cluster | |
| US8799460B2 (en) | Method and system of providing a summary of web application performance monitoring | |
| US10432490B2 (en) | Monitoring single content page application transitions | |
| CN114371888A (en) | Method and device for hot updating of log collection plug-in, electronic equipment and readable medium | |
| CN112565277B (en) | Data transmission method, system and electronic equipment | |
| CN110716841A (en) | Monitoring data collection method, device and equipment | |
| CN112527519A (en) | High-performance local cache method, system, equipment and medium | |
| CN111245880B (en) | Behavior trajectory reconstruction-based user experience monitoring method and device | |
| US20230118838A1 (en) | Advanced agent instrumentation for opentelemetry implementations | |
| CN112416698B (en) | Expansion method and device of monitoring system, storage medium and electronic equipment | |
| CN116955861A (en) | Page generation method and device and front-end fusion system | |
| US20220075492A1 (en) | Detecting paste and other types of user activities in computer environment | |
| CN109684158B (en) | State monitoring method, device, equipment and storage medium of distributed coordination system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210527 Address after: 25, 5th floor, shuangjingfang office building, 3 frisha street, Singapore Applicant after: Zhuomi Private Ltd. Address before: Room 1101, Santai Commercial Building, 139 Connaught Road, Hong Kong, China Applicant before: HONG KONG LIVE.ME Corp.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200121 |