CN116126636A

CN116126636A - Log data acquisition method, device, equipment and storage medium

Info

Publication number: CN116126636A
Application number: CN202310020918.1A
Authority: CN
Inventors: 胡振泉; 陈松; 吴诗伟; 刘珊; 张文钰
Original assignee: Beijing Qihoo Technology Co Ltd
Current assignee: Beijing Qihoo Technology Co Ltd
Priority date: 2023-01-06
Filing date: 2023-01-06
Publication date: 2023-05-16

Abstract

The invention belongs to the technical field of data acquisition, and discloses a log data acquisition method, device, equipment and storage medium. The method comprises the following steps: receiving log data of data acquisition nodes distributed on each target client; screening the log data to obtain display log data; and storing the display log data and displaying the display log data through a monitoring panel. By the method, the collection nodes of the logs are distributed on each client to collect the log data, and the collected log data is transmitted to the server, so that the log data from different clients can be received from one host or the server, and are displayed through the monitoring panel after screening, so that the data of multiple ends can be collected more simply and conveniently, large-area indexes are not needed, and the efficiency and cost of log collection and monitoring are improved.

Description

Log data acquisition method, device, equipment and storage medium

Technical Field

The present invention relates to the field of data acquisition technologies, and in particular, to a method, an apparatus, a device, and a storage medium for acquiring log data.

Background

With the rapid development of domestic internet software technology, the code of each software project becomes more and more huge, the complexity is also higher and higher, the normal operation of maintenance projects becomes important, and the maintenance work is also more complicated and difficult as the projects become more and more huge. Normally, in order to facilitate the development and debugging of software, a log record code of a key service is added; in the running process of the software, the software records the program processing conditions of the key business in the log file, and if the system running abnormality of the program occurs, the corresponding problem can be located by searching and analyzing the log information. The maintenance system is deployed on different hosts through a traditional log method, and needs to log on different machines to check log files at specified positions. The number of hosts required to monitor login is large, and the account passwords are complex. A large set of items, with potentially hundreds or thousands of deployed nodes, often makes log-on impractical.

The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.

Disclosure of Invention

The invention mainly aims to provide a log data acquisition method, device, equipment and storage medium, and aims to solve the technical problem that a plurality of nodes are troublesome to operate due to the fact that a log monitoring system in the prior art is deployed on different hosts.

In order to achieve the above object, the present invention provides a log data acquisition method, which includes the following steps:

receiving log data of data acquisition nodes distributed on each target client;

screening the log data to obtain display log data;

and storing the display log data and displaying the display log data through a monitoring panel.

Optionally, before receiving the log data of the data collection nodes distributed to each client, the method further includes:

acquiring client information;

determining identity authentication information according to the client information;

determining a target client according to the identity authentication information;

and taking the acquisition node pre-configured in the target client as a data acquisition node.

Optionally, the receiving log data of the data collection nodes distributed to each target client includes:

When a socket connection request from the data acquisition nodes is monitored, socket description information is sent to each data acquisition node according to the socket connection request;

when description confirmation information fed back by each data acquisition node based on the socket description information is received, a log receiving connection thread is established with each data acquisition node;

and receiving log data transmitted by each data acquisition node through the log receiving connection thread.

Optionally, before the log data transmitted by each data collection node is received by the log receiving connection thread, the method further includes:

and sending a polling update instruction to each data acquisition node so that each data acquisition node queries update file information and carries out log acquisition according to the update file information.

Optionally, before the log collection according to the updated file information, the method further includes:

and sending a junk data authority control instruction to each data acquisition node so as to enable each data acquisition node to increase an identity verification process and enable each data acquisition node to intercept data which does not pass through the identity verification process.

Optionally, after receiving the log data of the data collection nodes distributed to each target client, the method further includes:

Acquiring data source information of the log data;

determining data source identity information according to the data source information;

and determining an unverified data source according to the data source identity information, and eliminating the unverified data source in the log data.

Optionally, the filtering the log data to obtain the display log data includes:

acquiring log information and determining whether the log data reach a filtering condition according to the log information;

when the log data reach the filtering condition, acquiring data acquisition settings of all data acquisition nodes;

determining collected information of each data collection node according to the data collection setting and the log data;

and carrying out data filtering on the acquired information according to a preset filtering rule to obtain display log data, wherein the preset filtering rule is used for filtering the acquired information of each data acquisition node until only the latest updated data is reserved.

determining event queue information waiting to be stored according to the log data;

determining event quantity information according to the event queue information;

Determining a target node of which the data acquisition strategy needs to be adjusted according to the event quantity information;

and adjusting the data acquisition strategy of the target node to update the log data and display the log data through the monitoring panel.

Optionally, the adjusting the data collection policy of the target node to update the log data and display the log data through the monitoring panel includes:

sending a continuous acquisition instruction to the target node so that the target node continuously monitors the log and feeds back updated content information of the log;

updating the log data according to the log updated content information to obtain updated log data;

and cleaning and storing the data of the update log data and displaying the data through the monitoring panel.

Optionally, after the displaying log data is stored and displayed through the monitoring panel, the method further includes:

when a problem inquiry instruction is received, determining an inquiry keyword according to the problem inquiry instruction;

carrying out document searching on the log data according to the query keywords so as to determine a target problem document;

and acquiring the abnormal log information corresponding to the target problem document, and displaying the abnormal log information through the monitoring panel.

when a problem locking instruction is received, inquiring container storage information of a data slice container;

determining a data acquisition node without data nodes according to the container storage information;

determining a problem node according to the non-data node;

and acquiring node address information corresponding to the problem node, and displaying the node address information through the monitoring panel.

Optionally, when the problem locking instruction is received, before querying the container storage information of the data slice container, the method further includes:

creating an initial slice container;

and setting an expiration time period in the initial slice container to obtain a data slice container, wherein the data slice container stores the log data sent by each data acquisition node in the expiration time period and does not store the log data when receiving the stored log data from the same data acquisition node.

Optionally, after the initial slicing container sets the expiration time period to obtain the data slicing container, the method further includes:

determining an expiration time according to the expiration time period;

When the expiration time is reached, the data slice container is emptied for the next round of log collection.

In addition, in order to achieve the above object, the present invention also provides a log data acquisition apparatus, including:

the data receiving module is used for receiving log data of the data acquisition nodes distributed on each target client;

the data screening module is used for screening the log data to obtain display log data;

and the data display module is used for displaying the display log data through the monitoring panel after storing the display log data.

Optionally, the data receiving module is further configured to send socket description information to each data acquisition node according to the socket connection request when the socket connection request from the data acquisition node is monitored; when description confirmation information fed back by each data acquisition node based on the socket description information is received, a log receiving connection thread is established with each data acquisition node; and receiving log data transmitted by each data acquisition node through the log receiving connection thread.

Optionally, the data display module is further configured to query container storage information of a data slice container when a problem locking instruction is received; determining a data acquisition node without data nodes according to the container storage information; determining a problem node according to the non-data node; and acquiring node address information corresponding to the problem node, and displaying the node address information through the monitoring panel.

Optionally, the data presentation module is further configured to create an initial slice container; and setting an expiration time period in the initial slice container to obtain a data slice container, wherein the data slice container stores the log data sent by each data acquisition node in the expiration time period and does not store the log data when receiving the stored log data from the same data acquisition node.

Optionally, the data display module is further configured to determine an expiration time according to the expiration time period; when the expiration time is reached, the data slice container is emptied for the next round of log collection.

In addition, to achieve the above object, the present invention also proposes a log data collection apparatus comprising: the system comprises a memory, a processor and a log data acquisition program stored on the memory and executable on the processor, wherein the log data acquisition program is configured to implement the steps of the log data acquisition method as described above.

In addition, in order to achieve the above object, the present invention also proposes a storage medium having stored thereon a log data collection program which, when executed by a processor, implements the steps of the log data collection method as described above.

The method comprises the steps of receiving log data of data acquisition nodes distributed on each target client; screening the log data to obtain display log data; and storing the display log data and displaying the display log data through a monitoring panel. By the method, the collection nodes of the logs are distributed on each client to collect the log data, and the collected log data is transmitted to the server, so that the log data from different clients can be received from a host or the server, and are displayed through the monitoring panel after being screened, so that the data of multiple ends can be collected more simply and conveniently, large-area indexes are not needed, and the efficiency and cost of log collection and monitoring are improved.

Drawings

FIG. 1 is a schematic diagram of a log data collection device of a hardware operating environment according to an embodiment of the present invention;

FIG. 2 is a flowchart of a log data collection method according to a first embodiment of the present invention;

FIG. 3 is a schematic diagram of a system flow in an embodiment of a log data collection method according to the present invention;

FIG. 4 is a flowchart of a log data collection method according to a second embodiment of the present invention;

FIG. 5 is a flowchart of a log data collection method according to a third embodiment of the present invention;

Fig. 6 is a block diagram of a log data acquisition apparatus according to a first embodiment of the present invention.

The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

Referring to fig. 1, fig. 1 is a schematic structural diagram of a log data collection device of a hardware running environment according to an embodiment of the present invention.

As shown in fig. 1, the log data collection device may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) Memory or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.

Those skilled in the art will appreciate that the structure shown in fig. 1 is not limiting of the log data acquisition apparatus and may include more or fewer components than shown, or may combine certain components, or may be a different arrangement of components.

As shown in fig. 1, an operating system, a network communication module, a user interface module, and a log data collection program may be included in the memory 1005 as one type of storage medium.

In the log data collection device shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the log data acquisition device of the present invention may be disposed in the log data acquisition device, where the log data acquisition device invokes a log data acquisition program stored in the memory 1005 through the processor 1001, and executes the log data acquisition method provided by the embodiment of the present invention.

The embodiment of the invention provides a log data acquisition method, referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of the log data acquisition method.

In this embodiment, the log data acquisition method includes the following steps:

Step S10: and receiving log data of data acquisition nodes distributed on each target client.

It should be noted that, the execution body of the embodiment is one server, which may be an entity server or a cloud server, and the embodiment is not limited to this. And the servers of the execution body together with the respective target clients constitute a C-S service architecture client/server architecture.

It should be appreciated that journals have evolved greatly from initially human oriented to now machine oriented. The primary consumer of the original journal is a software engineer who troubleshoots the problem by reading the journal and today, a large number of machines process the journal data day and night to generate a report of readability to help humans make decisions. During this transition, the logging Agent plays an important role therein. Log collection is the cornerstone of big data. Many business platforms of companies generate large amounts of log data each day. The collection of business log data for use by both offline and online analysis systems is what the log collection system is to do. The Agent for collecting the logs is simply a program for delivering data from a source end to a destination end, and usually the destination end is a centralized storage with a data subscription function, so that the purpose of the method is to decouple log analysis and log storage, the same log may be interested by different consumers, the processing mode after the log is obtained is different, and different consumers can subscribe the log of interest by themselves and select corresponding analysis tools for analysis after decoupling the data storage and the data analysis. The data source end can be divided into three types, one type is a common text file, the other type is log data received through a network, and the last type is in a mode of sharing a memory. This is generally the case for a log collection Agent to be the most central function. Further, the functions of log filtering, log formatting, routing and the like can be introduced on the basis, and the system looks like a production shop. From the log delivery mode, log collection can be further divided into a push mode and a pull mode. The push mode refers to that the log collection Agent actively obtains data from the source end and then sends the data to the destination end, and the pull mode refers to that the destination end actively obtains the data of the source end from the log collection Agent. Currently, there are common open source log collection systems such as Flume, scribe, etc. The Flume is a highly available, highly reliable, distributed system for collecting, aggregating and transmitting massive logs, which is currently a sub-project of Apache. The script is a Facebook open-source log collection system, and provides a high-availability, high-reliability and high-expandability log collection system for distributed collection and unified processing of logs. The existing schemes for acquiring a plurality of logs adopt full-text retrieval to index the logs (such as an ELK scheme), and have the advantages of rich functions and complex operation permission. However, these schemes tend to be complex in scale, high in resource occupation, and difficult to operate. Many functions are often not available, most queries focus only on a certain time frame and on some simple parameters (e.g. host, service, etc.), and using these solutions there is a sense of a point chicken-killing knife. In this embodiment, as shown in fig. 3, the system architecture diagram of the present invention is shown, and the distributed nodes collect data and then uniformly send the data to the server, so as to clean, deliver and store the data and finally display the data, distribute the collected nodes of the log on each client to collect the log data, and transmit the collected log data to the server, so that log data from different clients can be received from a host or server, and screened and displayed through the monitoring panel, so that the data of multiple ends can be collected more simply and conveniently, a large-area index is not needed, and the efficiency and cost of log collection and monitoring are improved.

In a specific implementation, the target client refers to each client that has established a connection with the server, and may be a client distributed on a different host. The data acquisition nodes are Agent nodes which are respectively and correspondingly arranged on all target clients, and each target client is provided with one data acquisition node. The log data refers to information and data related to logs collected by each data collection node on each target client.

Further, in order to pre-establish the connection between the server and the client and perform the authentication, before step S10, the method further includes: acquiring client information; determining identity authentication information according to the client information; determining a target client according to the identity authentication information; and taking the acquisition node pre-configured in the target client as a data acquisition node.

It should be noted that the client information refers to information such as an address, an identity, etc. of a client that the server receives to request to establish a connection.

It should be understood that the authentication information refers to authentication related information of each client in the client information, and may include related information such as an IP address. And then determining the client passing the identity verification as a target client according to the identity authentication information.

In a specific implementation, after the target client is determined, log data is collected by taking a collection node which is preset on the target client, namely an Agent node, as a data collection node.

In this way, it is achieved that the data collection nodes are reconfigured after the identity of each client has been confirmed, so that the security of the connection is ensured.

Further, in order to perform the secondary authentication after receiving the log data, after step S10, the method further includes: acquiring data source information of the log data; determining data source identity information according to the data source information; and determining an unverified data source according to the data source identity information, and eliminating the unverified data source in the log data.

It should be noted that, the data source information refers to a data collection node that determines a source of sending each log data by backtracking the log data.

It should be understood that determining the data source identity information from the data source information refers to: according to the data source information, firstly determining the data source of each log data, and then determining the identity information of each data source when transmitting data as the identity information of the data source.

In a specific implementation, after the identity information of the data source is obtained, the identity information of the data source is compared with a white list of the identity information maintained by a background storage, and the data source without the white list is the unverified data source, and at the moment, all log data acquired by the unverified data source are removed.

In this way, only the type of setting (category) can be entered into the storage system, so the current rights control is category filtering. The part of the authority control at the Agent end can better control the circulation of the garbage data in the system. The authentication portion in the log configuration file may be added at a time and then the Agent's configuration may be reloaded. The authority control part at the Server end can disconnect the data source without identity verification from the data source which cannot be determined. Unregistered data without authentication cannot be transferred between agents/servers.

Further, in order to prevent the event list from being excessively displayed after receiving the large amount of data, the method further includes, after step S10: determining event queue information waiting to be stored according to the log data; determining event quantity information according to the event queue information; determining a target node of which the data acquisition strategy needs to be adjusted according to the event quantity information; and adjusting the data acquisition strategy of the target node to update the log data and display the log data through the monitoring panel.

In a specific implementation, a distributed Agent node monitors a file through an Inotify, so that an event is triggered as long as the file has newly added data, and the acquisition can be continued after the event is obtained. However, this causes an overflow of the event queue in a scenario where a large number of files are written, for example, a user continuously writes the log N times, N events are generated, which is only needed to update the log collection Agent as long as the user knows the content, and it is not important to update several times, because each collection is actually to continuously read the file until EOF, and the collection is continued as long as the user continuously writes the log. In addition, the number of files that Inotify can monitor is also limited.

The event queue information refers to a queue of log events waiting to be written into the storage, which is determined according to log data, wherein each log data is counted as one log event. And determining the number of the events, namely the number of the events waiting to be stored, according to the event queue information.

It should be understood that determining the target node to adjust the data acquisition strategy according to the event number information refers to: and determining the data quantity uploaded by each data acquisition node according to the time quantity information, and taking the data acquisition node with the uploaded data quantity exceeding the compaction threshold value as a target node. The reduced threshold is a preset positive integer threshold, and is used for limiting the number of events of the log data.

In a specific implementation, after the target node is determined, the data collection strategy of the target node is adjusted to reduce and update log data, so that the number of event queues is reduced, and the data is more simplified when the presentation is performed. The data acquisition strategy refers to a storage and display strategy of each data acquisition node when acquiring logs.

In this way, a reduction in the number of event queues is achieved and the data is more compact when exposed.

Further, for further updating and displaying the log data, the step of adjusting the data collection policy of the target node to update the log data and display the log data through the monitoring panel includes: sending a continuous acquisition instruction to the target node so that the target node continuously monitors the log and feeds back updated content information of the log; updating the log data according to the log updated content information to obtain updated log data; and cleaning and storing the data of the update log data and displaying the data through the monitoring panel.

It should be noted that, sending a continuous acquisition instruction to the target node so that the target node continuously monitors the log and feeds back updated log content information refers to: and sending a continuous acquisition instruction to the target node, so that the target node continuously monitors the log data after receiving the continuous acquisition instruction, and only feeds back the updated content information of the log during feedback, and does not feed back excessive information such as the updated time, the content and the like of each update, namely only feeds back the final updated content of the log, and does not feed back the updated content of each midway.

It should be understood that updating the log data according to the log update content information, and obtaining updated log data refers to: after receiving the log updated content information, the log data is updated, so that updated log data can be obtained. And finally, processing the update log data in the steps of removing errors and checking and repeating the data cleaning, storing the update log data into a system, and finally displaying the update log data through a monitoring panel.

By the method, when a large amount of log data is received, the log data is subjected to simplifying processing and displaying, the system pressure is reduced, and the information displaying is simplified.

Step S20: and screening the log data to obtain the display log data.

In specific implementation, filtering refers to filtering and screening the log data, so that the obtained display data are screened, and the update condition of the log data can be displayed more intuitively when the display is performed.

Further, in order to filter the log data according to the requirement, step S20 includes: acquiring log information and determining whether the log data reach a filtering condition according to the log information; when the log data reach the filtering condition, acquiring data acquisition settings of all data acquisition nodes; determining collected information of each data collection node according to the data collection setting and the log data; and carrying out data filtering on the acquired information according to a preset filtering rule to obtain display log data, wherein the preset filtering rule is used for filtering the acquired information of each data acquisition node until only the latest updated data is reserved.

It should be noted that the log message information refers to information related to push messages in the received log data, that is, one log message information is updated every time one log data is received. And judging whether the filtering condition is reached or not according to the log message information. The filtering condition refers to that the log message is turned on when the log message reaches a specified number, wherein the specified number is a preset positive integer number, and the embodiment is not limited thereto.

It should be understood that, after the log data reaches the filtering condition, the respective data acquisition settings of each data acquisition node, that is, the rules and principles of data acquisition of each data acquisition node, are acquired.

In a specific implementation, determining the collected information of each data collection node according to the data collection setting and the log data refers to: and determining the acquired information corresponding to each data acquisition node according to the data acquisition setting and the log data, so that the acquired information corresponding to each data acquisition node can be determined.

It should be noted that, the preset filtering rule is: only one piece of data finally collected by each data collection node is reserved and used as the latest updated data, namely, the latest information is simply filtered to the same node, and then the data is filtered.

By the method, the collected log data is filtered and then displayed, and the final state of log updating of each client can be displayed more intuitively.

Step S30: and storing the display log data and displaying the display log data through a monitoring panel.

It should be understood that the message queue is delivered at the server end to store the log, and then the preset display interface is used for filtering the data, so that the monitoring panel is finally displayed.

Further, in order to implement the relevant log corresponding to the query keyword, after step S30, the method further includes: when a problem inquiry instruction is received, determining an inquiry keyword according to the problem inquiry instruction; carrying out document searching on the log data according to the query keywords so as to determine a target problem document; and acquiring the abnormal log information corresponding to the target problem document, and displaying the abnormal log information through the monitoring panel.

In a specific implementation, the question query instruction is a preset instruction for querying a keyword, and after the question query instruction sent by the user is received, the query keyword is determined according to information input by the user in the question query instruction, where the query keyword may be a word, a term, or a specified code. The data store and search engine may be provided as a plug-in function.

After determining the query keyword, the full text search of the document search is performed on all the log data, so that the document containing the query keyword is determined as the target problem document.

It should be understood that after the target problem document is determined, the corresponding abnormal log information in the re-log data of the target problem document is determined, and finally the abnormal log information is displayed through the monitoring panel.

By the method, keyword retrieval based on the requirements of the user is achieved, so that log data to be checked by the user is confirmed to be displayed.

The embodiment receives log data of data acquisition nodes distributed on each target client; screening the log data to obtain display log data; and storing the display log data and displaying the display log data through a monitoring panel. By the method, the collection nodes of the logs are distributed on each client to collect the log data, and the collected log data is transmitted to the server, so that the log data from different clients can be received from a host or the server, and are displayed through the monitoring panel after being screened, so that the data of multiple ends can be collected more simply and conveniently, large-area indexes are not needed, and the efficiency and cost of log collection and monitoring are improved.

Referring to fig. 4, fig. 4 is a flowchart of a second embodiment of a log data collection method according to the present invention.

Based on the above first embodiment, the log data collection method in this embodiment includes, in step S10:

step S101: and when the socket connection request from the data acquisition nodes is monitored, the socket description information is sent to each data acquisition node according to the socket connection request.

The socket connection request refers to a connection request of a socket (socket) transmitted from each data collection node. In this embodiment, the server and the client are transmitted through a socket connection.

It should be understood that the socket description information refers to description information of a socket of a server side transmitted to each data acquisition node. Step S102: and when description confirmation information fed back by each data acquisition node based on the socket description information is received, establishing a log receiving connection thread with each data acquisition node.

It should be understood that, after each data acquisition node receives the socket description information, description acknowledgement information of each client is fed back. And if the socket description information of the server is the description information of the allowed connection which is confirmed in advance, feeding back the description confirmation information, and if the description information is not, feeding back the description confirmation information.

In a specific implementation, after receiving the description confirmation information, log receiving connection threads can be respectively established with each data acquisition node through a socket.

Step S103: and receiving log data transmitted by each data acquisition node through the log receiving connection thread.

It should be noted that, after the log receiving connection thread is established, the connection is established through the socket, so log data may be obtained through the log receiving connection thread.

It should be understood that since the log is always deleted at a moment, if the log is deleted in the process of collection, the file in Linux has a reference count, and the opened file is only reduced by 1 even if the file is deleted, so that the log collection Agent can continue to read the content as long as the process references, and the log collection Agent can release the fd file descriptor of the file after the log is completely read, so that the system can truly delete the file.

Further, in order to enable each data collection node to collect data stably and comprehensively, before step S103, the method further includes: and sending a polling update instruction to each data acquisition node so that each data acquisition node queries update file information and carries out log acquisition according to the update file information.

In a specific implementation, the polling update instruction refers to an instruction for notifying each data acquisition node to perform polling update, and may be any form of instruction, which is not limited in this embodiment.

It should be noted that, after the data collection node receives the polling update instruction, it will query the stat information of the file to be collected, and find that the file content is updated, and trigger the next polling after the collection is completed.

By the method, continuous log collection which can be uninterrupted by the log collection Agent is realized.

Further, in order to control the transmission of the junk data in the system, before the step of collecting the log according to the updated file information, the method further includes: and sending a junk data authority control instruction to each data acquisition node so as to enable each data acquisition node to increase an identity verification process and enable each data acquisition node to intercept data which does not pass through the identity verification process.

It should be appreciated that after each data collection node receives the spam data authority control instruction, an authentication flow may be added, such that each data collection node automatically discards and does not transmit data that does not pass through the authentication flow. The authentication procedure may be any procedure capable of verifying data source and data authentication.

By the method, garbage data transmission in the control system is realized, and system pressure is reduced.

When the socket connection request from the data acquisition nodes is monitored, the method sends socket description information to each data acquisition node according to the socket connection request; when description confirmation information fed back by each 5 data acquisition node based on the socket description information is received, a log receiving connection thread is established with each data acquisition node; and receiving log data transmitted by each data acquisition node through the log receiving connection thread. By the method, communication connection and data interaction between the server and the client based on socket connection are achieved, so that the data interaction is more convenient, and distributed log data monitoring is achieved.

Referring to fig. 5, fig. 5 is a flowchart of a log data collection method according to a third embodiment of the present invention.

Based on the above first embodiment, the log data collection method of this embodiment further includes, after the step S30:

step S301: when a problem lock instruction is received, the container of the data slice container is queried to store information 5.

The problem lock instruction is an instruction for locking the data collection node having an abnormality, and may be any instruction capable of realizing this function, and the present embodiment is not limited thereto.

It should be understood that container storage information refers to information about logs, data that have been stored in the slice container.

0 further, in order to be able to establish a slice container for the problem determination, before step S301

Comprising the following steps: creating an initial slice container; and setting an expiration time period in the initial slice container to obtain a data slice container, wherein the data slice container stores the log data sent by each data acquisition node in the expiration time period and does not store the log data when receiving the stored log data from the same data acquisition node.

5 in an embodiment, the initial slice container is set up as a slice that is globally initialized at the server side

A sheet container.

It should be noted that, the expiration time period is that an expiration time is set in the configuration file, and in this unit time, the data slicing container stores data from each data collecting node, but if the data received by the server end already exists in the data slicing container, the content is discarded, the next data receiving is performed, and finally the data slicing container collects data from different sources, so that a problem can be known about which node does not send data.

In this way, a data acquisition node is established that establishes a data slice container and accurately determines the occurrence of anomalies.

Further, in order to clean up the expiration data in a further period, after the expiration time period is set in the initial slice container to obtain the data slice container, the method further includes: determining an expiration time according to the expiration time period; when the expiration time is reached, the data slice container is emptied for the next round of log collection.

After the expiration time period is determined, the expiration time, that is, the expiration time of each time is determined, and after the expiration time is reached, the information stored in the data slice container is emptied, so that the log collection and fault check of the next round can be performed.

In this way, periodic emptying of the data slice container is achieved, preventing excessive data of the data slice container and errors caused by stacking.

Step S302: and determining a data-free node in each data acquisition node according to the container storage information.

It should be understood that a data-free node is a node that does not store data for an expiration period of time for each data acquisition node determined from the container storage information, and is used as a data-free node.

Step S303: and determining a problem node according to the non-data node.

In a specific implementation, the non-data node is identified as the problem node.

Step S304: and acquiring node address information corresponding to the problem node, and displaying the node address information through the monitoring panel.

The node address information refers to network address information related to the node IP of the problem node, and the node address information is displayed to the user through the monitoring panel.

When a problem locking instruction is received, the embodiment inquires container storage information of a data slice container; determining a data acquisition node without data nodes according to the container storage information; determining a problem node according to the non-data node; and acquiring node address information corresponding to the problem node, and displaying the node address information through the monitoring panel. By the method, the problem node which does not upload data is accurately determined based on the problem locking instruction, so that the system can automatically check abnormality periodically.

In addition, the embodiment of the invention also provides a storage medium, wherein the storage medium is stored with a log data acquisition program, and the log data acquisition program realizes the steps of the log data acquisition method when being executed by a processor.

The storage medium adopts all the technical solutions of all the embodiments, so that the storage medium has at least all the beneficial effects brought by the technical solutions of the embodiments, and is not described in detail herein.

Referring to fig. 6, fig. 6 is a block diagram illustrating a first embodiment of a log data acquisition apparatus according to the present invention.

As shown in fig. 6, the log data collecting device provided by the embodiment of the present invention includes:

the data receiving module 10 is configured to receive log data of data collection nodes distributed to each target client.

And the data screening module 20 is used for screening the log data to obtain the display log data.

The data display module 30 is configured to store the display log data and display the display log data through the monitoring panel.

In an embodiment, the data receiving module 10 is further configured to obtain client information; determining identity authentication information according to the client information; determining a target client according to the identity authentication information; and taking the acquisition node pre-configured in the target client as a data acquisition node.

In an embodiment, the data receiving module 10 is further configured to, when monitoring a socket connection request from the data collection node, send socket description information to each data collection node according to the socket connection request; when description confirmation information fed back by each data acquisition node based on the socket description information is received, a log receiving connection thread is established with each data acquisition node; and receiving log data transmitted by each data acquisition node through the log receiving connection thread.

In an embodiment, the data receiving module 10 is further configured to send a polling update command to each data collection node, so that each data collection node queries update file information, and performs log collection according to the update file information.

In an embodiment, the data receiving module 10 is further configured to send a control instruction of authority of the junk data to each data collecting node, so that each data collecting node adds an authentication process, and each data collecting node intercepts data that does not pass the authentication process.

In an embodiment, the data receiving module 10 is further configured to obtain data source information of the log data; determining data source identity information according to the data source information; and determining an unverified data source according to the data source identity information, and eliminating the unverified data source in the log data.

In an embodiment, the data filtering module 20 is further configured to obtain log message information, and determine whether the log data reaches a filtering condition according to the log message information; when the log data reach the filtering condition, acquiring data acquisition settings of all data acquisition nodes; determining collected information of each data collection node according to the data collection setting and the log data; and carrying out data filtering on the acquired information according to a preset filtering rule to obtain display log data, wherein the preset filtering rule is used for filtering the acquired information of each data acquisition node until only the latest updated data is reserved.

In an embodiment, the data receiving module 10 is further configured to determine event queue information waiting to be stored according to the log data; determining event quantity information according to the event queue information; determining a target node of which the data acquisition strategy needs to be adjusted according to the event quantity information; and adjusting the data acquisition strategy of the target node to update the log data and display the log data through the monitoring panel.

In an embodiment, the data display module 30 is further configured to send a continuous acquisition instruction to the target node, so that the target node continuously monitors the log and feeds back the log update content information; updating the log data according to the log updated content information to obtain updated log data; and cleaning and storing the data of the update log data and displaying the data through the monitoring panel.

In one embodiment, the data display module 30 is further configured to, when a question query instruction is received, determine a query keyword according to the question query instruction; carrying out document searching on the log data according to the query keywords so as to determine a target problem document; and acquiring the abnormal log information corresponding to the target problem document, and displaying the abnormal log information through the monitoring panel.

In one embodiment, the data display module 30 is further configured to query the container storage information of the data slice container when a problem lock instruction is received; determining a data acquisition node without data nodes according to the container storage information; determining a problem node according to the non-data node; and acquiring node address information corresponding to the problem node, and displaying the node address information through the monitoring panel.

In one embodiment, the data presentation module 30 is further configured to create an initial slice container; and setting an expiration time period in the initial slice container to obtain a data slice container, wherein the data slice container stores the log data sent by each data acquisition node in the expiration time period and does not store the log data when receiving the stored log data from the same data acquisition node.

In one embodiment, the data presentation module 30 is further configured to determine an expiration time according to the expiration time period; when the expiration time is reached, the data slice container is emptied for the next round of log collection.

It should be understood that the foregoing is illustrative only and is not limiting, and that in specific applications, those skilled in the art may set the invention as desired, and the invention is not limited thereto.

It should be noted that the above-described working procedure is merely illustrative, and does not limit the scope of the present invention, and in practical application, a person skilled in the art may select part or all of them according to actual needs to achieve the purpose of the embodiment, which is not limited herein.

In addition, technical details not described in detail in the present embodiment may refer to the log data collection method provided in any embodiment of the present invention, which is not described herein.

Furthermore, it should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. Read Only Memory)/RAM, magnetic disk, optical disk) and including several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.

The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

The invention discloses A1, a log data acquisition method, which comprises the following steps:

Receiving log data of data acquisition nodes distributed on each target client;

screening the log data to obtain display log data;

A2, the method of A1, before receiving the log data of the data collection nodes distributed to each client, further comprises:

acquiring client information;

A3, the method of A1, the receiving log data of the data collection nodes distributed to each target client, includes:

The method of A4, the method of A3, before the log receiving connection thread receives log data transmitted by each data collection node, further includes:

A5, the method of A4, before the log collection according to the updated file information, further comprises:

A6, the method of A1, after receiving the log data of the data collection nodes distributed to each target client, further comprises:

acquiring data source information of the log data;

A7, the method of A1, wherein the filtering the log data to obtain the display log data comprises the following steps:

A8, the method of A1, after receiving the log data of the data collection nodes distributed to each target client, further comprises:

A9. the method of A8, wherein the adjusting the data collection policy of the target node to update the log data and display the log data through the monitoring panel includes:

A10, the method of A1, after the display log data is stored and displayed by the monitoring panel, further comprises:

A11, the method of A1, after the displaying log data is stored and displayed by the monitoring panel, further comprises:

Determining a problem node according to the non-data node;

A12, the method of A11, when receiving the problem lock instruction, further comprising, before querying the container storage information of the data slice container:

creating an initial slice container;

A13, the method of a12, after the initial slicing container sets an expiration time period to obtain a data slicing container, further comprising:

determining an expiration time according to the expiration time period;

The invention also discloses a B14 and a log collection monitoring device, wherein the log collection monitoring device comprises:

The log collection monitoring device as described in B15, wherein the data receiving module is further configured to send socket description information to each data collection node according to a socket connection request when the socket connection request from the data collection node is monitored; when description confirmation information fed back by each data acquisition node based on the socket description information is received, a log receiving connection thread is established with each data acquisition node; and receiving log data transmitted by each data acquisition node through the log receiving connection thread.

The log collection monitoring device as described in B16, wherein the data display module is further configured to query container storage information of a data slice container when a problem locking instruction is received; determining a data acquisition node without data nodes according to the container storage information; determining a problem node according to the non-data node; and acquiring node address information corresponding to the problem node, and displaying the node address information through the monitoring panel.

B17, the log collection monitoring device of B15, wherein the data display module is further configured to create an initial slice container; and setting an expiration time period in the initial slice container to obtain a data slice container, wherein the data slice container stores the log data sent by each data acquisition node in the expiration time period and does not store the log data when receiving the stored log data from the same data acquisition node.

B18, the log collection monitoring device of B16, the data display module is further configured to determine an expiration time according to the expiration time period; when the expiration time is reached, the data slice container is emptied for the next round of log collection.

The invention also discloses C19, a log collection monitoring device, the device includes: a memory, a processor, and a log collection monitoring program stored on the memory and executable on the processor, the log collection monitoring program configured to implement the log collection monitoring method as described above.

The invention also discloses D20, a storage medium, the storage medium stores a log collection monitoring program, and the log collection monitoring program realizes the log collection monitoring method when being executed by a processor.

Claims

1. The log data acquisition method is characterized by comprising the following steps of:

receiving log data of data acquisition nodes distributed on each target client;

screening the log data to obtain display log data;

2. The method of claim 1, wherein prior to receiving log data for data collection nodes distributed to each client, further comprising:

acquiring client information;

3. The method of claim 1, wherein the receiving log data of the data collection nodes distributed to each target client comprises:

4. The method of claim 3, wherein prior to receiving log data transmitted by each data collection node via the log receiving connection thread, further comprising:

5. The method of claim 4, wherein prior to the log collection based on the updated file information, further comprising:

6. The method of claim 1, wherein after receiving log data of the data collection nodes distributed to each target client, further comprising:

acquiring data source information of the log data;

7. The method of claim 1, wherein the filtering the log data to obtain presentation log data comprises:

8. A log collection monitoring device, characterized in that the log collection monitoring device comprises:

9. A log collection monitoring device, the device comprising: a memory, a processor, and a log collection monitoring program stored on the memory and executable on the processor, the log collection monitoring program configured to implement the log collection monitoring method of any one of claims 1 to 7.

10. A storage medium having stored thereon a log collection monitoring program which when executed by a processor implements the log collection monitoring method of any one of claims 1 to 7.