CN110781143B - Method and device for querying and extracting server logs - Google Patents
Method and device for querying and extracting server logs Download PDFInfo
- Publication number
- CN110781143B CN110781143B CN201911069704.3A CN201911069704A CN110781143B CN 110781143 B CN110781143 B CN 110781143B CN 201911069704 A CN201911069704 A CN 201911069704A CN 110781143 B CN110781143 B CN 110781143B
- Authority
- CN
- China
- Prior art keywords
- log
- query
- server
- instruction
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000013515 script Methods 0.000 claims abstract description 120
- 238000000605 extraction Methods 0.000 claims abstract description 51
- 238000012986 modification Methods 0.000 claims description 13
- 230000004048 modification Effects 0.000 claims description 13
- 239000000284 extract Substances 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 14
- 238000012544 monitoring process Methods 0.000 description 6
- 238000012423 maintenance Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
- G06F11/3096—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents wherein the means or processing minimize the use of computing system or of computing system component resources, e.g. non-intrusive monitoring which minimizes the probe effect: sniffing, intercepting, indirectly deriving the monitored data from other directly available data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Library & Information Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a method and a device for inquiring and extracting server logs, wherein the method comprises the following steps: receiving log query command information carrying query keyword information and query range information, and converting the log query command information into a log query command write instruction set; starting a timing task to identify and execute a log query instruction in an instruction set, and generating a corresponding query log collection script; inquiring and extracting a target log on the server according to the inquiry keyword information and the inquiry range information; writing the target log into a log receiving folder according to the server identification mark; when the instruction set is executed, generating a finish identifier corresponding to the query log and adding the finish identifier to the shared folder; and acquiring target logs from each log receiving folder according to the query keyword information and the query range information, and merging to obtain a query log set. The invention greatly saves the time and cost required by the extraction of the server log.
Description
Technical Field
The invention relates to the technical field of server log query, in particular to a method and a device for server log query extraction.
Background
As the number of websites, applications and usage increases, many problems and defects (commonly called bugs) of the websites or the applications themselves are exposed, and if the bugs are not solved in time, the traffic loss and evaluation of the applications or the websites are reduced. At present, the problems and the defects are mainly searched and solved by operation and maintenance personnel. It is one of the main tasks of operation and maintenance personnel to help develop positioning and analyzing problems by accurately and quickly searching application logs.
However, with the increase of traffic and the enhancement of functional complexity, most companies now adopt server clustering, load balancing, etc. to share the access pressure of a single server, and logs of the same set of servers may be distributed on several, even dozens of servers. At this time, the operation and maintenance personnel needs to manually pull the logs to obtain the logs, which takes a lot of time and labor costs.
Compared with the manual pulling mode, the mode of collecting the server logs through the script greatly reduces time cost and labor cost. Generally, in log extraction, a script for collecting logs is placed in a certain server, and logs in other servers to collect logs in a remote login manner. The problem with this approach is that: the safety is not enough, and the realization has a bottleneck: remote login requires knowledge of the login password, which is constant and secret in many companies for security reasons. The operation conditions are harsh: the log collection work is centralized on one server, and has higher requirements on the performance of the server. In the existing log analysis technology, the most representative log analysis platform belongs to (elastic search + Logstash + Kibana) ELK log analysis platform, and is a set of log management scheme with open source, and the Logstash is responsible for collecting, processing and storing logs; the Elasticisearch is responsible for log retrieval and analysis; kibana is responsible for visualization of logs. However, this platform has two objective disadvantages: firstly, the method comprises the following steps: logstash consumes large resources and occupies a high CPU and memory for operation; secondly, the method comprises the following steps: the Logstash needs to store the processed log, and then waits for the log to be retrieved and analyzed, and the storage itself occupies a large memory of the server.
Therefore, it is an urgent technical problem to be solved in the art to provide a server log query extraction scheme that consumes less resources and saves the server log extraction time and cost.
Disclosure of Invention
In view of the above, the present invention provides a method and an apparatus for querying and extracting a server log, which solve the technical problems of the prior art that the resource consumption of querying and extracting the server log is large, the server log extraction time is long, and the cost is high.
In order to solve the above technical problem, the present invention provides a method for querying and extracting a server log, including:
setting a shared folder on a server, deploying a log collection script on the shared folder, and creating a log receiving folder in the shared folder according to the server identification mark; adding a timing task of a timing execution log collection script on the server, and deploying an instruction set for the timing task to execute on the server;
receiving log query command information which is input by a client and carries query keyword information and query range information, converting the log query command information into a log query instruction, and writing the log query instruction into the instruction set; enabling the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script;
when the process does not exist in the query log collection script and the shared folder does not have a finish mark corresponding to the query log, querying and extracting a target log on the server according to the query keyword information and the query range information; writing the target log into the log receiving folder according to the server identification mark; when the instruction set is executed, generating a finish identifier corresponding to the query log and adding the finish identifier to the shared folder;
and acquiring the target logs from each log receiving folder according to the query keyword information and the query range information, and combining to obtain a query log set.
Optionally, the obtaining the target log from each log receiving folder according to the query keyword information and the query range information and merging the target logs to obtain a query log set includes:
checking the shared folder of each server according to the identification characteristics of the server identification and the finalization identification;
when all the shared folders have the ending marks, acquiring the target logs from all the log receiving folders according to the query keyword information and the query range information, and merging to obtain a query log set;
and generating a query log acquisition completion message, sending the query log acquisition completion message to the client, and deleting the log query instruction.
Optionally, receiving log query command information which is input by a client and carries query keyword information and query range information, converting the log query command information into a log query instruction, and writing the log query instruction into the instruction set; starting the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script, wherein the query log collection script comprises the following steps:
receiving log query command information which is input by a client and carries query keyword information and query range information, converting the log query command information into a log query instruction, and writing the log query instruction into the instruction set; comparing the instruction set to an application name on the server;
and when the application name on the server contains the query range information of the log query instruction, starting the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script.
Optionally, wherein the method further comprises:
receiving a modification instruction for modifying the timing task;
and analyzing the modification instruction to obtain update time and an update timing execution command, and replacing related contents in the timing task according to the update time and the update timing execution command.
Optionally, when the query log collection script does not have a process and the shared folder does not have a finish identifier corresponding to the query log, querying and extracting a target log on the server according to the query keyword information and the query range information is that:
judging that the query log collection script has a process by utilizing a supprocess library according to a native shell instruction, and/or deleting the log query instruction and the query log collection script when a finishing identifier corresponding to the query log exists in the shared folder;
and when the process does not exist in the query log collection script and the shared folder does not have the finishing identification corresponding to the query log, querying and extracting the target log on the server according to the query keyword information and the query range information.
In another aspect, the present invention further provides a device for query and extraction of server logs, including: the log collection system comprises a log query setter, a log collection script creating processor, a log collection processor and a log merging processor; wherein,
the log query setter is connected with the log collection script creating processor, a shared folder is arranged on a server, a log collection script is deployed on the shared folder, and a log receiving folder is created in the shared folder according to the server identification mark; adding a timing task of a timing execution log collection script on the server, and deploying an instruction set for the timing task to execute on the server;
the log collection script creating processor is connected with the log query setter and the log collection processor, receives log query command information which is input by a client and carries query keyword information and query range information, converts the log query command information into a log query command and writes the log query command into the command set; enabling the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script;
the log collection processor is connected with the log collection script creating processor and the log merging processor, and when the process does not exist in the query log collection script and the shared folder does not have a finish mark corresponding to the query log, the target log on the server is queried and extracted according to the query keyword information and the query range information; writing the target log into the log receiving folder according to the server identification mark; when the instruction set is executed, generating a finish identifier corresponding to the query log and adding the finish identifier to the shared folder;
and the log merging processor is connected with the log acquisition processor, acquires the target logs from each log receiving folder according to the query keyword information and the query range information, and merges to obtain a query log set.
Optionally, wherein the log merge processor includes: a log extraction completion determination unit and an extracted log merging unit; wherein,
the log extraction completion judging unit is connected with the log acquisition processor and the extracted log merging unit and is used for checking the shared folder of each server according to the identification of the server and the identification characteristics of the completion identification;
the extracted log merging unit is connected with the log extraction completion judging unit, and when completion identifiers exist in all the shared folders, the extracted log merging unit acquires the target logs from each log receiving folder according to the query keyword information and the query range information and merges the target logs to obtain a query log set; and generating a query log acquisition completion message, sending the query log acquisition completion message to the client, and deleting the log query instruction.
Optionally, wherein the log collection script creating processor comprises: a query range judging unit and a log collection script creating unit;
the query range judging unit is connected with the log query setter and the log collection script creating unit, receives log query command information which is input by a client and carries query keyword information and query range information, converts the log query command information into a log query instruction and writes the log query instruction into the instruction set; comparing the instruction set to an application name on the server;
and the log collection script creating unit is connected with the query range judging unit and the log collection processor, and when the application name on the server contains query range information of the log query instruction, the timing task is started to identify and execute the log query instruction in the instruction set so as to generate a corresponding query log collection script.
Optionally, wherein the apparatus further comprises: the timing task updating processor is connected with the log query setter and receives a modification instruction for modifying the timing task;
and analyzing the modification instruction to obtain update time and an update timing execution command, and replacing related contents in the timing task according to the update time and the update timing execution command.
Optionally, wherein the log collection processor comprises: a log collection script process judgment unit and a query log collection unit; wherein,
the log collection script process judgment unit is connected with the log collection script creation processor and the query log collection unit, and judges that the query log collection script has a process by utilizing a subpacess library according to a native shell instruction, and/or deletes the log query instruction and the query log collection script when a finish mark corresponding to the query log exists in the shared folder;
and the query log acquisition unit is connected with the log collection script process judgment unit and the log merging processor, and queries and extracts the target log on the server according to the query keyword information and the query range information when the query log collection script has no process and the shared folder has no finish mark corresponding to the query log.
Compared with the prior art, the method and the device for querying and extracting the server log at least realize one of the following beneficial effects:
(1) the method and the device for inquiring and extracting the server log realize message pushing and instruction execution by utilizing the self-contained timing task of the server instead of writing a agent, and can effectively reduce the resource consumption of the server. And the log extraction idea of collecting the finally combined logs independently is adopted, so that the concurrent effect of the log extraction is realized, and the time required by the log extraction is greatly saved.
(2) According to the server log query and extraction method and device, logs of the script acquisition server are executed by utilizing the timing any instruction set and the timing task execution instruction set of the log acquisition script, different operation and maintenance requirements can be realized by configuring different instructions in the instruction file, and the expansibility of log acquisition is improved. Compared with the existing log management platform, the log management platform has the advantages of small size, high remodeability and low consumption, and can be subjected to demand expansion according to business demands.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention. Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
fig. 1 is a schematic flowchart illustrating steps of a method for query extraction of a server log according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a method for query extraction of server logs according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a second method for query extraction of server logs according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a third method for server log query extraction according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a fourth method for server log query extraction according to an embodiment of the present invention;
fig. 6 is a schematic flowchart of a fifth method for query extraction of server logs according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an apparatus for server log query extraction according to an embodiment of the present invention;
FIG. 8 is a schematic structural diagram of a second apparatus for server log query extraction according to an embodiment of the present invention;
FIG. 9 is a block diagram illustrating an apparatus for query extraction of a third server log according to an embodiment of the present invention;
FIG. 10 is a block diagram illustrating an apparatus for query extraction of a fourth server log according to an embodiment of the present invention;
fig. 11 is a schematic structural diagram of a fifth apparatus for query extraction of server logs according to an embodiment of the present invention.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant application and are not limiting of the application. It should be noted that, for convenience of description, only the portions related to the present application are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1 to 2, fig. 1 is a schematic flow chart illustrating steps of a method for query extraction of a server log according to the present embodiment; fig. 2 is a schematic diagram illustrating a method for query extraction of a server log according to this embodiment, where the method includes the following steps:
102, receiving log query command information which is input by a client and carries query keyword information and query range information, and converting the log query command information into a log query instruction writing instruction set; and starting the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script.
103, when the process does not exist in the query log collection script and no finishing mark corresponding to the query log exists in the shared folder, querying and extracting a target log on the server according to the query keyword information and the query range information; writing the target log into a log receiving folder according to the server identification mark; and when the instruction set is executed, generating a finishing identifier corresponding to the query log and adding the finishing identifier to the shared folder.
And step 104, acquiring target logs from each log receiving folder according to the query keyword information and the query range information, and merging to obtain a query log set.
As shown in fig. 2, according to the command information for extracting the server log input by the user, the log collection script is executed by starting the timing task of each server (server a, server B, server C), and the corresponding log files on each server are extracted and stored in the corresponding log files respectively. And executing a monitoring script to monitor the instruction file for starting the timed task and the stored log file for circulating monitoring, judging whether the extracted log is finished, combining the dispersed server log files if the log acquisition is finished, and calling a python sending interface (python library) to send a log acquisition completion message.
The method can solve the problem of manually pulling pain points distributed on a plurality of server logs to a certain extent, adopts a message notification mode, and realizes batch extraction of logs distributed on different servers by matching the following scripts.
Timing tasks: using a Linux system timed task Crond to realize inquiring once per minute for the instruction file placed in the shared folder, if the instruction file is written into an instruction set needing to be executed, reading the instruction set and executing the instruction set on the current server,
a log collection script: the method comprises the steps of executing a timing task, finding a log file under tomcat in operation, judging whether a current server needs to extract logs and which logs according to an application name (input by a user through an execution monitoring script) in an instruction set, extracting log contents to a specific file of a shared folder if the current server needs to extract the logs, wherein the file takes an intranet IP (Internet protocol) of the server as a name, one server corresponds to one file, and no processing is performed if the file is not needed.
Executing the monitoring script: the script is executed by a user direct writing command line, keywords required to be inquired by a log and an application name required to be inquired are carried when the script is executed, the script can write the instruction of the user into an instruction file for executing a timing task, a circulation thread can be started, whether the log is completely extracted or not is judged by traversing specific files of a shared folder (each file has a finished mark after the log is completely extracted, and whether the log is completely extracted or not is judged by a circulation process under the condition that whether the number of finished files is equal to the number required to be finished or not (which is calculated by program logic), and after the log is completely extracted, the script deletes an instruction set of the instruction file, combines the specified files of the log, and sends a message to provide a downloading link.
The problem can not exist by adopting a message notification mode, only one shared folder is hung on each server, and the log collection pressure is much lower because each server is provided with a log collection script for storage and management.
The method specifically comprises the following steps: all servers needing to be managed are made to mount a windows shared folder; deploying a log collection script on the windows shared folder; a timed task (crontab-e edit) of executing a log collection script at regular time is added to each server, typically set to one minute in view of performance and real-time requirements. The format of the crontab file set time is as follows:
the command is used for executing a log collection script and carrying keywords and application name parameters; deploying an instruction file on the windows shared folder (the file is modified by an execution monitoring script, and the execution monitoring script converts the content input by a user into a timing task recognizable format for the timing task to execute); a specific file corresponding to each server is created in the windows shared folder, and the file is a blank file (suffix) with an IP (Internet protocol) name, and is mainly used for receiving a log collected by a server corresponding to the name.
In some optional embodiments, as shown in fig. 3, which is a schematic flowchart of a step of a second method for query extraction of server logs in this embodiment, different from fig. 1, the method obtains target logs from each log receiving folder according to query keyword information and query range information, and obtains a query log set by merging the target logs, where the method includes:
And step 302, when all the shared folders have the ending marks, acquiring target logs from all the log receiving folders according to the query keyword information and the query range information, and merging to obtain a query log set.
In some optional embodiments, as shown in fig. 4, which is a schematic step flow diagram of a third method for server log query extraction in this embodiment, different from that in fig. 1, log query command information carrying query keyword information and query range information and input by a client is received, and the log query command information is converted into a log query instruction write instruction set; starting a timing task to identify and execute a log query instruction in an instruction set, and generating a corresponding query log collection script, wherein the query log collection script comprises the following steps:
In some optional embodiments, as shown in fig. 5, a schematic flow chart of steps of a fourth method for server log query extraction in this embodiment is different from that in fig. 1, further including:
In some optional embodiments, as shown in fig. 6, which is a schematic step flow diagram of a fifth method for querying and extracting a server log in this embodiment, different from that in fig. 1, when a process does not exist in a query log collection script and there is no ending identifier corresponding to the query log in a shared folder, a target log on a server is queried and extracted according to query keyword information and query range information, and the method includes:
The method mainly comprises the step of executing a shell instruction set by using a subpacess library (the subpacess library executes an external instruction through a subprocess and acquires the executed return information of the subprocess through an input/output/error pipeline) to realize corresponding log extraction. The concrete implementation is as follows:
and executing a 'ps-ef | grep script name' (native shell instruction) by using a subpacess library to acquire whether the script has a process, thereby judging whether the server executes the script (preventing multiple problems caused by timing tasks), checking whether a corresponding file under a shared folder has a finish identifier or not, and if so, proving that the script has been executed.
And acquiring a server IP (calling linux instruction) for executing the script, reading the instruction in the instruction file below the shared folder, and judging whether the current server needs to execute log collection work according to the instruction set.
If the server has an application name in the operation instruction set, the script enters the application, and the grep (native shell instruction) is used for extracting the log and writing the log into a corresponding file under the shared folder. After the instruction set is executed, a finish identifier is added to a corresponding file in the shared folder (so that multiple times of execution are prevented).
In some optional embodiments, as shown in fig. 7, a schematic structural diagram of an apparatus 700 for server log query extraction according to this embodiment is provided, and the apparatus may be used to implement the method for server log query extraction described above. Specifically, the apparatus includes: a log query setter 701, a log collection script creation processor 702, a log collection processor 703, and a log merge processor 704.
The log query setter 701 is connected with the log collection script creating processor 702, sets a shared folder on the server, deploys the log collection script on the shared folder, and creates a log receiving folder in the shared folder according to the server identification mark; adding a timing task of a timing execution log collection script on the server, and deploying an instruction set for the timing task to execute on the server.
A log collection script creating processor 702, connected to the log query setter 701 and the log collection processor 703, for receiving log query command information carrying query keyword information and query range information (with an application name as a log range identifier) input by a client, and converting the log query command information into a log query command writing instruction set; and starting the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script.
A log collection processor 703 connected to the log collection script creating processor 702 and the log merging processor 704, for querying and extracting a target log on the server according to the query keyword information and the query range information when the query log collection script has no process and there is no finalization identifier corresponding to the query log in the shared folder; writing the target log into a log receiving folder according to the server identification mark; and when the instruction set is executed, generating a finishing identifier corresponding to the query log and adding the finishing identifier to the shared folder.
And a log merging processor 704 connected to the log collection processor 705, for obtaining target logs from each log receiving folder according to the query keyword information and the query range information, and merging the target logs to obtain a query log set.
In some optional embodiments, as shown in fig. 8, a schematic structural diagram of an apparatus 800 for extracting a second server log query in this embodiment is different from that in fig. 7, in which a log merging processor 704 includes: log extraction completion determination section 801 and extracted log merging section 802.
The log extraction completion determining unit 801 is connected to the log collection processor 703 and the extracted log merging unit 802, and checks the shared folder of each server according to the identification identifier of the server and the identification feature of the completion identifier.
An extracted log merging unit 802, connected to the log extraction completion determining unit 801, for obtaining target logs from each log receiving folder according to the query keyword information and the query range information when all the shared folders have completion identifiers, and merging the target logs to obtain a query log set; and generating a query log acquisition completion message, sending the query log acquisition completion message to the client, and deleting the log query instruction.
In some alternative embodiments, as shown in fig. 9, a schematic structural diagram of an apparatus 900 for extracting a third server log query in this embodiment is different from that in fig. 7 in that a log collection script creating processor 702 includes: a query scope determination unit 901 and a log collection script creation unit 902.
The query range determining unit 901 is connected to the log query setter 701 and the log collection script creating unit 902, receives log query command information which is input by the client and carries query keyword information and query range information, and converts the log query command information into a log query instruction writing instruction set; the instruction set is compared to the application name on the server.
And the log collection script creating unit 902 is connected with the query range determining unit 901 and the log collection processor 703, and when the application name on the server contains query range information of the log query instruction, starts the timing task to identify and execute the log query instruction in the instruction set, so as to generate a corresponding query log collection script.
In some optional embodiments, as shown in fig. 10, a schematic structural diagram of an apparatus 1000 for querying and extracting a fourth server log of this embodiment is different from that in fig. 7, further including: a timed task update processor 1001 connected to the log query setter 701 and configured to receive a modification instruction for modifying a timed task; and analyzing the modification instruction to obtain the updating time and the updating timing execution command, and replacing related contents in the timing task according to the updating time and the updating timing execution command.
In some optional embodiments, as shown in fig. 11, a schematic structural diagram of an apparatus 1100 for extracting a fifth server log query in this embodiment is different from that in fig. 7, where the log collection processor 703 includes: a log collection script process determination unit 1101 and an inquiry log collection unit 1102.
The log collection script process determining unit 1101 is connected to the log collection script creating processor 702 and the query log collecting unit 1102, and determines, by using the subparocess library, that the query log collection script exists in a process according to the native shell instruction, and/or deletes the log query instruction and the query log collection script when the shared folder has a finish identifier corresponding to the query log.
And the query log acquisition unit 1102 is connected with the log collection script process judgment unit 1101 and the log merging processor 704, and queries and extracts the target log on the server according to the query keyword information and the query range information when the query log collection script has no process and no finish identifier corresponding to the query log exists in the shared folder.
According to the embodiment, the method and the device for querying and extracting the server log have the following beneficial effects that:
(1) according to the method and the device for querying and extracting the server log, the linux instruction is executed in a message notification mode instead of a message push mode, so that resources occupied by the server can be effectively saved. On the basis of solving the problem of concurrent reading of files, the method adopts the step-by-step operation of firstly independent extraction and then combination, and does not use a mechanism similar to a lock, so that the log extraction time is effectively prolonged. The more servers that need to be operated, the more significant the benefits of such process optimization. For example: if two server logs need to be extracted, if the concept of lock is adopted, in order to avoid log loss caused by common operation, the next server log can be extracted after the previous server log is required to be extracted. Assuming that the average extraction time of the server log is 1 minute, it takes at least 2 minutes (ignoring other time) to extract the complete part of the log according to the idea, and the processing mode using the scheme can be reduced to less than one minute (ignoring other time). If the logs are distributed on thousands of servers, one thousand minutes is required to use the lock, and the theoretical shortest time for using the scheme is one minute.
(2) According to the method and the device for querying and extracting the server logs, the method for batch extracting the logs uses a large number of linux native instructions, the instructions are executed in a message notification mode, extra resource consumption is very low, the logs are used for querying and are not needed to be stored.
Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustrative purposes only and are not intended to limit the scope of the present invention. It will be appreciated by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.
Claims (10)
1. A method for server log query extraction, comprising:
setting a shared folder on a server, deploying a log collection script on the shared folder, and creating a log receiving folder in the shared folder according to the server identification mark; adding a timing task of a timing execution log collection script on the server, and deploying an instruction set for the timing task to execute on the server;
receiving log query command information which is input by a client and carries query keyword information and query range information, converting the log query command information into a log query instruction, and writing the log query instruction into the instruction set; enabling the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script;
when the process does not exist in the query log collection script and the shared folder does not have a finish mark corresponding to the query log, querying and extracting a target log on the server according to the query keyword information and the query range information; writing the target log into the log receiving folder according to the server identification mark; when the instruction set is executed, generating a finish identifier corresponding to the query log and adding the finish identifier to the shared folder;
and acquiring the target logs from each log receiving folder according to the query keyword information and the query range information, and combining to obtain a query log set.
2. The method according to claim 1, wherein the obtaining of the target log from each log receiving folder according to the query keyword information and the query range information and the merging of the target logs to obtain a query log set comprises:
checking the shared folder of each server according to the identification characteristics of the server identification and the finalization identification;
when all the shared folders have the ending marks, acquiring the target logs from all the log receiving folders according to the query keyword information and the query range information, and merging to obtain a query log set;
and generating a query log acquisition completion message, sending the query log acquisition completion message to the client, and deleting the log query instruction.
3. The method for server log query extraction as claimed in claim 1, wherein log query command information carrying query keyword information and query range information input by a client is received, and the log query command information is converted into a log query instruction and written into the instruction set; starting the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script, wherein the query log collection script comprises the following steps:
receiving log query command information which is input by a client and carries query keyword information and query range information, converting the log query command information into a log query instruction, and writing the log query instruction into the instruction set; comparing the instruction set to an application name on the server;
and when the application name on the server contains the query range information of the log query instruction, starting the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script.
4. The method of server log query extraction as claimed in claim 1, further comprising:
receiving a modification instruction for modifying the timing task;
and analyzing the modification instruction to obtain update time and an update timing execution command, and replacing related contents in the timing task according to the update time and the update timing execution command.
5. The method according to claim 1, wherein when the query log collection script has no process and the shared folder has no ending identifier corresponding to the query log, querying and extracting the target log on the server according to the query keyword information and the query range information includes:
judging that the query log collection script has a process by utilizing a supprocess library according to a native shell instruction, and/or deleting the log query instruction and the query log collection script when a finishing identifier corresponding to the query log exists in the shared folder;
and when the process does not exist in the query log collection script and the shared folder does not have the finishing identification corresponding to the query log, querying and extracting the target log on the server according to the query keyword information and the query range information.
6. An apparatus for server log query extraction, comprising: the log collection system comprises a log query setter, a log collection script creating processor, a log collection processor and a log merging processor; wherein,
the log query setter is connected with the log collection script creating processor, a shared folder is arranged on a server, a log collection script is deployed on the shared folder, and a log receiving folder is created in the shared folder according to the server identification mark; adding a timing task of a timing execution log collection script on the server, and deploying an instruction set for the timing task to execute on the server;
the log collection script creating processor is connected with the log query setter and the log collection processor, receives log query command information which is input by a client and carries query keyword information and query range information, converts the log query command information into a log query command and writes the log query command into the command set; enabling the timing task to identify and execute the log query instruction in the instruction set, and generating a corresponding query log collection script;
the log collection processor is connected with the log collection script creating processor and the log merging processor, and when the process does not exist in the query log collection script and the shared folder does not have a finish mark corresponding to the query log, the target log on the server is queried and extracted according to the query keyword information and the query range information; writing the target log into the log receiving folder according to the server identification mark; when the instruction set is executed, generating a finish identifier corresponding to the query log and adding the finish identifier to the shared folder;
and the log merging processor is connected with the log acquisition processor, acquires the target logs from each log receiving folder according to the query keyword information and the query range information, and merges to obtain a query log set.
7. The apparatus for server log query extraction as claimed in claim 6, wherein the log merge processor comprises: a log extraction completion determination unit and an extracted log merging unit; wherein,
the log extraction completion judging unit is connected with the log acquisition processor and the extracted log merging unit and is used for checking the shared folder of each server according to the identification of the server and the identification characteristics of the completion identification;
the extracted log merging unit is connected with the log extraction completion judging unit, and when completion identifiers exist in all the shared folders, the extracted log merging unit acquires the target logs from each log receiving folder according to the query keyword information and the query range information and merges the target logs to obtain a query log set; and generating a query log acquisition completion message, sending the query log acquisition completion message to the client, and deleting the log query instruction.
8. The apparatus for server log query extraction as claimed in claim 6, wherein the log collection script creation processor comprises: a query range judging unit and a log collection script creating unit; wherein,
the query range judging unit is connected with the log query setter and the log collection script creating unit, receives log query command information which is input by a client and carries query keyword information and query range information, converts the log query command information into a log query instruction and writes the log query instruction into the instruction set; comparing the instruction set to an application name on the server;
and the log collection script creating unit is connected with the query range judging unit and the log collection processor, and when the application name on the server contains query range information of the log query instruction, the timing task is started to identify and execute the log query instruction in the instruction set so as to generate a corresponding query log collection script.
9. The apparatus for server log query extraction as claimed in claim 6, further comprising: the timing task updating processor is connected with the log query setter and receives a modification instruction for modifying the timing task;
and analyzing the modification instruction to obtain update time and an update timing execution command, and replacing related contents in the timing task according to the update time and the update timing execution command.
10. The apparatus for server log query extraction as claimed in claim 6, wherein the log collection processor comprises: a log collection script process judgment unit and a query log collection unit; wherein,
the log collection script process judgment unit is connected with the log collection script creation processor and the query log collection unit, and is used for judging the process of the query log collection script according to a native shell instruction by using a subpacess library and/or deleting the log query instruction and the query log collection script when a finishing identifier corresponding to the query log exists in the shared folder;
and the query log acquisition unit is connected with the log collection script process judgment unit and the log merging processor, and queries and extracts the target log on the server according to the query keyword information and the query range information when the query log collection script has no process and the shared folder has no finish mark corresponding to the query log.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911069704.3A CN110781143B (en) | 2019-11-05 | 2019-11-05 | Method and device for querying and extracting server logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911069704.3A CN110781143B (en) | 2019-11-05 | 2019-11-05 | Method and device for querying and extracting server logs |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110781143A CN110781143A (en) | 2020-02-11 |
| CN110781143B true CN110781143B (en) | 2022-03-15 |
Family
ID=69388985
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911069704.3A Active CN110781143B (en) | 2019-11-05 | 2019-11-05 | Method and device for querying and extracting server logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781143B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111522714A (en) * | 2020-04-20 | 2020-08-11 | 京东数字科技控股有限公司 | Log query method and device, electronic equipment and storage medium |
| CN111865725B (en) * | 2020-07-29 | 2022-09-23 | 平安健康保险股份有限公司 | Flow consumption analysis method and system based on log |
| CN112084249B (en) * | 2020-09-11 | 2024-06-21 | 浙江立元科技有限公司 | Access record extraction method and device |
| CN112860456B (en) * | 2021-02-08 | 2023-07-21 | 青岛海尔科技有限公司 | Log processing method and device |
| CN113765694B (en) * | 2021-03-24 | 2024-04-19 | 北京京东拓先科技有限公司 | Log information display method, device, electronic equipment and computer readable medium |
| CN113282374B (en) * | 2021-06-25 | 2023-04-07 | 马上消费金融股份有限公司 | Server control method, device and system |
| CN113961518B (en) * | 2021-09-08 | 2022-09-23 | 北京百度网讯科技有限公司 | Log visual display method and device, electronic equipment and storage medium |
| CN114756573B (en) * | 2022-06-16 | 2022-10-04 | 恒生电子股份有限公司 | Data processing method, device and system |
| CN115378802B (en) * | 2022-08-24 | 2024-08-30 | 深圳市晨北科技有限公司 | Log collection method, device, equipment and computer readable storage medium |
| CN116866327B (en) * | 2023-07-11 | 2024-08-09 | 创泽智能机器人集团股份有限公司 | Method and device for viewing svn log based on web |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070100980A1 (en) * | 2005-11-01 | 2007-05-03 | Hitachi, Ltd. | Computer system and method for managing log information |
| US20080154842A1 (en) * | 2006-12-20 | 2008-06-26 | International Business Machines Corporation | Enhanced relational database management system and method |
| US8041683B1 (en) * | 2006-09-29 | 2011-10-18 | Emc Corporation | Methods and apparatus for locating network logs |
| CN103580899A (en) * | 2012-08-01 | 2014-02-12 | 中兴通讯股份有限公司 | Method and system for managing event logs, cloud service client side and virtualization platform |
| CN104580372A (en) * | 2014-12-12 | 2015-04-29 | 安徽富煌和利时科技股份有限公司 | Method for transmitting status files of electronic station board IPCs (Industrial Personal Computer) |
| US20160171505A1 (en) * | 2014-12-16 | 2016-06-16 | Verizon Patent And Licensing Inc. | Extract, transform, and load (etl) processing |
| CN107317862A (en) * | 2017-06-27 | 2017-11-03 | 昝立民 | The system of chain terminal system, distribution management method and the shared equipment of management |
| CN109257213A (en) * | 2018-09-07 | 2019-01-22 | 广东电网有限责任公司 | Judge the method and apparatus of terminal access authentication failed |
| CN109885548A (en) * | 2019-02-22 | 2019-06-14 | 网易（杭州）网络有限公司 | Log inquiring method, device, storage medium and electronic device |
-
2019
- 2019-11-05 CN CN201911069704.3A patent/CN110781143B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070100980A1 (en) * | 2005-11-01 | 2007-05-03 | Hitachi, Ltd. | Computer system and method for managing log information |
| US8041683B1 (en) * | 2006-09-29 | 2011-10-18 | Emc Corporation | Methods and apparatus for locating network logs |
| US20080154842A1 (en) * | 2006-12-20 | 2008-06-26 | International Business Machines Corporation | Enhanced relational database management system and method |
| CN103580899A (en) * | 2012-08-01 | 2014-02-12 | 中兴通讯股份有限公司 | Method and system for managing event logs, cloud service client side and virtualization platform |
| CN104580372A (en) * | 2014-12-12 | 2015-04-29 | 安徽富煌和利时科技股份有限公司 | Method for transmitting status files of electronic station board IPCs (Industrial Personal Computer) |
| US20160171505A1 (en) * | 2014-12-16 | 2016-06-16 | Verizon Patent And Licensing Inc. | Extract, transform, and load (etl) processing |
| CN107317862A (en) * | 2017-06-27 | 2017-11-03 | 昝立民 | The system of chain terminal system, distribution management method and the shared equipment of management |
| CN109257213A (en) * | 2018-09-07 | 2019-01-22 | 广东电网有限责任公司 | Judge the method and apparatus of terminal access authentication failed |
| CN109885548A (en) * | 2019-02-22 | 2019-06-14 | 网易（杭州）网络有限公司 | Log inquiring method, device, storage medium and electronic device |
Non-Patent Citations (1)
| Title |
|---|
| 基于大数据技术的IT基础设施日志分析系统设计与实现;张春生等;《微型电脑应用》;20160620(第06期);49-52 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110781143A (en) | 2020-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110781143B (en) | Method and device for querying and extracting server logs | |
| TWI524206B (en) | Program analysis/verification service providing system, method for controlling system, control program, control program for causing computer to operate, program analysis/verification device and program analysis/verification tool management device | |
| CN110569214B (en) | Index construction method and device for log file and electronic equipment | |
| JP5080428B2 (en) | Configuration management server, name identification method and name identification program | |
| CN103019874B (en) | Method and the device of abnormality processing is realized based on data syn-chronization | |
| US7913233B2 (en) | Performance analyzer | |
| US8832143B2 (en) | Client-side statement cache | |
| CN110968478B (en) | Log acquisition method, server and computer storage medium | |
| CN112083951A (en) | Software package unified management method and system supporting multiple operating system platforms | |
| CN109600385B (en) | Access control method and device | |
| CN111026709B (en) | Data processing method and device based on cluster access | |
| CN110597630B (en) | Method and system for processing content resources in distributed system | |
| CN111125213A (en) | Data acquisition method, device and system | |
| US20080162444A1 (en) | System and method for monitoring and providing patent information automatically | |
| US10303701B1 (en) | Real-time integrated replication system between databases | |
| CN114584486A (en) | Distributed network asset scanning detection platform and scanning detection method | |
| CN113641742A (en) | Data extraction method, device, equipment and storage medium | |
| CN110515979A (en) | Data query method, apparatus, equipment and storage medium | |
| KR20100037325A (en) | System and method for construction automatic bibliography based pattern, and recording medium therefor | |
| JP2003228498A (en) | History data collecting system and history data collecting program | |
| CN116257404A (en) | Log analysis method and computing device | |
| US20220121745A1 (en) | Method and system for labeling object and generating security policy of operating system | |
| US8037077B2 (en) | Computer-readable recording medium, method, and apparatus for creating message patterns | |
| CN107465762B (en) | Testing method and system for distributed storage system | |
| US8775528B2 (en) | Computer readable recording medium storing linking keyword automatically extracting program, linking keyword automatically extracting method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |