CN108153654B - Log collection method and device - Google Patents
Log collection method and device Download PDFInfo
- Publication number
- CN108153654B CN108153654B CN201711248381.5A CN201711248381A CN108153654B CN 108153654 B CN108153654 B CN 108153654B CN 201711248381 A CN201711248381 A CN 201711248381A CN 108153654 B CN108153654 B CN 108153654B
- Authority
- CN
- China
- Prior art keywords
- target task
- log
- task
- log collection
- shell
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 273
- 230000008569 process Effects 0.000 claims abstract description 220
- 238000004891 communication Methods 0.000 claims description 19
- 238000012544 monitoring process Methods 0.000 claims description 17
- 230000002159 abnormal effect Effects 0.000 claims description 15
- 230000002085 persistent effect Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 5
- 230000009545 invasion Effects 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 10
- 238000013515 script Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 241001178520 Stomatepia mongo Species 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a log collection method and a log collection device, wherein the method comprises the following steps: the method comprises the steps that a main process obtains task parameters of a target task to be subjected to log file collection; sending the task parameters to a shell process, and starting the shell process; starting a log collection thread by the shell process, sending the task parameters to the log collection thread, and obtaining a log file generated by the target task from output data of the target task according to the task parameters; and the log collection thread monitors the target task according to the task parameters, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted. By adopting the scheme provided by the embodiment of the invention to collect the logs, the invasion to the service of the target task is avoided, and the influence on the service execution is reduced.
Description
Technical Field
The invention relates to the technical field of internet software, in particular to a log collection method and device.
Background
The service programs running in the network equipment and the network system can generate log files when executing the service, and the log files can record the relevant contents of operations performed in the service executing process, so that developers can monitor the executing process of the service programs according to the log files, and especially when the service programs are abnormal, the log files can play a vital role in analyzing abnormal conditions.
However, the inventor finds that the prior art has at least the following problems in the process of implementing the invention:
at present, the collection of the log files is generally realized based on the scripts, that is, the scripts for collecting the log files are embedded in the service logic of the service executed by the service program, so that although the collection of the log files can be realized, the invasion to the service is too high, and the execution of the service is greatly influenced.
Disclosure of Invention
An object of the embodiments of the present invention is to provide a script-based log collection method, so as to avoid intrusion of log collection on a service executed by a service program, thereby improving performance of the service program when executing the service.
In order to achieve the above object, an embodiment of the present invention discloses a log collection method, where the method includes:
the method comprises the steps that a main process obtains task parameters of a target task to be subjected to log file collection; sending the task parameters to a shell process, and starting the shell process, wherein the main process is as follows: a process for executing the target task;
the shell process starts a log collection thread, sends the task parameters to the log collection thread, and obtains a log file generated by the target task from output data of the target task according to the task parameters;
and the log collection thread monitors the target task according to the task parameters, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted.
Preferably, the method further comprises:
the log collection thread monitors whether the target task is executed completely, and if the target task is executed completely, notification information is sent to the main process;
and the main process finishes the log collection thread and the shell process in sequence according to the received notification information sent by the log collection thread and exits.
Preferably, after the log collection thread monitors that the target task is executed completely, before the main process exits, the method further includes:
the main process acquires an exit code of the target task;
the main process detects whether the target task is abnormally exited or not according to the exit code;
if the main process detects that the target task is abnormally exited, determining the reason for the abnormal exit of the target task, and persisting the exit code and the reason for the abnormal exit;
and if the main process detects that the target task is not abnormal exit, directly persisting the exit code.
Preferably, the persisting the log file obtained by the shell process includes:
and the log collection thread stores the log file obtained by the shell process to a preset third-party database.
Preferably, the task parameters include: a name of the target task.
In order to achieve the above object, an embodiment of the present invention discloses a log collecting device, including:
the process starting module is used for acquiring task parameters of a target task to be subjected to log file collection by a main process; sending the task parameters to a shell process, and starting the shell process, wherein the main process is as follows: a process for executing the target task;
the log acquisition module is used for starting a log collection thread by the shell process, sending the task parameters to the log collection thread and acquiring a log file generated by the target task from output data of the target task according to the task parameters;
and the log storage module is used for monitoring the target task by the log collection thread according to the task parameters and persisting the log file obtained by the shell process when the generation of the log file is monitored.
Preferably, the apparatus further comprises:
the state monitoring module is used for monitoring whether the target task is executed or not by the log collection thread, and sending notification information to the main process if the target task is executed;
and the process quitting module is used for the main process to sequentially end the log collecting thread and the shell process according to the received notification information sent by the log collecting thread and quit.
Preferably, after the state monitoring module monitors that the target task is executed completely, and before the process exit module exits from the main process, the apparatus further includes:
a reason analysis module, configured to acquire, by the host process, an exit code of the target task; detecting whether the target task is abnormally exited or not according to the exit code; if the target task is detected to be abnormally exited, determining the reason for the abnormally exiting of the target task, and persisting the exit code and the reason for the abnormally exiting; and if the target task is detected not to be abnormally exited, directly saving the exit code.
Preferably, the log storage module is specifically configured to monitor the target task by the log collection thread according to the task parameter, and store a log file obtained by the shell process in a preset third-party database when it is monitored that the log file is generated.
In order to achieve the above object, an embodiment of the present invention discloses an electronic device, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
the processor, when executing the program stored in the memory, implements the method steps of any of the embodiments described above. In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to perform any one of the log collection methods described above.
In yet another aspect of the present invention, the present invention further provides a computer program product containing instructions, which when run on a computer, causes the computer to execute any one of the log collection methods described above.
According to the script-based log collection scheme provided by the embodiment of the invention, a log file generated by a target task is obtained from output data of the target task through a shell process; meanwhile, the target task is monitored through a log collection thread, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted, so that the mutual independence of log collection and service program service business service logic is realized, the invasion of the log collection to the service is avoided, the energy of a service developer is saved, and the performance of executing the service by the script is improved. Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a system block diagram of a log collection method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a log collection method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating another log collection method according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating another log collection method according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a log collection device according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of another log collection apparatus according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of another log collection apparatus according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to solve the problem that service execution is affected by overhigh service intrusion of log file collection on a service program in the prior art, the invention provides a log collection method and a log collection device.
The following generally describes a log collection method provided by an embodiment of the present invention.
As shown in fig. 1, which is a system block diagram of the log collection method provided in the embodiment of the present invention, it can be seen from the system block diagram shown in fig. 1 that the log collection process of the log collection method provided in the embodiment of the present invention is completely independent from the business logic inside the target task itself.
The shell process is a child process of the host process and the log collection thread is a child process of the shell process. The shell process is used for collecting the log files generated by the target task, and the log collection thread is used for monitoring the target task and also used for persisting data needing to be stored. Specifically, the data persisted by the log collection thread includes log files collected in the shell thread, and may further include information such as exit codes and abnormal exit reasons of the target task. The persistence process is a process of storing data in a database, for example, the database that can be used for storing data in the embodiment of the present invention includes a MySQL database, a Mongo database, and the like.
In one implementation, the log collection method includes:
the method comprises the steps that a main process obtains task parameters of a target task to be subjected to log file collection; sending the task parameters to a shell process, and starting the shell process, wherein the main process is as follows: a process for executing the target task;
the shell process starts a log collection thread, sends the task parameters to the log collection thread, and obtains a log file generated by the target task from output data of the target task according to the task parameters;
and the log collection thread monitors the target task according to the task parameters, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted.
By the log collection method provided by the embodiment of the invention, mutual independence of log collection and service logic of service program service is realized, invasion of log collection to service is avoided, the energy of service developers is saved, and the performance of script execution service is improved.
The following describes in detail a log collection method provided by an embodiment of the present invention with reference to specific embodiments.
As shown in fig. 2, a schematic flow chart of a log collection method provided in an embodiment of the present invention includes the following steps:
step S201: the method comprises the steps that a main process obtains task parameters of a target task to be subjected to log file collection; and sending the task parameters to a shell process, and starting the shell process.
Wherein, the main process is as follows: a process for executing the target task.
Specifically, the target task may be a task executed by a service program in the electronic device as an execution subject, and the task parameter of the target task may be a name of the target task or information such as an execution time of the target task.
In a specific application, the shell process may be written using python or other computer languages, and may or may not be consistent with the writing language of the target task, which is not limited in the present invention.
Because the same computer language is not needed to be used between the shell process and the target task, the developer is not limited to write the target task and does not need to care whether the used computer language supports the third-party database.
Step S202: and the shell process starts a log collection thread, sends the task parameters to the log collection thread, and obtains a log file generated by the target task from the output data of the target task according to the task parameters.
Specifically, after acquiring task parameters of the target task, the shell process may determine the target task according to the parameters, and may further collect all input and output data of the target task, where the output data of the target task includes a log file generated by the target task in the execution process. Of course, the shell process may also collect only the output data of the target task, or may also collect only the log file generated by the target task. Because the shell process and the target task are two mutually independent processes, the collection process of the shell process does not influence the operation of the target task.
In addition, the shell process can further start a log collection thread and send task parameters of the target task to the log collection thread, so that the log collection thread can identify the target task and perform the next monitoring on the target task.
Step S203: and the log collection thread monitors the target task according to the task parameters, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted.
Specifically, the log collection thread monitors the target task in real time, and if the target task is monitored to generate a log file, the log file collected in the shell process is persisted. Because the log collection thread monitors the target task in real time, the log collection thread collects log files in real time, and the real-time performance of log collection is improved.
In this embodiment, for example, the log file collected in the shell process may be stored in a third-party database. Specifically, the third-party database includes Redis, MySQL, Mongo, and the like.
As can be seen from the above, in the scheme provided in this embodiment, by starting the shell process and the log collection thread, the log collection process and the execution process of the target task are independent from each other, so that intrusion of log collection on the target task is avoided, and stable operation of the target task is facilitated.
In another embodiment of the invention, the host process finishes the shell process and the log collection thread at the same time after the target task is executed. As shown in fig. 3, which is a schematic flow chart of this embodiment, compared with the embodiment shown in fig. 1, step S304 and step S305 are added to this embodiment.
Step S301: the method comprises the steps that a main process obtains task parameters of a target task to be subjected to log file collection; and sending the task parameters to a shell process, and starting the shell process.
Wherein, the main process is as follows: a process for executing the target task.
Step S302: and the shell process starts a log collection thread, sends the task parameters to the log collection thread, and obtains a log file generated by the target task from the output data of the target task according to the task parameters.
Step S303: and the log collection thread monitors the target task according to the task parameters, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted.
Step S304: and the log collection thread monitors whether the target task is executed completely, and if the target task is executed completely, notification information is sent to the main process.
Specifically, when the log collection thread monitors the target task, the monitoring includes monitoring the running state of the target task, where the running state includes that the target task is running or the target task is finished running. And when the log collection thread monitors that the running state of the target task is running, sending notification information to the main process, and notifying the main process of the running state of the target task.
Step S305: and the main process finishes the log collection thread and the shell process in sequence according to the received notification information sent by the log collection thread and exits.
The main process can receive the notification information of the log collection thread, if the notification that the target task is completely operated and sent by the log collection thread is received, the log collection thread and the shell process are sequentially ended, and after the log collection thread and the shell process are quitted, the log collection thread and the shell process are automatically quitted, and the operation state is ended.
Steps S301 to S303 are the same as steps S201 to S203 in the embodiment of the invention shown in fig. 2, and are not described again.
As can be seen from the above, in the solution provided in this embodiment, the main process can learn the running state of the target task according to the monitoring result of the log collection thread on the target task, and when the running state of the target task is running, the log collection thread and the shell process are ended, thereby avoiding waste of resources.
In another embodiment of the invention, the main process detects the exit reason of the target task before exiting. As shown in fig. 4, which is a schematic flow chart of this embodiment, step S405 is added to this embodiment as compared with the embodiment shown in fig. 3.
Step S401: the method comprises the steps that a main process obtains task parameters of a target task to be subjected to log file collection; and sending the task parameters to a shell process, and starting the shell process.
Wherein, the main process is as follows: a process for executing the target task.
Step S402: and the shell process starts a log collection thread, sends the task parameters to the log collection thread, and obtains a log file generated by the target task from the output data of the target task according to the task parameters.
Step S403: and the log collection thread monitors the target task according to the task parameters, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted.
Step S404: and the log collection thread monitors whether the target task is executed completely, and if the target task is executed completely, notification information is sent to the main process.
Step S405: the main process acquires an exit code of the target task; detecting whether the target task is abnormally exited or not according to the exit code; if the target task is detected to be abnormally exited, determining the reason for the abnormally exiting of the target task, and persisting the exit code and the reason for the abnormally exiting; and if the target task is detected not to be abnormal exit, directly saving the exit code.
Each process generates an exit code when exiting, the exit code is generated by a preset rule, and different exit codes can directly reflect different exit reasons of the process, such as: the process normally exits or the process abnormally exits, wherein the process abnormally exits possibly because the process is forcibly stopped, the process is automatically exited after being blocked, and the like.
Specifically, after the main process obtains the information that the target task is executed, the exit code of the target task is further obtained, the exit code is analyzed according to a preset exit code generation rule, the exit code is matched with the exit reason of the process, and whether the target task is abnormally exited or not is judged. If the log file is abnormal exit, the exit code and the abnormal exit reason corresponding to the exit code are saved in the database together with the log file; if the exit is not abnormal, only the exit code is saved.
Step S406: and the main process finishes the log collection thread and the shell process in sequence according to the received notification information sent by the log collection thread and exits.
Steps S401 to S404 are the same as steps S301 to S304 of the embodiment of the present invention shown in fig. 2, and are not repeated here.
As can be seen from the above, in the scheme provided in this embodiment, the host process acquires and analyzes the exit code of the target task, stores the abnormal exit reason of the target task in the database, and supplements the collected log file, so that the running state and the result of the target task are recorded in more detail, which is beneficial for the service developer to subsequently detect and improve the target task.
Corresponding to the log collection method, the embodiment of the application also provides a log collection device.
Fig. 5 is a schematic structural diagram of a log collection device according to an embodiment of the present invention, where the log collection device includes:
a process starting module 510, configured to obtain a task parameter of a target task to be subjected to log file collection by a host process; and sending the task parameters to a shell process, and starting the shell process.
Wherein, the main process is as follows: a process for executing the target task.
And the log obtaining module 520 is configured to start a log collecting thread by the shell process, send the task parameter to the log collecting thread, and obtain a log file generated by the target task from output data of the target task according to the task parameter.
And the log storage module 530 is configured to monitor the target task by the log collection thread according to the task parameters, and when it is monitored that a log file is generated, persist the log file obtained by the shell process.
Specifically, the log storage module 530 is configured to monitor the target task by the log collection thread according to the task parameters, and store the log file obtained by the shell process in a preset third-party database when it is monitored that the log file is generated.
As can be seen from the above, in the present solution, the process starting module 510 starts the shell process and the log collection thread, and the log obtaining module 520 and the log storage module 530 mutually independent the log collection process and the execution process of the target task, so as to avoid the invasion of the log collection to the target task, and facilitate the stable operation of the target task.
Fig. 6 is a schematic structural diagram of a log collection device in another embodiment of the present invention, where the log collection device includes:
the process starting module 610 is used for the main process to obtain task parameters of a target task to be subjected to log file collection; sending the task parameters to a shell process, and starting the shell process, wherein the main process is as follows: a process for executing the target task.
And the log obtaining module 620 is configured to start a log collection thread by the shell process, send the task parameter to the log collection thread, and obtain a log file generated by the target task from output data of the target task according to the task parameter.
And the log storage module 630 is configured to monitor the target task by the log collection thread according to the task parameters, and when it is monitored that a log file is generated, persist the log file obtained by the shell process.
And the state monitoring module 640 is configured to monitor, by the log collection thread, whether the target task is executed completely, and send notification information to the host process if the target task is executed completely.
And a process exit module 650, configured to, by the host process, sequentially end the log collection thread and the shell process according to the received notification information sent by the log collection thread, and exit.
As can be seen from the above, in the present solution, in the state monitoring module 640, the main process monitors the target task in real time according to the log collection thread, knows that the target task is executed in real time, and immediately ends the log collection thread and the shell process, thereby avoiding the waste of resources.
Fig. 7 is a schematic structural diagram of a log collection device in another embodiment of the present invention, which includes:
the process starting module 710 is used for the main process to obtain task parameters of a target task to be subjected to log file collection; sending the task parameters to a shell process, and starting the shell process, wherein the main process is as follows: a process for executing the target task.
And the log obtaining module 720 is configured to start a log collection thread by the shell process, send the task parameter to the log collection thread, and obtain a log file generated by the target task from output data of the target task according to the task parameter.
And the log storage module 730 is used for monitoring the target task by the log collection thread according to the task parameters, and persisting the log file obtained by the shell process when the generation of the log file is monitored.
And a state monitoring module 740, configured to monitor, by the log collection thread, whether the target task is executed completely, and send notification information to the host process if it is detected that the target task is executed completely.
A reason analysis module 750, configured to acquire, by the host process, an exit code of the target task; detecting whether the target task is abnormally exited or not according to the exit code; if the target task is detected to be abnormally exited, determining the reason for the abnormally exiting of the target task, and persisting the exit code and the reason for the abnormally exiting; and if the target task is detected not to be abnormally exited, directly saving the exit code.
And the process exit module 760 is configured to, by the host process, sequentially end the log collection thread and the shell process according to the received notification information sent by the log collection thread, and exit.
As can be seen from the above, in the present solution, the reason analysis module 750 stores the abnormal exit reason of the target task in the database through the acquisition and analysis of the exit code of the target task by the host process, and supplements the collected log file, so that the running state and the result record of the target task are more detailed, which is beneficial to the subsequent detection and improvement of the target task by the service developer.
An embodiment of the present invention further provides an electronic device, as shown in fig. 8, which includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804, where the processor 801, the communication interface 802, and the memory 803 complete mutual communication through the communication bus 804,
a memory 803 for storing a computer program;
the processor 801 is configured to implement the log collection method according to the embodiment of the present invention when executing the program stored in the memory 803.
In an embodiment of the present invention, the log collecting method includes:
the method comprises the steps that a main process obtains task parameters of a target task to be subjected to log file collection; sending the task parameters to a shell process, and starting the shell process, wherein the main process is as follows: a process for executing the target task.
And the shell process starts a log collection thread, sends the task parameters to the log collection thread, and obtains a log file generated by the target task from the output data of the target task according to the task parameters.
And the log collection thread monitors the target task according to the task parameters, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted.
It should be noted that, other embodiments of the log collection method implemented by the processor 801 executing the program stored in the memory 803 are the same as the embodiments mentioned in the previous embodiment, and are not described herein again.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
Therefore, according to the scheme, the shell process and the log collection thread are started, the log collection process and the execution process of the target task are mutually independent, the invasion of the log collection to the target task is avoided, and the stable operation of the target task is facilitated; the same computer language is not needed to be used between the shell process and the target task, so that the writing of the target task by developers is not limited, and whether the used computer language supports a third-party database is not needed to be concerned; meanwhile, the real-time performance of log collection is improved due to the fact that the log collection thread monitors the target task in real time and stores the log file in real time.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, which has instructions stored therein, and when the instructions are executed on a computer, the instructions cause the computer to execute the log collection method described in any one of the above embodiments.
In yet another embodiment, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the log collection method of any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments.
In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A method of log collection, the method comprising:
the method comprises the steps that a main process obtains task parameters of a target task to be subjected to log file collection; sending the task parameters to a shell process, and starting the shell process, wherein the main process is as follows: a process for executing the target task, wherein the shell process is a child process of the main process;
the shell process starts a log collection thread, sends the task parameters to the log collection thread, and obtains a log file generated by the target task from output data of the target task according to the task parameters, wherein the log collection thread is a sub-thread of the shell process;
and the log collection thread monitors the target task according to the task parameters, and when the generation of a log file is monitored, the log file obtained by the shell process is persisted.
2. The method of claim 1, further comprising:
the log collection thread monitors whether the target task is executed completely, and if the target task is executed completely, notification information is sent to the main process;
and the main process finishes the log collection thread and the shell process in sequence according to the received notification information sent by the log collection thread and exits.
3. The method of claim 2, wherein: after the log collection thread monitors that the target task is executed completely, and before the main process exits, the method further includes:
the main process acquires an exit code of the target task;
the main process detects whether the target task is abnormally exited or not according to the exit code;
if the main process detects that the target task is abnormally exited, determining the reason for the abnormal exit of the target task, and persisting the exit code and the reason for the abnormal exit;
and if the main process detects that the target task is not abnormal exit, directly persisting the exit code.
4. The method according to any one of claims 1-3, wherein the persisting the log file obtained by the shell process comprises:
and the log collection thread stores the log file obtained by the shell process to a preset third-party database.
5. The method of any of claims 1-3, wherein the task parameters include: a name of the target task.
6. An apparatus for collecting logs, the apparatus comprising:
the process starting module is used for acquiring task parameters of a target task to be subjected to log file collection by a main process; sending the task parameters to a shell process, and starting the shell process, wherein the main process is as follows: a process for executing the target task, wherein the shell process is a child process of the main process;
the log acquisition module is used for starting a log collection thread by the shell process, sending the task parameters to the log collection thread and acquiring a log file generated by the target task from output data of the target task according to the task parameters, wherein the log collection thread is a sub-thread of the shell process;
and the log storage module is used for monitoring the target task by the log collection thread according to the task parameters and persisting the log file obtained by the shell process when the generation of the log file is monitored.
7. The apparatus of claim 6, further comprising:
the state monitoring module is used for monitoring whether the target task is executed or not by the log collection thread, and if the target task is executed, sending notification information to the main process;
and the process quitting module is used for the main process to sequentially end the log collecting thread and the shell process according to the received notification information sent by the log collecting thread and quit.
8. The apparatus of claim 7, wherein: after the state monitoring module monitors that the target task is executed completely, and before the process exit module exits from the main process, the device further comprises:
a reason analysis module, configured to acquire, by the host process, an exit code of the target task; detecting whether the target task is abnormally exited or not according to the exit code; if the target task is detected to be abnormally exited, determining the reason for the abnormally exiting of the target task, and persisting the exit code and the reason for the abnormally exiting; and if the target task is detected not to be abnormally exited, directly saving the exit code.
9. The apparatus according to any one of claims 6-8, wherein:
the log storage module is specifically configured to monitor the target task by the log collection thread according to the task parameters, and when it is monitored that a log file is generated, store the log file obtained by the shell process in a preset third-party database.
10. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 5 when executing a program stored in the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711248381.5A CN108153654B (en) | 2017-12-01 | 2017-12-01 | Log collection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711248381.5A CN108153654B (en) | 2017-12-01 | 2017-12-01 | Log collection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108153654A CN108153654A (en) | 2018-06-12 |
| CN108153654B true CN108153654B (en) | 2021-01-22 |
Family
ID=62465994
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711248381.5A Active CN108153654B (en) | 2017-12-01 | 2017-12-01 | Log collection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108153654B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913847B (en) * | 2020-07-21 | 2021-04-27 | 上海冰鉴信息科技有限公司 | Method and system for acquiring remote task execution progress |
| CN112732475A (en) * | 2021-01-13 | 2021-04-30 | 北京字节跳动网络技术有限公司 | Log collection method and device, electronic equipment and storage medium |
| CN114513502B (en) * | 2022-02-16 | 2024-01-26 | 中银金融科技有限公司 | File processing method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198007A (en) * | 2012-01-06 | 2013-07-10 | 腾讯科技（深圳）有限公司 | Multi-process log output method and system |
| CN105302483A (en) * | 2015-10-14 | 2016-02-03 | 北京乐动卓越科技有限公司 | High-stability log access method and system |
| CN106412061A (en) * | 2016-09-28 | 2017-02-15 | 上海爱数信息技术股份有限公司 | Linux-based log folder remote transmission system |
| CN106598815A (en) * | 2016-12-29 | 2017-04-26 | 广州华多网络科技有限公司 | Real-time asynchronous log collection method and system |
| CN106649071A (en) * | 2016-12-30 | 2017-05-10 | 广州华多网络科技有限公司 | Reporting method and device for internal process journals |
| CN107133086A (en) * | 2016-02-29 | 2017-09-05 | 阿里巴巴集团控股有限公司 | Task processing method, device and system based on distributed system |
| CN107291928A (en) * | 2017-06-29 | 2017-10-24 | 国信优易数据有限公司 | A kind of daily record storage system and method |
-
2017
- 2017-12-01 CN CN201711248381.5A patent/CN108153654B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198007A (en) * | 2012-01-06 | 2013-07-10 | 腾讯科技（深圳）有限公司 | Multi-process log output method and system |
| CN105302483A (en) * | 2015-10-14 | 2016-02-03 | 北京乐动卓越科技有限公司 | High-stability log access method and system |
| CN107133086A (en) * | 2016-02-29 | 2017-09-05 | 阿里巴巴集团控股有限公司 | Task processing method, device and system based on distributed system |
| CN106412061A (en) * | 2016-09-28 | 2017-02-15 | 上海爱数信息技术股份有限公司 | Linux-based log folder remote transmission system |
| CN106598815A (en) * | 2016-12-29 | 2017-04-26 | 广州华多网络科技有限公司 | Real-time asynchronous log collection method and system |
| CN106649071A (en) * | 2016-12-30 | 2017-05-10 | 广州华多网络科技有限公司 | Reporting method and device for internal process journals |
| CN107291928A (en) * | 2017-06-29 | 2017-10-24 | 国信优易数据有限公司 | A kind of daily record storage system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108153654A (en) | 2018-06-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109062809B (en) | Online test case generation method and device and electronic equipment | |
| CN109284269B (en) | Abnormal log analysis method and device, storage medium and server | |
| US11055169B2 (en) | Forecasting workload transaction response time | |
| CN108153654B (en) | Log collection method and device | |
| CN110515808B (en) | Database monitoring method and device, computer equipment and storage medium | |
| US8752027B2 (en) | Injecting faults into program for testing software | |
| CN109995612B (en) | Service inspection method and device and electronic equipment | |
| US20160274997A1 (en) | End user monitoring to automate issue tracking | |
| US8694831B2 (en) | Automatic bug reporting tool | |
| CN109344066B (en) | Method, system and terminal for testing browser page | |
| CN109324968B (en) | System testing method and device | |
| CN110851471A (en) | Distributed log data processing method, device and system | |
| CN110879781A (en) | Program debugging method and device, electronic equipment and computer readable storage medium | |
| CN110727572A (en) | Buried point data processing method, device, equipment and storage medium | |
| CN110597704B (en) | Pressure test method, device, server and medium for application program | |
| US11055416B2 (en) | Detecting vulnerabilities in applications during execution | |
| CN113114680A (en) | Detection method and detection device for file uploading vulnerability | |
| CN114020432A (en) | Task exception handling method and device and task exception handling system | |
| CN113918438A (en) | Method and device for detecting server abnormality, server and storage medium | |
| US20130007535A1 (en) | Trace capture of successfully completed transactions for trace debugging of failed transactions | |
| CN104077210A (en) | Positioning method and system of crashed client | |
| CN116738091A (en) | Page monitoring method and device, electronic equipment and storage medium | |
| CN116166536A (en) | Test method, test device, electronic equipment and storage medium | |
| CN114422175A (en) | Network security supervision and inspection behavior auditing method and device | |
| CN109165127B (en) | Problem interface positioning method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240207 Address after: 200335 room 205, floor 2, building 9, No. 365, Linhong Road, Changning District, Shanghai Patentee after: Shanghai iqiyi New Media Technology Co.,Ltd. Country or region after: China Address before: 10 / F and 11 / F, Hongcheng development building, 2 Haidian North 1st Street, Haidian District, Beijing 100080 Patentee before: BEIJING QIYI CENTURY SCIENCE & TECHNOLOGY Co.,Ltd. Country or region before: China |
|
| TR01 | Transfer of patent right |