CN108153654A - A kind of log collecting method and device - Google Patents
A kind of log collecting method and device Download PDFInfo
- Publication number
- CN108153654A CN108153654A CN201711248381.5A CN201711248381A CN108153654A CN 108153654 A CN108153654 A CN 108153654A CN 201711248381 A CN201711248381 A CN 201711248381A CN 108153654 A CN108153654 A CN 108153654A
- Authority
- CN
- China
- Prior art keywords
- task
- goal task
- log collection
- journal file
- exits
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 253
- 230000008569 process Effects 0.000 claims abstract description 200
- 230000002688 persistence Effects 0.000 claims abstract description 30
- 230000000977 initiatory effect Effects 0.000 claims abstract description 19
- 238000012544 monitoring process Methods 0.000 claims description 20
- 230000002159 abnormal effect Effects 0.000 claims description 19
- 238000004891 communication Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 8
- 238000004458 analytical method Methods 0.000 claims description 7
- 238000004321 preservation Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 230000009545 invasion Effects 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 16
- 238000003860 storage Methods 0.000 description 10
- 238000011161 development Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 241001178520 Stomatepia mongo Species 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Abstract
An embodiment of the present invention provides a kind of log collecting method and device, this method includes：Host process obtains the task parameters for the goal task that pending journal file is collected；The task parameters are sent to shell process, and start the shell process；The task parameters are sent to the log collection thread, and according to the task parameters, from the output data of the goal task, obtain the journal file of the goal task generation by shell process initiation log collection thread；Log collection thread is monitored the goal task according to the task parameters, when having monitored journal file generation, by the journal file persistence of shell process acquisition.Log collection is carried out using scheme provided in an embodiment of the present invention, the invasion to the business of goal task is avoided, reduces the influence performed to business.
Description
Technical field
The present invention relates to internet works software technical field, more particularly to a kind of log collecting method and device.
Background technology
The service routine run in the network equipment, network system can all generate journal file in the business of execution, these days
Will file would generally record the related content for performing and being operated conducted in business procedure, and such developer can be according to above-mentioned day
Will file is monitored the implementation procedure of above-mentioned service routine, especially when service routine occurs abnormal, above-mentioned daily record text
Part can play the analysis of abnormal conditions vital.
However, inventor has found in the implementation of the present invention, at least there are the following problems for the prior art：
It is generally based on script at present and realizes what journal file was collected, it is, by for the script of collector journal file
It is nested in the service logic of business performed by service routine, although the collection of journal file can be realized in this way, to industry
The intrusion of business is excessively high, so as to be brought greater impact to the execution of business.
Invention content
The embodiment of the present invention is designed to provide a kind of log collecting method based on script, to avoid log collection pair
The intrusion of business performed by service routine, so as to improve performance when service routine performs business.
In order to achieve the above objectives, the embodiment of the invention discloses a kind of log collecting method, the method includes：
Host process obtains the task parameters for the goal task that pending journal file is collected；The task parameters are sent to
Shell process, and start the shell process, wherein, the host process is：For performing the process of the goal task；
The task parameters are sent to the log collection thread, and root by the shell process initiation log collection thread
According to the task parameters, from the output data of the goal task, the journal file of the goal task generation is obtained；
The log collection thread is monitored the goal task according to the task parameters, when having monitored daily record
During file generated, by the journal file persistence of shell process acquisition.
Preferably, the method further includes：
The log collection thread monitors whether the goal task is finished, if monitoring that the goal task is held
After row, then notification information is sent to the host process；
The notification information that the host process is sent according to the log collection thread received, terminates the daily record successively
Thread and shell process are collected, and is exited.
Preferably, after the log collection thread monitors that the goal task is finished, the host process exits
Before, the method further includes：
The host process acquisition goal task exits code；
The host process exits code according to, detects whether the goal task is abnormal exit；
The host process, if detecting, the goal task is abnormal exits, it is determined that the goal task exits extremely
The reason of, and exit code and persistence the reason of the exception exits by described；
The host process, if detecting, the goal task is not that exception exits, and directly exits a yard persistence by described.
Preferably, the journal file persistence that the shell process is obtained, including：
The log collection thread stores the journal file that the shell process obtains to preset third party database.
Preferably, the task parameters include：The title of the goal task.
In order to achieve the above objectives, the embodiment of the invention discloses a kind of log collection device, described device includes：
Process initiation module obtains the task parameters for the goal task that pending journal file is collected for host process；It will
The task parameters are sent to shell process, and start the shell process, wherein, the host process is：For performing the target
The process of task；
Log acquisition module for the shell process initiation log collection thread, the task parameters is sent to described
Log collection thread, and according to the task parameters, from the output data of the goal task, obtain the goal task life
Into journal file；
Daily record memory module supervises the goal task according to the task parameters for the log collection thread
It surveys, when having monitored journal file generation, by the journal file persistence of shell process acquisition.
Preferably, described device further includes：
State monitoring module monitors whether the goal task is finished for the log collection thread, if prison
It measures after the goal task is finished, then sends notification information to the host process；
Process exits module, is believed for the host process according to the notice that the log collection thread received is sent
Breath, terminates the log collection thread and shell process, and exit successively.
Preferably, after the state monitoring module monitors that the goal task is finished, the process exits mould
Block is exited before host process, and described device further includes：
Analysis of causes module exits code for the host process acquisition goal task；Code, inspection are exited according to described
Survey whether the goal task is abnormal exit；If it is abnormal exit to detect the goal task, it is determined that the target is appointed
Business is abnormal the reason of exiting, and exits code and persistence the reason of the exception exits by described；If detecting, the target is appointed
Business is not that exception exits, then directly exits code described in preservation.
Preferably, the daily record memory module, specifically for the log collection thread according to the task parameters to institute
Goal task is stated to be monitored, when having monitored journal file generation, by the shell process obtain journal file store to
Preset third party database.
In order to achieve the above objectives, the embodiment of the invention discloses a kind of electronic equipment, including processor, communication interface, deposit
Reservoir and communication bus, wherein, processor, communication interface, memory completes mutual communication by communication bus；
Memory, for storing computer program；
Processor during for performing the program stored on memory, realizes the method step of any of the above-described specific embodiment
Suddenly.At the another aspect that the present invention is implemented, a kind of computer readable storage medium, the computer-readable storage medium are additionally provided
Instruction is stored in matter, when run on a computer so that computer performs any of the above-described log collecting method.
At the another aspect that the present invention is implemented, the embodiment of the present invention additionally provides a kind of computer program production comprising instruction
Product, when run on a computer so that computer performs any of the above-described log collecting method.
Log collection scheme provided in an embodiment of the present invention based on script, by shell process from the output number of goal task
In, the journal file of the goal task generation is obtained；The goal task is supervised by log collection thread simultaneously
It surveys, when having monitored journal file generation, by the journal file persistence of shell process acquisition, it is achieved thereby that daily record is received
Collection is mutual indepedent with the service logic of service routine business, avoids invasion of the log collection to business, saves business development
The energy of personnel, while improve the performance that script performs business.Certainly, it implements any of the products of the present invention or method must differ
It is fixed to need to reach all the above advantage simultaneously.
Description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described.
Fig. 1 is the system block diagram of log collecting method provided in an embodiment of the present invention；
A kind of Fig. 2 flow diagrams of log collecting method provided in an embodiment of the present invention；
Fig. 3 is the flow diagram of another log collecting method provided in an embodiment of the present invention；
Fig. 4 is the flow diagram of another log collecting method provided in an embodiment of the present invention；
Fig. 5 is a kind of structure diagram of log collection device provided in an embodiment of the present invention；
Fig. 6 is the structure diagram of another log collection device provided in an embodiment of the present invention；
Fig. 7 is the structure diagram of another log collection device provided in an embodiment of the present invention；
Fig. 8 is the structure diagram of a kind of electronic equipment provided in an embodiment of the present invention.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other without making creative work
Embodiment shall fall within the protection scope of the present invention.
It is excessively high to the business invasion performed by service routine in order to solve journal file collection in the prior art, influence business
The problem of execution, the present invention propose a kind of log collecting method and device.
First log collecting method provided in an embodiment of the present invention is illustrated on the whole below.
As shown in Figure 1, the system block diagram for log collecting method provided in an embodiment of the present invention, system according to figure 1
System block diagram can be seen that in the log collection process of the log collecting method provided in the embodiment of the present invention and goal task itself
The service logic in portion is completely independent.
Shell process is the subprocess of host process, and log collection thread is the sub-line journey of shell process.Wherein, shell process for pair
The journal file of goal task generation is collected, and log collection thread is additionally operable to for monitoring goal task to needing
The data stored carry out persistence.Specifically, the data of log collection thread persistence include collecting in shell thread
Journal file, can additionally include exiting code, abnormal exiting the information such as reason for goal task.The process of persistence is exactly
Data are stored to the database that the process of database, for example, in the embodiment of the present invention can be used for storing data to include
MySQL database and Mongo databases.
In a kind of realization method, above-mentioned log collecting method includes：
Host process obtains the task parameters for the goal task that pending journal file is collected；The task parameters are sent to
Shell process, and start the shell process, wherein, the host process is：For performing the process of the goal task；
The task parameters are sent to the log collection thread, and root by the shell process initiation log collection thread
According to the task parameters, from the output data of the goal task, the journal file of the goal task generation is obtained；
The log collection thread is monitored the goal task according to the task parameters, when having monitored daily record
During file generated, by the journal file persistence of shell process acquisition.
The log collecting method provided through the embodiment of the present invention realizes the business of log collection and service routine business
Logic is mutual indepedent, avoids invasion of the log collection to business, saves the energy of business development personnel, while improves foot
The performance of this execution business.
Below by specific embodiment, log collecting method provided in an embodiment of the present invention is described in detail.
As shown in Fig. 2, for a kind of flow diagram of log collecting method provided in an embodiment of the present invention, including walking as follows
Suddenly：
Step S201：Host process obtains the task parameters for the goal task that pending journal file is collected；By the task
Parameter is sent to shell process, and starts the shell process.
Wherein, host process is：For the process of performance objective task.
Specifically, goal task can be the task that service routine performs in the electronic equipment as executive agent, target
The task parameters of task can be the information such as the title of goal task or the execution time of goal task.
In concrete application, shell process can utilize python or other computer languages to write, with writing for goal task
Language can be consistent, can not also be consistent, and the present invention is defined not to this.
Since same computer language need not be used between shell process and goal task so that developer appoints target
Writing for business is unrestricted, it is not necessary to which whether the computer language used when being concerned about supports third party database.
Step S202：The task parameters are sent to the log collection by the shell process initiation log collection thread
Thread, and according to the task parameters, from the output data of the goal task, obtain the daily record of the goal task generation
File.
Specifically, shell process is after the task parameters for obtaining goal task, it is possible to goal task is determined according to the parameter,
And then all data that output and input of goal task can be collected, wherein, in the output data of goal task, packet
Include the journal file that goal task generates in the process of implementation.Certainly, shell process can also only collect the defeated of goal task
Go out data or can also only collect the journal file of goal task generation.Since shell process and goal task are two mutual
Mutually independent process, therefore the collection process of shell process does not interfere with the operation of goal task itself.
In addition, shell process further starting log can collect thread, and the task parameters of goal task are sent to day
Will collects thread so that log collection thread can identify goal task, carry out the monitoring in next step to goal task.
Step S203：The log collection thread is monitored the goal task according to the task parameters, works as prison
When having measured journal file generation, by the journal file persistence of shell process acquisition.
Specifically, log collection thread monitors goal task in real time, if monitoring that goal task generates journal file,
The journal file persistence being just collected into shell process.Since log collection thread is the real-time monitoring to goal task,
Therefore it is also to be collected in real time to journal file, so as to improve the real-time of log collection.
Wherein, persistence is exactly to preserve the journal file in shell process to can be in the equipment of persistence, for example, at this
In embodiment, the journal file being collected into shell process can be stored into third party database.Specifically, third party's data
Library is including Redis, MySQL, Mongo etc..
As seen from the above, in scheme provided in this embodiment, by starting shell process and log collection thread, by daily record
Collection process and the implementation procedure of goal task are mutual indepedent, avoid invasion of the log collection to goal task, are conducive to mesh
The stable operation of mark task.
In another embodiment of the invention, host process is after goal task is finished, at the same finish shell into
Journey and log collection thread.As shown in figure 3, the flow diagram for the embodiment, compared with embodiment illustrated in fig. 1, this implementation
Example increases step S304 and step S305.
Step S301：Host process obtains the task parameters for the goal task that pending journal file is collected；By the task
Parameter is sent to shell process, and starts the shell process.
Wherein, host process is：For the process of performance objective task.
Step S302：The task parameters are sent to the log collection by the shell process initiation log collection thread
Thread, and according to the task parameters, from the output data of the goal task, obtain the daily record of the goal task generation
File.
Step S303：The log collection thread is monitored the goal task according to the task parameters, works as prison
When having measured journal file generation, by the journal file persistence of shell process acquisition.
Step S304：The log collection thread monitors whether the goal task is finished, if detected described
After goal task is finished, then notification information is sent to the host process.
Specifically, when log collection thread is monitored goal task, include the prison of the operating status to goal task
It surveys, wherein, operating status is currently running including goal task or goal task operation finishes.When log collection thread monitors mesh
When the operating status of mark task is finished for operation, then notification information is sent to host process, the operating status of goal task is notified
Host process.
Step S305：The notification information that the host process is sent according to the log collection thread received, is tied successively
Shu Suoshu log collections thread and shell process, and exit.
Host process can receive the notification information of log collection thread, if having received the mesh of log collection thread transmission
The notice that mark task run finishes, then end log collects thread and shell process successively, and in log collection thread and shell process
It after exiting, automatically exits from, terminates operating status.
Step S301 to step S303 is identical to step S203 with the step S201 of inventive embodiments shown in Fig. 2, here not
It repeats one by one again.
As seen from the above, in scheme provided in this embodiment, host process is according to log collection thread to goal task
Monitoring result can learn the operating status of goal task, when the operating status of goal task is finished for operation, end log
Thread and shell process are collected, avoids the waste of resource.
In another embodiment of the invention, host process before exiting, carries out the reason that exits of goal task
Detection.As shown in figure 4, the flow diagram for the embodiment, compared with embodiment illustrated in fig. 3, embodiment adds steps
S405。
Step S401：Host process obtains the task parameters for the goal task that pending journal file is collected；By the task
Parameter is sent to shell process, and starts the shell process.
Wherein, host process is：For the process of performance objective task.
Step S402：The task parameters are sent to the log collection by the shell process initiation log collection thread
Thread, and according to the task parameters, from the output data of the goal task, obtain the daily record of the goal task generation
File.
Step S403：The log collection thread is monitored the goal task according to the task parameters, works as prison
When having measured journal file generation, by the journal file persistence of shell process acquisition.
Step S404：The log collection thread monitors whether the goal task is finished, if detected described
After goal task is finished, then notification information is sent to the host process.
Step S405：The host process acquisition goal task exits code；Code is exited according to described, detects the mesh
Whether mark task is abnormal exit；If it is abnormal exit to detect the goal task, it is determined that the goal task moves back extremely
The reason of going out, and exit code and persistence the reason of the exception exits by described；If detecting that the goal task is not
Exception exits, then directly exits code described in preservation.
Each process can all generate when exiting and exit code, exit code and generated by default rule, and different exits code meeting
The difference of directly reflection process exits reason, such as：Process normally exits or process exception exits, and wherein process exception has exited
May be since process operation is forcibly stopped, process interim card automatically exits from.
Specifically, after host process obtains the message that goal task is finished, exiting for goal task is further obtained
Code, analyzes exiting code according to the preset yard create-rule that exits, will move out code and matched with the reason that exits of process,
Judge whether goal task is abnormal exit.If abnormal exit, then it will move out code exception corresponding with code is exited and exit original
Because being preserved together with journal file into database；It is exited if not abnormal, then need to only will move out code and be preserved.
Step S406：The notification information that the host process is sent according to the log collection thread received, is tied successively
Shu Suoshu log collections thread and shell process, and exit.
Step S401 to step S404 is identical to step S304 with the step S301 of inventive embodiments shown in Fig. 2, here not
It repeats one by one again.
As seen from the above, in scheme provided in this embodiment, goal task is exited by host process code acquisition and
Analysis, exits reason by the exception of goal task and stores to database, the journal file being collected into is supplemented so that right
Operating status and the result record of goal task in further detail, be conducive to business development personnel subsequently to the detection of goal task and
It improves.
Corresponding with above-mentioned log collecting method, the embodiment of the present application additionally provides a kind of log collection device.
As shown in figure 5, for the structure diagram of log collection device a kind of in the embodiment of the present invention, which includes：
Process initiation module 510 obtains the task parameters for the goal task that pending journal file is collected for host process；
The task parameters are sent to shell process, and start the shell process.
Wherein, host process is：For the process of performance objective task.
The task parameters for the shell process initiation log collection thread, are sent to institute by log acquisition module 520
Log collection thread is stated, and according to the task parameters, from the output data of the goal task, obtains the goal task
The journal file of generation.
Daily record memory module 530, for the log collection thread according to the task parameters to the goal task into
Row monitoring, when having monitored journal file generation, by the journal file persistence of shell process acquisition.
Specifically, daily record memory module 530 carries out goal task according to task parameters for the log collection thread
Monitoring, when having monitored journal file generation, the journal file that shell process obtains is stored to preset third party database.
As seen from the above, in the present solution, starting shell process and log collection thread, day by process initiation module 510
Will acquisition module 520 and daily record memory module 530 are mutual indepedent by the implementation procedure of log collection process and goal task, avoid
Invasion of the log collection to goal task, is conducive to the stable operation of goal task.
The structure diagram of log collection device in another embodiment of the present invention is illustrated in figure 6, which includes：
Process initiation module 610 obtains the task parameters for the goal task that pending journal file is collected for host process；
The task parameters are sent to shell process, and start the shell process, wherein, the host process is：For performing the mesh
The process of mark task.
The task parameters for the shell process initiation log collection thread, are sent to institute by log acquisition module 620
Log collection thread is stated, and according to the task parameters, from the output data of the goal task, obtains the goal task
The journal file of generation.
Daily record memory module 630, for the log collection thread according to the task parameters to the goal task into
Row monitoring, when having monitored journal file generation, by the journal file persistence of shell process acquisition.
State monitoring module 640 monitors whether the goal task is finished for the log collection thread, if
It detects that the goal task is finished, then sends notification information to the host process.
Process exits module 650, the notice sent for the host process according to the log collection thread received
Information terminates the log collection thread and shell process, and exits successively.
As seen from the above, in the present solution, in state monitoring module 640, host process is according to log collection thread to mesh
The real-time monitoring of mark task learns that goal task is finished in real time, and end log collects thread and shell process immediately, avoids
The waste of resource.
The structure diagram of log collection device in another embodiment of the present invention is illustrated in figure 7, which includes：
Process initiation module 710 obtains the task parameters for the goal task that pending journal file is collected for host process；
The task parameters are sent to shell process, and start the shell process, wherein, the host process is：For performing the mesh
The process of mark task.
The task parameters for the shell process initiation log collection thread, are sent to institute by log acquisition module 720
Log collection thread is stated, and according to the task parameters, from the output data of the goal task, obtains the goal task
The journal file of generation.
Daily record memory module 730, for the log collection thread according to the task parameters to the goal task into
Row monitoring, when having monitored journal file generation, by the journal file persistence of shell process acquisition.
State monitoring module 740 monitors whether the goal task is finished for the log collection thread, if
It detects that the goal task is finished, then sends notification information to the host process.
Analysis of causes module 750 exits code for the host process acquisition goal task；It is exited according to described
Code detects whether the goal task is abnormal exit；If it is abnormal exit to detect the goal task, it is determined that the mesh
Mark task abnormity the reason of exiting, and exit code and persistence the reason of the exception exits by described；If detect the mesh
Mark task is not that exception exits, then directly exits code described in preservation.
Process exits module 760, the notice sent for the host process according to the log collection thread received
Information terminates the log collection thread and shell process, and exits successively.
As seen from the above, in the present solution, analysis of causes module 750 by host process exits goal task the acquisition of code
And analysis, the exception of goal task is exited into reason and is stored to database, the journal file being collected into is supplemented so that
Operating status and result record to goal task are conducive to business development personnel subsequently to the detection of goal task in further detail
And improvement.
The embodiment of the present invention additionally provides a kind of electronic equipment, as shown in figure 8, including processor 801, communication interface 802,
Memory 803 and communication bus 804, wherein, processor 801, communication interface 802, memory 803 is complete by communication bus 804
Into mutual communication,
Memory 803, for storing computer program；
Processor 801 during for performing the program stored on memory 803, realizes day provided in an embodiment of the present invention
Will collection method.
In one embodiment of the present of invention, above-mentioned log collecting method, including：
Host process obtains the task parameters for the goal task that pending journal file is collected；The task parameters are sent to
Shell process, and start the shell process, wherein, the host process is：For performing the process of the goal task.
The task parameters are sent to the log collection thread, and root by the shell process initiation log collection thread
According to the task parameters, from the output data of the goal task, the journal file of the goal task generation is obtained.
The log collection thread is monitored the goal task according to the task parameters, when having monitored daily record
During file generated, by the journal file persistence of shell process acquisition.
It should be noted that the program stored on the above-mentioned execution of processor 801 memory 803, and the log collection realized
The other embodiment of method, mutually identical with the preceding method embodiment part embodiment referred to, which is not described herein again.
The communication bus that above-mentioned electronic equipment is mentioned can be Peripheral Component Interconnect standard (Peripheral Component
Interconnect, PCI) bus or expanding the industrial standard structure (Extended Industry Standard
Architecture, EISA) bus etc..The communication bus can be divided into address bus, data/address bus, controlling bus etc..For just
It is only represented in expression, figure with a thick line, it is not intended that an only bus or a type of bus.
Communication interface is for the communication between above-mentioned electronic equipment and other equipment.
Memory can include random access memory (Random Access Memory, RAM), can also include non-easy
The property lost memory (Non-Volatile Memory, NVM), for example, at least a magnetic disk storage.Optionally, memory may be used also
To be at least one storage device for being located remotely from aforementioned processor.
Above-mentioned processor can be general processor, including central processing unit (Central Processing Unit,
CPU), network processing unit (Network Processor, NP) etc.；It can also be digital signal processor (Digital Signal
Processing, DSP), it is application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), existing
It is field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete
Door or transistor logic, discrete hardware components.
As seen from the above, in the present solution, by starting shell process and log collection thread, by log collection process and mesh
The implementation procedure of mark task is mutual indepedent, avoids invasion of the log collection to goal task, is conducive to the stabilization of goal task
Operation；Same computer language need not be used between shell process and goal task so that developer is to the volume of goal task
It writes unrestricted, it is not necessary to which whether the computer language used when being concerned about supports third party database；Simultaneously as log collection line
Real-time preservation of the journey to the real-time monitoring of goal task and to journal file improves the real-time of log collection.
In another embodiment provided by the invention, a kind of computer readable storage medium is additionally provided, which can
It reads to be stored with instruction in storage medium, when run on a computer so that computer performs any institute in above-described embodiment
The log collecting method stated.
In another embodiment provided by the invention, a kind of computer program product for including instruction is additionally provided, when it
When running on computers so that computer performs any log collecting method in above-described embodiment.
In the above-described embodiments, can come wholly or partly by software, hardware, firmware or its arbitrary combination real
It is existing.When implemented in software, it can entirely or partly realize in the form of a computer program product.The computer program
Product includes one or more computer instructions.When loading on computers and performing the computer program instructions, all or
It partly generates according to the flow or function described in the embodiment of the present invention.The computer can be all-purpose computer, special meter
Calculation machine, computer network or other programmable devices.The computer instruction can be stored in computer readable storage medium
In or from a computer readable storage medium to another computer readable storage medium transmit, for example, the computer
Instruction can pass through wired (such as coaxial cable, optical fiber, number from a web-site, computer, server or data center
User's line (DSL)) or wireless (such as infrared, wireless, microwave etc.) mode to another web-site, computer, server or
Data center is transmitted.The computer readable storage medium can be any usable medium that computer can access or
It is the data storage devices such as server, the data center integrated comprising one or more usable mediums.The usable medium can be with
It is magnetic medium, (for example, floppy disk, hard disk, tape), optical medium (for example, DVD) or semiconductor medium (such as solid state disk
SolidState Disk (SSD)) etc..
It should be noted that herein, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any this practical relationship or sequence.Moreover, term " comprising ", "comprising" or its any other variant are intended to
Non-exclusive inclusion, so that process, method, article or equipment including a series of elements not only will including those
Element, but also including other elements that are not explicitly listed or further include as this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that
Also there are other identical elements in process, method, article or equipment including the element.
Each embodiment in this specification is described using relevant mode, identical similar portion between each embodiment
Point just to refer each other, and the highlights of each of the examples are difference from other examples.
For device embodiment, since it is substantially similar to embodiment of the method, so the comparison of description is simple
Single, the relevent part can refer to the partial explaination of embodiments of method.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present invention
It is interior.
Claims (10)
1. a kind of log collecting method, which is characterized in that the method includes：
Host process obtains the task parameters for the goal task that pending journal file is collected；By the task parameters be sent to shell into
Journey, and start the shell process, wherein, the host process is：For performing the process of the goal task；
The task parameters are sent to the log collection thread, and according to institute by the shell process initiation log collection thread
Task parameters are stated, from the output data of the goal task, obtain the journal file of the goal task generation；
The log collection thread is monitored the goal task according to the task parameters, when having monitored journal file
During generation, by the journal file persistence of shell process acquisition.
2. according to the method described in claim 1, it is characterized in that, the method further includes：
The log collection thread monitors whether the goal task is finished, if detecting that the goal task has performed
Finish, then send notification information to the host process；
The notification information that the host process is sent according to the log collection thread received, terminates the log collection successively
Thread and shell process, and exit.
3. according to the method described in claim 2, it is characterized in that：The goal task is monitored in the log collection thread
After being finished, before the host process exits, the method further includes：
The host process acquisition goal task exits code；
The host process exits code according to, detects whether the goal task is abnormal exit；
The host process, if detecting, the goal task is abnormal exits, it is determined that the original that the goal task exits extremely
Cause, and exit code and persistence the reason of the exception exits by described；
The host process, if detecting, the goal task is not that exception exits, and directly exits a yard persistence by described.
4. method according to any one of claim 1-3, which is characterized in that the daily record for obtaining the shell process
File persistence, including：
The log collection thread stores the journal file that the shell process obtains to preset third party database.
5. method according to any one of claim 1-3, which is characterized in that the task parameters include：The target
The title of task.
6. a kind of log collection device, which is characterized in that described device includes：
Process initiation module obtains the task parameters for the goal task that pending journal file is collected for host process；By described in
Task parameters are sent to shell process, and start the shell process, wherein, the host process is：For performing the goal task
Process；
The task parameters for the shell process initiation log collection thread, are sent to the daily record by log acquisition module
Thread is collected, and according to the task parameters, from the output data of the goal task, obtains the goal task generation
Journal file；
Daily record memory module is monitored the goal task according to the task parameters for the log collection thread,
When having monitored journal file generation, by the journal file persistence of shell process acquisition.
7. device according to claim 6, which is characterized in that described device further includes：
State monitoring module monitors whether the goal task is finished, if detected for the log collection thread
The goal task is finished, then sends notification information to the host process；
Process exits module, for the notification information that the host process is sent according to the log collection thread that receives, according to
It is secondary to terminate the log collection thread and shell process, and exit.
8. device according to claim 7, it is characterised in that：The goal task is monitored in the state monitoring module
After being finished, the process exits module and exits before host process, and described device further includes：
Analysis of causes module exits code for the host process acquisition goal task；Code is exited according to described, detects institute
State whether goal task is abnormal exit；If it is abnormal exit to detect the goal task, it is determined that the goal task is different
Often the reason of exiting, and exit code and persistence the reason of the exception exits by described；If detect the goal task not
It is that exception exits, then directly exits code described in preservation.
9. according to the device described in any one of claim 6-8, it is characterised in that：
The daily record memory module, specifically for the log collection thread according to the task parameters to the goal task into
When having monitored journal file generation, the journal file that the shell process obtains is stored to preset third party for row monitoring
Database.
10. a kind of electronic equipment, which is characterized in that including processor, communication interface, memory and communication bus, wherein, processing
Device, communication interface, memory complete mutual communication by communication bus；
Memory, for storing computer program；
Processor during for performing the program stored on memory, realizes any method and steps of claim 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711248381.5A CN108153654B (en) | 2017-12-01 | 2017-12-01 | Log collection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711248381.5A CN108153654B (en) | 2017-12-01 | 2017-12-01 | Log collection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108153654A true CN108153654A (en) | 2018-06-12 |
| CN108153654B CN108153654B (en) | 2021-01-22 |
Family
ID=62465994
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711248381.5A Active CN108153654B (en) | 2017-12-01 | 2017-12-01 | Log collection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108153654B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913847A (en) * | 2020-07-21 | 2020-11-10 | 上海冰鉴信息科技有限公司 | Method and system for acquiring remote task execution progress |
| CN112732475A (en) * | 2021-01-13 | 2021-04-30 | 北京字节跳动网络技术有限公司 | Log collection method and device, electronic equipment and storage medium |
| CN114513502A (en) * | 2022-02-16 | 2022-05-17 | 中银金融科技有限公司 | File processing method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198007A (en) * | 2012-01-06 | 2013-07-10 | 腾讯科技（深圳）有限公司 | Multi-process log output method and system |
| CN105302483A (en) * | 2015-10-14 | 2016-02-03 | 北京乐动卓越科技有限公司 | High-stability log access method and system |
| CN106412061A (en) * | 2016-09-28 | 2017-02-15 | 上海爱数信息技术股份有限公司 | Linux-based log folder remote transmission system |
| CN106598815A (en) * | 2016-12-29 | 2017-04-26 | 广州华多网络科技有限公司 | Real-time asynchronous log collection method and system |
| CN106649071A (en) * | 2016-12-30 | 2017-05-10 | 广州华多网络科技有限公司 | Reporting method and device for internal process journals |
| CN107133086A (en) * | 2016-02-29 | 2017-09-05 | 阿里巴巴集团控股有限公司 | Task processing method, device and system based on distributed system |
| CN107291928A (en) * | 2017-06-29 | 2017-10-24 | 国信优易数据有限公司 | A kind of daily record storage system and method |
-
2017
- 2017-12-01 CN CN201711248381.5A patent/CN108153654B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198007A (en) * | 2012-01-06 | 2013-07-10 | 腾讯科技（深圳）有限公司 | Multi-process log output method and system |
| CN105302483A (en) * | 2015-10-14 | 2016-02-03 | 北京乐动卓越科技有限公司 | High-stability log access method and system |
| CN107133086A (en) * | 2016-02-29 | 2017-09-05 | 阿里巴巴集团控股有限公司 | Task processing method, device and system based on distributed system |
| CN106412061A (en) * | 2016-09-28 | 2017-02-15 | 上海爱数信息技术股份有限公司 | Linux-based log folder remote transmission system |
| CN106598815A (en) * | 2016-12-29 | 2017-04-26 | 广州华多网络科技有限公司 | Real-time asynchronous log collection method and system |
| CN106649071A (en) * | 2016-12-30 | 2017-05-10 | 广州华多网络科技有限公司 | Reporting method and device for internal process journals |
| CN107291928A (en) * | 2017-06-29 | 2017-10-24 | 国信优易数据有限公司 | A kind of daily record storage system and method |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913847A (en) * | 2020-07-21 | 2020-11-10 | 上海冰鉴信息科技有限公司 | Method and system for acquiring remote task execution progress |
| CN111913847B (en) * | 2020-07-21 | 2021-04-27 | 上海冰鉴信息科技有限公司 | Method and system for acquiring remote task execution progress |
| CN112732475A (en) * | 2021-01-13 | 2021-04-30 | 北京字节跳动网络技术有限公司 | Log collection method and device, electronic equipment and storage medium |
| CN114513502A (en) * | 2022-02-16 | 2022-05-17 | 中银金融科技有限公司 | File processing method and device, electronic equipment and storage medium |
| CN114513502B (en) * | 2022-02-16 | 2024-01-26 | 中银金融科技有限公司 | File processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108153654B (en) | 2021-01-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11275641B2 (en) | Automatic correlation of dynamic system events within computing devices | |
| US10585774B2 (en) | Detection of misbehaving components for large scale distributed systems | |
| CN107666410B (en) | Network security analysis system and method | |
| US10048995B1 (en) | Methods and apparatus for improved fault analysis | |
| CN106101130B (en) | A kind of network malicious data detection method, apparatus and system | |
| US9389943B2 (en) | Determining a number of unique incidents in a plurality of incidents for incident processing in a distributed processing system | |
| US20180314576A1 (en) | Automatic application repair by network device agent | |
| US20120284697A1 (en) | Code evaluation based on failure prediction | |
| US11455230B2 (en) | Event specific log file generation | |
| CN110888783A (en) | Monitoring method and device of micro-service system and electronic equipment | |
| US10536505B2 (en) | Intelligent data transmission by network device agent | |
| CN108153654A (en) | A kind of log collecting method and device | |
| CN109358975A (en) | A kind of analysis method, device, electronic equipment and storage medium that software is operating abnormally | |
| US9158639B2 (en) | Method and apparatus for correlating input and output messages of system under test | |
| CN109542786A (en) | Visual testing method and device | |
| US20170244595A1 (en) | Dynamic data collection profile configuration | |
| Lee et al. | Eadro: An end-to-end troubleshooting framework for microservices on multi-source data | |
| US20180293147A1 (en) | Autonomous Information Technology Diagnostic Checks | |
| CN108804914A (en) | A kind of method and device of anomaly data detection | |
| CN103823743A (en) | Monitoring method and monitoring device of software system | |
| CN107483280B (en) | Method and device for monitoring service node device | |
| US20180060987A1 (en) | Identification of abnormal behavior in human activity based on internet of things collected data | |
| CN111130882A (en) | Monitoring system and method of network equipment | |
| WO2016063816A1 (en) | Device and method for detecting abnormality pre-indications in a computer system | |
| US9674060B2 (en) | Dynamic and selective management of integration points using performance metrics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240207 Address after: 200335 room 205, floor 2, building 9, No. 365, Linhong Road, Changning District, Shanghai Patentee after: Shanghai iqiyi New Media Technology Co.,Ltd. Country or region after: China Address before: 10 / F and 11 / F, Hongcheng development building, 2 Haidian North 1st Street, Haidian District, Beijing 100080 Patentee before: BEIJING QIYI CENTURY SCIENCE & TECHNOLOGY Co.,Ltd. Country or region before: China |
|
| TR01 | Transfer of patent right |