CN110162448A - The method and apparatus of log collection - Google Patents
The method and apparatus of log collection Download PDFInfo
- Publication number
- CN110162448A CN110162448A CN201810150258.8A CN201810150258A CN110162448A CN 110162448 A CN110162448 A CN 110162448A CN 201810150258 A CN201810150258 A CN 201810150258A CN 110162448 A CN110162448 A CN 110162448A
- Authority
- CN
- China
- Prior art keywords
- log
- journal file
- file
- collection
- journal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses the method and apparatus of log collection, are related to field of computer technology.One specific embodiment of this method includes: the time interval of the generation time and each journal file according to each log, determining journal file corresponding with each log；Each log is written in journal file corresponding with the log, journal file collection is formed；The log that the journal file is concentrated is stored to target storage position.Log is written in corresponding journal file according to the generation time of log for the embodiment, can be avoided the occurrence of log leakage is adopted；The acquisition state of each log is kept, convenient for understanding the collection site of each log, realizes that breakpoint is continuous and adopts.The technical solution of the embodiment of the present invention can be realized the lossless carrying of system log to off-line analysis system.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of method and apparatus of log collection.
Background technique
With the continuous development of internet, internet has entered the DT epoch from information-technology age, and data have become
The raw material of company's Denver Nuggets, has become a most valuable wealth of company, and the data before data mining, machine learning
Collecting work is landed, and a wherein critically important ring, reliable metadata acquisition tool play very big work to subsequent work
With.
The prior art first stores the log of generation into a.log file when carrying out data acquisition, then will
Log in a.log file is sent to target position.Specifically, when storing the log of generation into a.log file, judgement
Log generate time whether across day, if the log is not stored into a.log file across day；If, will be current across day
A.log file renaming, and an a.log file is created, then current log is stored into newly-built a.log file.
Each newly generated log is stored into the file for being named as a.log in the prior art.Fig. 1 is to save day in the prior art
The flow diagram of will, as shown in Figure 1, in process across day from 2017-09-01:23:59:59 to 2017-09-02:00:00:00
In, if current log file is named as a.log file, for the log of each generation, judge that the generation time of the log is
It is no across day, if the generation time of the log is a certain moment of 2017-09-01, the generation time of the log is not across day, by this
Log is stored into a.log file；If the generation time of the log is a certain moment of 2017-09-02, the generation of the log
When span day, at this point, by current a.log file renamed as a.log.2017-09-01, and create an a.log file,
Then current log is stored into newly-built a.log file.
In realizing process of the present invention, at least there are the following problems in the prior art for inventor's discovery:
1, during file renaming, the newly generated log in part still can be stored in the file of renaming, log
Storage location is chaotic, and log leakage is easy to cause to adopt；
2, log collection scene can not be saved, breakpoint is can not achieve and adopts again.
Summary of the invention
In view of this, the embodiment of the present invention provides a kind of method and apparatus of log collection, according to the generation time of log
Log is written in corresponding journal file, can be avoided the occurrence of log leakage is adopted；The acquisition state of each log is kept,
Convenient for understanding the collection site of each log, realizes that breakpoint is continuous and adopt.The technical solution of the embodiment of the present invention can be realized system day
Lossless carrying of the will to off-line analysis system.
To achieve the above object, according to an aspect of an embodiment of the present invention, a kind of method of log collection is provided, is wrapped
It includes:
According to the generation time of each log and the time interval of each journal file, determining and each log pair
The journal file answered；
Each log is written in journal file corresponding with the log, journal file collection is formed；
The log that the journal file is concentrated is stored to target storage position.
Optionally, the log that the journal file is concentrated is stored to target storage position, further includes:
The acquisition state of each log is saved using vernier persistence thread, forms status file collection；The acquisition state
It include: log mark, and for indicating whether the log acquires the status information of completion.
Optionally, the log that the journal file is concentrated is stored to target storage position, comprising:
The journal file collection is monitored using catalogue listening thread, obtains the log of the presently written journal file collection
Log mark, which is identified and is compared with the status file collection, if log mark is not present in the state text
Part collection, then it is that the journal file concentrates newly-increased log that the log, which identifies corresponding log,；By the newly-increased log storage
To target storage position.
Optionally, the status information includes: state value；If the log collection is completed, the state value is first
Preset value；If the log does not acquire completion, the state value is the second preset value；
The log that the journal file is concentrated is stored to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；It is obtained from journal file concentration
Take log corresponding with log mark；The log is sent to target storage position.
Optionally, the status information further include: acquisition byte number；
The log that the journal file is concentrated is stored to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；It is obtained from journal file concentration
Take log corresponding with log mark；Byte is not acquired according to what the acquisition byte number determined the log；By the log
Byte is not acquired to store to target storage position.
Optionally it is determined that before journal file corresponding with each log, further includes:
Every the first preset time generate a journal file, by the time from increase in the way of determine the journal file when
Between section.
Optionally, the method for the log collection of the embodiment of the present invention further include:
The journal file is monitored using file listening thread；For each journal file, if the journal file exists
Without newly-increased log in second preset time, then the thread to the journal file output journal is closed.
Another aspect according to an embodiment of the present invention provides a kind of device of log collection, comprising:
Output module, it is determining and each according to the generation time of each log and the time interval of each journal file
The corresponding journal file of the log；Each log is written in journal file corresponding with the log, log text is formed
Part collection；
Acquisition module stores the log that the journal file is concentrated to target storage position.
Optionally, the log that the journal file is concentrated is stored to target storage position, the acquisition module is also
For:
The acquisition state of each log is saved using vernier persistence thread, forms status file collection；The acquisition state
It include: log mark, and for indicating whether the log acquires the status information of completion.
Optionally, the acquisition module stores the log that the journal file is concentrated to target storage position, comprising:
The journal file collection is monitored using catalogue listening thread, obtains the log of the presently written journal file collection
Log mark, which is identified and is compared with the status file collection, if log mark is not present in the state text
Part collection, then it is that the journal file concentrates newly-increased log that the log, which identifies corresponding log,；By the newly-increased log storage
To target storage position.
Optionally, the status information includes: state value；If the log collection is completed, the state value is first
Preset value；If the log does not acquire completion, the state value is the second preset value；
The acquisition module stores the log that the journal file is concentrated to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；It is obtained from journal file concentration
Take log corresponding with log mark；The log is stored to target storage position.
Optionally, the status information further include: acquisition byte number；
The acquisition module stores the log that the journal file is concentrated to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；It is obtained from journal file concentration
Take log corresponding with log mark；Byte is not acquired according to what the acquisition byte number determined the log；By the log
Byte is not acquired to store to target storage position.
Optionally it is determined that the output module is also used to before journal file corresponding with each log:
Every the first preset time generate a journal file, by the time from increase in the way of determine the journal file when
Between section.
Optionally, the output module is also used to:
The journal file is monitored using file listening thread；For each journal file, if the journal file exists
Without newly-increased log in second preset time, then the thread to the journal file output journal is closed.
Other side according to an embodiment of the present invention provides a kind of electronic equipment of log collection, comprising:
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processing
The method that device realizes the log collection that first aspect of the embodiment of the present invention provides.
Still another aspect according to an embodiment of the present invention provides a kind of computer-readable medium, is stored thereon with computer
Program realizes the method for the log collection that first aspect of the embodiment of the present invention provides when described program is executed by processor.
One embodiment in foregoing invention have the following advantages that or the utility model has the advantages that
(1) the corresponding time interval of each journal file, is written corresponding day for log according to the generation time of log
In will file, the occurrence of log leakage is adopted can be avoided；
(2) acquisition state of each log is kept, convenient for understanding the collection site of each log, realizes that breakpoint is continuous and adopts；
It (3), can by monitoring the variation of journal file collection and comparing the log listened to and status file collection
Determine newly-increased log, quickly and accurately convenient for being acquired to newly-increased log；
(4) log for not acquiring completion is obtained by the second status indicator, and is stored target storage position, Neng Goushi
The now acquisition to the log for not acquiring completion is realized that breakpoint is continuous and is adopted；
(5) by saving the sub- joint number of acquisition when log collection, aim at a certain day previous time convenient for quickly and accurately determining
Breakpoint location when acquisition is realized that breakpoint is continuous and is adopted；
(6) if the newly-increased log of nothing in the second preset time of a certain journal file, closes and export to the journal file
The thread of log can discharge computing resource and the space of system in time, improve the operational efficiency of system.
Further effect possessed by above-mentioned non-usual optional way adds hereinafter in conjunction with specific embodiment
With explanation.
Detailed description of the invention
Attached drawing for a better understanding of the present invention, does not constitute an undue limitation on the present invention.Wherein:
Fig. 1 is the flow diagram for saving log in the prior art；
Fig. 2 is the schematic diagram of the main flow of the method for log collection according to an embodiment of the present invention；
Fig. 3 is the schematic diagram of the main flow of log output according to an embodiment of the present invention；
Fig. 4 is the schematic diagram of the main modular of the device of log collection according to an embodiment of the present invention；
Fig. 5 is that the embodiment of the present invention can be applied to exemplary system architecture figure therein；
Fig. 6 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present invention
Figure.
Specific embodiment
Below in conjunction with attached drawing, an exemplary embodiment of the present invention will be described, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
It arrives, it can be with various changes and modifications are made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
Sample, for clarity and conciseness, descriptions of well-known functions and structures are omitted from the following description.
Various logs can be generated in the operational process such as on-line system.In the embodiment of the present invention, pass through the day for generating system
Will is sequentially written in corresponding journal file according to the generation time of log, can be avoided log rollback when log collection；It is logical
Acquisition state when saving log collection is crossed, log collection scene can be saved, realizes that breakpoint is continuous and adopts.The embodiment of the present invention can
Avoid the occurrence of that leakage is adopted.
It should be noted that in the absence of conflict, various embodiments of the present invention and each technical characteristic can be mutual
Combination.
Fig. 2 is the schematic diagram of the main flow of the method for log collection according to an embodiment of the present invention, as shown in Fig. 2, day
Will acquisition method include:
Step S101, determining and each according to the generation time of each log and the time interval of each journal file
The corresponding journal file of the log.
It can also include: pre- every first in actual application, before determining journal file corresponding with each log
If the time generates a journal file, the time interval of the journal file is determined in such a way that the time is from increasing.Each log text
Part corresponds to a time interval, is not overlapped between the time interval of any two journal file.
First preset time can be selected and be set according to the actual situation.For example, the first preset time is one month,
The log that the same moon generates is exported into the same journal file, and the log that the different moons generate is exported to different journal files
In.First preset time can be week, for example, the day that 2017-09-01:00:00:00 to 2017-09-07:23:59:59 is generated
Will is exported into a journal file, the log output that 2017-09-08:00:00:00 to 2017-09-14:23:59:59 is generated
Into another journal file.
Fig. 3 is the schematic diagram of the main flow of log output according to an embodiment of the present invention.As shown in figure 3, from 2017-
During 09-01:23:59:59 to 2017-09-02:00:00:00 is across day, if the first preset time is one day, current log text
Part is named as a.2017-09-01.log file, for the log of each generation, judge the log generation time whether across
It, if the generation time of the log is a certain moment of 2017-09-01, the generation time of the log is not across day, by the log
Output is into a.2017-09-01.log file；If the generation time of the log is a certain moment of 2017-09-02, create
A.2017-09-02.log then file stores the log into a.2017-09-02.log file.
During prior art log saves process renaming, the newly generated log in part still can be stored to renaming
File in, log storage location is chaotic, and by taking log shown in fig. 1 saves process as an example, the log that 2017-09-02 is generated is not
It is only stored in a.log file, it is also possible to be stored in a.log.2017-09-01 file.It is produced to acquisition 2017-09-02
Raw log then not only needs to search acquisition from a.log file, it is also necessary to search and adopt from a.log.2017-09-01 file
Collection.This scene is known as rollback.On the one hand, since log storage location is chaotic, leakage is easy to cause to adopt.On the other hand, rollback is grasped
It will be greatly reduced the efficiency of log collection, and influence the reliability of log collection.
In the embodiment of the present invention, the corresponding time interval of each journal file, according to the generation time of log by log
Be written in corresponding journal file, can be avoided log leakage the occurrence of adopting and rolling back action, the efficiency of log collection and
Accuracy is high.
Step S102, each log is written in journal file corresponding with the log, forms journal file collection.
It will be saved in the log write-in journal file of generation, the log of generation, energy then acquired from journal file
It enough carries out log output and log collection distribution, avoids due to the bad shadow that execution journal acquires operation and generates to system
Ring, can flexible choice log collection according to the actual situation time, improve the convenience of log collection.
Step S103, the log that the journal file is concentrated is stored to target storage position.
Target storage position refers to that execution journal acquisition operation is the position for needing the log for generating system to be transported to,
It can be server, being also possible to other can storage medium.For example, when target storage position can be off-line analysis system, it will
The log that current system generates is stored to off-line analysis system, and the log convenient for generating to current system carries out off-line analysis.It adopts
With the log collection method of the embodiment of the present invention, the lossless carrying of system log to off-line analysis system can be realized.
The log that the journal file is concentrated is stored to target storage position, can also include: to be held using vernier
Longization thread saves the acquisition state of each log, forms status file collection；The acquisition state includes: log mark, and
For indicating whether the log acquires the status information of completion.
For a certain log, which is stored to target storage position, can be saved according to following file format
The acquisition state of the log: filename _ acquisition state _ acquisition byte number.Wherein, acquisition state can pass through 0 and 1 mark, 0 mark
Show that log does not acquire completion, 1 mark log collection is completed；When acquiring byte number can be operated using the acquisition of a preceding execution journal
The byte number acquired or breakpoint location of log indicate.
It, can be lasting by the acquisition state scanned with the acquisition state of each log of timing scan in actual application
Change to save to status file and concentrate.Compared with real time scan, timing scan can substantially reduce scan frequency, improve log collection
Efficiency.The acquisition state persistence scanned is saved to status file and is concentrated, can prevent the acquisition state of log from losing.
The acquisition state of each log is kept, (collection site is adopted with log convenient for the collection site that understands each log
The related information of the state of collection, for example, acquisition source, intend collected target storage position and whether acquire completion etc. letter
Breath), it realizes that breakpoint is continuous and adopts.
The log that the journal file is concentrated is stored to target storage position, may include: using catalogue listening thread
Monitor the journal file collection, obtain the log mark of the log of the presently written journal file collection, by log mark with
The status file collection is compared, if log mark is not present in the status file collection, log mark is corresponding
Log is that the journal file concentrates newly-increased log；The newly-increased log is stored to target storage position.In some realities
It applies in example, newly-increased log can be sent to message queue, then the log in message queue is stored to target and stores position
It sets.
By monitoring the variation of journal file collection and comparing the log listened to and status file collection, Neng Gouzhun
Newly-increased log is quickly determined, really convenient for being acquired to newly-increased log.
The status information may include: state value；If the log collection is completed, the state value is first default
Value；If the log does not acquire completion, the state value is the second preset value.For example, indicating log with the first preset value 1
Acquisition is completed, and indicates that log does not acquire completion with the second preset value 0.The log that the journal file is concentrated is stored to target and is deposited
Storage space is set, and may include: to obtain the log that the status file collected state value is the second preset value to identify；From the log text
Part, which is concentrated, obtains log corresponding with log mark；The log is stored to target storage position.In some embodiments, may be used
The log for not acquiring completion is sent to message queue, then the log in message queue is stored to target storage position.
The log for not acquiring completion is obtained by state value, and is stored to target storage position, can be realized to not
The acquisition for the log completed is acquired, realizes that breakpoint is continuous and adopts.
Status information can also include: acquisition byte number.The log that the journal file is concentrated is stored to target and is stored
Position may include: to obtain the log that the status file collected state value is the second preset value to identify；From the journal file
It concentrates and obtains log corresponding with log mark；Byte is not acquired according to what the acquisition byte number determined the log；It should
The byte that do not acquire of log is stored to target storage position.
By saving the sub- joint number of acquisition when log collection, aim at previous acquisition a certain day convenient for quickly and accurately determining
When breakpoint location, realize that breakpoint is continuous and adopt.
The method of the log collection of the embodiment of the present invention can also include: to monitor the log text using file listening thread
Part；For each journal file, if the journal file newly-increased log of nothing in the second preset time, was closed to the day
The thread of will file output journal.Second preset time can carry out selection setting according to practical application scene, such as be set as
24 hours or 2 days etc., the embodiment of the present invention was not specifically limited in this embodiment.
For example, a journal file is generated daily, and with the log that the storage same day generates, the day that wherein 2017-09-01 is generated
Will is delivered in a.2017-09-01.log file.Then, if within 24 hours before current time a.2017-09-01.log
Without newly-increased log in file, the log for showing that 2017-09-01 is generated all is delivered to a.2017-09-01.log file
In, the thread to a.2017-09-01.log file output journal can be closed at this time.
If a certain journal file, without newly-increased log, is closed to the journal file output journal in the second preset time
Thread, computing resource and the space of system can be discharged in time, improve the operational efficiency of system.
If a certain file listening thread cannot achieve closing function, show that this document monitors scene failure.Practical application
In the process, monitoring thread can be set, to monitor whether the file listening thread of failure.When the file for monitoring failure is monitored
When thread, thread can be created, again to monitor journal file.
Another aspect according to an embodiment of the present invention provides a kind of device of log collection.
Fig. 4 is the schematic diagram of the main modular of the device of log collection according to an embodiment of the present invention, as shown in figure 4, day
Will acquisition device 400 include:
Output module 401, it is determining and every according to the generation time of each log and the time interval of each journal file
The corresponding journal file of a log；Each log is written in journal file corresponding with the log, log is formed
File set；
Acquisition module 402 stores the log that the journal file is concentrated to target storage position.
Optionally, the log that the journal file is concentrated is stored to target storage position, the acquisition module is also
For:
The acquisition state of each log is saved using vernier persistence thread, forms status file collection；The acquisition state
It include: log mark, and for indicating whether the log acquires the status information of completion.
Optionally, the acquisition module stores the log that the journal file is concentrated to target storage position, comprising:
The journal file collection is monitored using catalogue listening thread, obtains the log of the presently written journal file collection
Log mark, which is identified and is compared with the status file collection, if log mark is not present in the state text
Part collection, then it is that the journal file concentrates newly-increased log that the log, which identifies corresponding log,；By the newly-increased log storage
To target storage position.
Optionally, the status information includes: state value；If the log collection is completed, the state value is first
Preset value；If the log does not acquire completion, the state value is the second preset value；
The acquisition module stores the log that the journal file is concentrated to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；It is obtained from journal file concentration
Take log corresponding with log mark；The log is stored to target storage position.
Optionally, the status information further include: acquisition byte number；
The acquisition module stores the log that the journal file is concentrated to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；It is obtained from journal file concentration
Take log corresponding with log mark；Byte is not acquired according to what the acquisition byte number determined the log；By the log
Byte is not acquired to store to target storage position.
Optionally it is determined that the output module is also used to before journal file corresponding with each log:
Every the first preset time generate a journal file, by the time from increase in the way of determine the journal file when
Between section.
Optionally, the output module is also used to:
The journal file is monitored using file listening thread；For each journal file, if the journal file exists
Without newly-increased log in second preset time, then the thread to the journal file output journal is closed.
Other side according to an embodiment of the present invention provides a kind of electronic equipment of log collection, comprising:
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processing
The method that device realizes the log collection that first aspect of the embodiment of the present invention provides.
Fig. 5 is shown can be using the exemplary of the device of the method or log collection of the log collection of the embodiment of the present invention
System architecture 500.
As shown in figure 5, system architecture 500 may include terminal device 501,502,503, network 504 and server 505.
Network 504 between terminal device 501,502,503 and server 505 to provide the medium of communication link.Network 504 can be with
Including various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be used terminal device 501,502,503 and be interacted by network 504 with server 505, to receive or send out
Send message etc..Various telecommunication customer end applications, such as the application of shopping class, net can be installed on terminal device 501,502,503
(merely illustrative) such as the application of page browsing device, searching class application, instant messaging tools, mailbox client, social platform softwares.
Terminal device 501,502,503 can be the various electronic equipments with display screen and supported web page browsing, packet
Include but be not limited to smart phone, tablet computer, pocket computer on knee and desktop computer etc..
Server 505 can be to provide the server of various services, such as utilize terminal device 501,502,503 to user
The shopping class website browsed provides the back-stage management server (merely illustrative) supported.Back-stage management server can be to reception
To the data such as information query request analyze etc. processing, and by processing result (such as target push information, product letter
Breath -- merely illustrative) feed back to terminal device.
It should be noted that the method for log collection provided by the embodiment of the present invention is generally executed by server 505, phase
Ying Di, the device of log collection are generally positioned in server 505.
It should be understood that the number of terminal device, network and server in Fig. 5 is only schematical.According to realization need
It wants, can have any number of terminal device, network and server.
Below with reference to Fig. 6, it illustrates the computer systems 600 for the terminal device for being suitable for being used to realize the embodiment of the present invention
Structural schematic diagram.Terminal device shown in Fig. 6 is only an example, function to the embodiment of the present invention and should not use model
Shroud carrys out any restrictions.
As shown in fig. 6, computer system 600 includes central processing unit (CPU) 601, it can be read-only according to being stored in
Program in memory (ROM) 602 or be loaded into the program in random access storage device (RAM) 603 from storage section 608 and
Execute various movements appropriate and processing.In RAM 603, also it is stored with system 600 and operates required various programs and data.
CPU 601, ROM 602 and RAM 603 are connected with each other by bus 604.Input/output (I/O) interface 605 is also connected to always
Line 604.
I/O interface 605 is connected to lower component: the importation 606 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 607 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage section 608 including hard disk etc.；
And the communications portion 609 of the network interface card including LAN card, modem etc..Communications portion 609 via such as because
The network of spy's net executes communication process.Driver 610 is also connected to I/O interface 605 as needed.Detachable media 611, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 610, in order to read from thereon
Computer program be mounted into storage section 608 as needed.
Particularly, disclosed embodiment, the process described above with reference to flow chart may be implemented as counting according to the present invention
Calculation machine software program.For example, embodiment disclosed by the invention includes a kind of computer program product comprising be carried on computer
Computer program on readable medium, the computer program include the program code for method shown in execution flow chart.?
In such embodiment, which can be downloaded and installed from network by communications portion 609, and/or from can
Medium 611 is dismantled to be mounted.When the computer program is executed by central processing unit (CPU) 601, system of the invention is executed
The above-mentioned function of middle restriction.
It should be noted that computer-readable medium shown in the present invention can be computer-readable signal media or meter
Calculation machine readable storage medium storing program for executing either the two any combination.Computer readable storage medium for example can be --- but not
Be limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or any above combination.Meter
The more specific example of calculation machine readable storage medium storing program for executing can include but is not limited to: have the electrical connection, just of one or more conducting wires
Taking formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In the present invention, computer readable storage medium can be it is any include or storage journey
The tangible medium of sequence, the program can be commanded execution system, device or device use or in connection.And at this
In invention, computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By the use of instruction execution system, device or device or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to: wireless, electric wire, optical cable, RF etc. are above-mentioned
Any appropriate combination.
Flow chart and block diagram in attached drawing are illustrated according to the system of various embodiments of the invention, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of above-mentioned module, program segment or code include one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants
It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule
The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction
It closes to realize.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part is realized.Described module also can be set in the processor, for example, can be described as: a kind of processor packet
It includes: output module, according to the generation time of each log and the time interval of each journal file, determining and each day
The corresponding journal file of will；Each log is delivered in journal file corresponding with the log, journal file collection is formed；
The log that the journal file is concentrated is sent to target storage position by acquisition module.Wherein, the title of these modules is at certain
In the case of do not constitute restriction to the module itself, for example, output module is also described as " by the journal file collection
In log be sent to the module of target storage position ".
As on the other hand, the present invention also provides a kind of computer-readable medium, which be can be
Included in equipment described in above-described embodiment；It is also possible to individualism, and without in the supplying equipment.Above-mentioned calculating
Machine readable medium carries one or more program, when said one or multiple programs are executed by the equipment, makes
The time interval that the equipment includes: the generation time and each journal file according to each log is obtained, it is determining and each described
The corresponding journal file of log；Each log is written in journal file corresponding with the log, journal file collection is formed；
The log that the journal file is concentrated is stored to target storage position.
Technical solution according to an embodiment of the present invention, have the following advantages that or the utility model has the advantages that
(1) the corresponding time interval of each journal file, is written corresponding day for log according to the generation time of log
In will file, the occurrence of log leakage is adopted can be avoided；
(2) acquisition state of each log is kept, convenient for understanding the collection site of each log, realizes that breakpoint is continuous and adopts；
It (3), can by monitoring the variation of journal file collection and comparing the log listened to and status file collection
Determine newly-increased log, quickly and accurately convenient for being acquired to newly-increased log；
(4) log for not acquiring completion is obtained by the second status indicator, and is stored to target storage position, it can
It realizes the acquisition to the log for not acquiring completion, realizes that breakpoint is continuous and adopt；
(5) by saving the sub- joint number of acquisition when log collection, aim at a certain day previous time convenient for quickly and accurately determining
Breakpoint location when acquisition is realized that breakpoint is continuous and is adopted；
(6) if the newly-increased log of nothing in the second preset time of a certain journal file, closes and export to the journal file
The thread of log can discharge computing resource and the space of system in time, improve the operational efficiency of system.
Above-mentioned specific embodiment, does not constitute a limitation on the scope of protection of the present invention.Those skilled in the art should be bright
It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and substitution can occur.It is any
Made modifications, equivalent substitutions and improvements etc. within the spirit and principles in the present invention, should be included in the scope of the present invention
Within.
Claims (16)
1. a kind of method of log collection characterized by comprising
According to the generation time of each log and the time interval of each journal file, determination is corresponding with each log
Journal file；
Each log is written in journal file corresponding with the log, journal file collection is formed；
The log that the journal file is concentrated is stored to target storage position.
2. the method as described in claim 1, which is characterized in that store the log that the journal file is concentrated to target and store
After position, further includes:
The acquisition state of each log is saved using vernier persistence thread, forms status file collection；The acquisition state includes:
Log mark, and for indicating whether the log acquires the status information of completion.
3. method according to claim 2, which is characterized in that store the log that the journal file is concentrated to target and store
Position, comprising:
The journal file collection is monitored using catalogue listening thread, obtains the log of the log of the presently written journal file collection
The log is identified and is compared with the status file collection by mark, if log mark is not present in the status file collection,
It is that the journal file concentrates newly-increased log that then the log, which identifies corresponding log,；The newly-increased log is stored to target
Storage location.
4. method according to claim 2, which is characterized in that the status information includes: state value；If the log collection
It completes, then the state value is the first preset value；If the log does not acquire completion, the state value is the second preset value；
The log that the journal file is concentrated is stored to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；From the journal file concentrate obtain with
The log identifies corresponding log；The log is stored to target storage position.
5. method as claimed in claim 4, which is characterized in that the status information further include: acquisition byte number；
The log that the journal file is concentrated is stored to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；From the journal file concentrate obtain with
The log identifies corresponding log；Byte is not acquired according to what the acquisition byte number determined the log；By not adopting for the log
Collection byte is stored to target storage position.
6. the method as described in claim 1, which is characterized in that before determining journal file corresponding with each log,
Further include:
A journal file is generated every the first preset time, the time zone of the journal file is determined in such a way that the time is from increasing
Between.
7. the method as described in claim 1, which is characterized in that further include:
The journal file is monitored using file listening thread；For each journal file, if the journal file is second
Without newly-increased log in preset time, then the thread to the journal file output journal is closed.
8. a kind of device of log collection characterized by comprising
Output module, it is determining and each described according to the generation time of each log and the time interval of each journal file
The corresponding journal file of log；Each log is written in journal file corresponding with the log, journal file collection is formed；
Acquisition module stores the log that the journal file is concentrated to target storage position.
9. device as claimed in claim 8, which is characterized in that store the log that the journal file is concentrated to target and store
After position, the acquisition module is also used to:
The acquisition state of each log is saved using vernier persistence thread, forms status file collection；The acquisition state includes:
Log mark, and for indicating whether the log acquires the status information of completion.
10. device as claimed in claim 9, which is characterized in that the log that the acquisition module concentrates the journal file
It stores to target storage position, comprising:
The journal file collection is monitored using catalogue listening thread, obtains the log of the log of the presently written journal file collection
The log is identified and is compared with the status file collection by mark, if log mark is not present in the status file collection,
It is that the journal file concentrates newly-increased log that then the log, which identifies corresponding log,；The newly-increased log is stored to target
Storage location.
11. device as claimed in claim 9, which is characterized in that the status information includes: state value；If the log is adopted
Collection is completed, then the state value is the first preset value；If the log does not acquire completion, the state value is second default
Value；
The acquisition module stores the log that the journal file is concentrated to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；From the journal file concentrate obtain with
The log identifies corresponding log；The log is stored to target storage position.
12. device as claimed in claim 11, which is characterized in that the status information further include: acquisition byte number；
The acquisition module stores the log that the journal file is concentrated to target storage position, comprising:
The log that the status file collected state value is the second preset value is obtained to identify；From the journal file concentrate obtain with
The log identifies corresponding log；Byte is not acquired according to what the acquisition byte number determined the log；By not adopting for the log
Collection byte is stored to target storage position.
13. device as claimed in claim 8, which is characterized in that before determining journal file corresponding with each log,
The output module is also used to:
A journal file is generated every the first preset time, the time zone of the journal file is determined in such a way that the time is from increasing
Between.
14. device as claimed in claim 8, which is characterized in that the output module is also used to:
The journal file is monitored using file listening thread；For each journal file, if the journal file is second
Without newly-increased log in preset time, then the thread to the journal file output journal is closed.
15. a kind of electronic equipment of log collection characterized by comprising
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
The now method as described in any in claim 1-7.
16. a kind of computer-readable medium, is stored thereon with computer program, which is characterized in that described program is held by processor
The method as described in any in claim 1-7 is realized when row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810150258.8A CN110162448A (en) | 2018-02-13 | 2018-02-13 | The method and apparatus of log collection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810150258.8A CN110162448A (en) | 2018-02-13 | 2018-02-13 | The method and apparatus of log collection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110162448A true CN110162448A (en) | 2019-08-23 |
Family
ID=67635386
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810150258.8A Pending CN110162448A (en) | 2018-02-13 | 2018-02-13 | The method and apparatus of log collection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110162448A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625419A (en) * | 2020-05-15 | 2020-09-04 | 浪潮电子信息产业股份有限公司 | Log acquisition method, system, equipment and computer readable storage medium |
| CN112905106A (en) * | 2019-12-04 | 2021-06-04 | 贵州白山云科技股份有限公司 | Data processing method, device, system, equipment and storage medium |
| CN113051121A (en) * | 2019-12-26 | 2021-06-29 | 百度在线网络技术（北京）有限公司 | Log information retrieval method and device, electronic equipment and medium |
| CN113760683A (en) * | 2021-08-31 | 2021-12-07 | 银清科技有限公司 | Log collection method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101571827A (en) * | 2008-04-30 | 2009-11-04 | 国际商业机器公司 | Method for saving logs and log system |
| CN104486107A (en) * | 2014-12-05 | 2015-04-01 | 曙光信息产业(北京)有限公司 | Log collection device and method |
| CN106095959A (en) * | 2016-06-16 | 2016-11-09 | 北京中电普华信息技术有限公司 | A kind of collecting method, Apparatus and system |
| CN106991177A (en) * | 2017-04-06 | 2017-07-28 | 广东亿迅科技有限公司 | The nondestructive collection method and its system of real-time logs under a kind of distributed environment |
| CN107341258A (en) * | 2017-07-12 | 2017-11-10 | 中国工商银行股份有限公司 | A kind of log data acquisition method and system |
-
2018
- 2018-02-13 CN CN201810150258.8A patent/CN110162448A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101571827A (en) * | 2008-04-30 | 2009-11-04 | 国际商业机器公司 | Method for saving logs and log system |
| CN104486107A (en) * | 2014-12-05 | 2015-04-01 | 曙光信息产业(北京)有限公司 | Log collection device and method |
| CN106095959A (en) * | 2016-06-16 | 2016-11-09 | 北京中电普华信息技术有限公司 | A kind of collecting method, Apparatus and system |
| CN106991177A (en) * | 2017-04-06 | 2017-07-28 | 广东亿迅科技有限公司 | The nondestructive collection method and its system of real-time logs under a kind of distributed environment |
| CN107341258A (en) * | 2017-07-12 | 2017-11-10 | 中国工商银行股份有限公司 | A kind of log data acquisition method and system |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112905106A (en) * | 2019-12-04 | 2021-06-04 | 贵州白山云科技股份有限公司 | Data processing method, device, system, equipment and storage medium |
| CN112905106B (en) * | 2019-12-04 | 2023-04-18 | 贵州白山云科技股份有限公司 | Data processing method, device, system, equipment and storage medium |
| CN113051121A (en) * | 2019-12-26 | 2021-06-29 | 百度在线网络技术（北京）有限公司 | Log information retrieval method and device, electronic equipment and medium |
| CN111625419A (en) * | 2020-05-15 | 2020-09-04 | 浪潮电子信息产业股份有限公司 | Log acquisition method, system, equipment and computer readable storage medium |
| CN111625419B (en) * | 2020-05-15 | 2023-03-31 | 浪潮电子信息产业股份有限公司 | Log acquisition method, system, equipment and computer readable storage medium |
| CN113760683A (en) * | 2021-08-31 | 2021-12-07 | 银清科技有限公司 | Log collection method and device |
| CN113760683B (en) * | 2021-08-31 | 2024-02-20 | 银清科技有限公司 | Log acquisition method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109902105B (en) | Data query system, method, device and storage medium for micro-service architecture | |
| CN110162448A (en) | The method and apparatus of log collection | |
| CN107609890A (en) | A kind of method and apparatus of order tracking | |
| CN109660397A (en) | For acquiring system, the method and apparatus of log | |
| CN108256070A (en) | For generating the method and apparatus of information | |
| CN110245011A (en) | A kind of method for scheduling task and device | |
| CN109766082A (en) | The method and apparatus that the application program page jumps | |
| CN109871311A (en) | A kind of method and apparatus for recommending test case | |
| CN110858172A (en) | Automatic test code generation method and device | |
| CN110297659A (en) | Algorithm model disposes loading method and device | |
| CN110019214A (en) | The method and apparatus that data split result is verified | |
| CN109388551A (en) | There are the method for loophole probability, leak detection method, relevant apparatus for prediction code | |
| CN109002925A (en) | Traffic prediction method and apparatus | |
| CN113360376A (en) | Buried point testing method and device | |
| CN110209662A (en) | A kind of method and apparatus of automation load data | |
| CN110019367A (en) | A kind of method and apparatus of statistical data feature | |
| CN110019539A (en) | A kind of method and apparatus that the data of data warehouse are synchronous | |
| CN109002385A (en) | Method for testing pressure and device for data flow system | |
| CN109684172A (en) | Log method for pushing, system, equipment and storage medium based on access frequency | |
| CN111444077A (en) | Method and device for generating flow node test data | |
| CN109992494A (en) | A kind of automatic test execution method and apparatus | |
| CN109960212A (en) | Task sending method and device | |
| CN109814957A (en) | A kind of label addition method and device for IOS system | |
| CN109976919A (en) | A kind of transmission method and device of message request | |
| CN110334248A (en) | A kind of system configuration information treating method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |