CN106569936A - Method and system for acquiring scrolling log in real time - Google Patents
Method and system for acquiring scrolling log in real time Download PDFInfo
- Publication number
- CN106569936A CN106569936A CN201610851865.8A CN201610851865A CN106569936A CN 106569936 A CN106569936 A CN 106569936A CN 201610851865 A CN201610851865 A CN 201610851865A CN 106569936 A CN106569936 A CN 106569936A
- Authority
- CN
- China
- Prior art keywords
- file
- filepointer
- filelength
- event
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a method and system for acquiring a scrolling log in real time, and relates to the technical field of computers, wherein the method comprises the following steps of: monitoring whether a log file directory has pre-set events or not, wherein the pre-set events include a file new-building event and a file modification event; and, if the pre-set events occur, obtaining the current reading position filepointer of a file pointer in a file and the file length filelength of the file, reading a newly increased file content according to the filepointer of the file pointer in the file and the filelength of the file, and sending the read file content to a data serialization message source Avro Source of a log collection system Flume. By means of the method and the system provided by the invention, influence of new building of files, continuous modification of file contents and renaming of temporary files in the log file directory on log acquisition can be effectively eliminated; real-time acquisition of the scrolling log is realized; and thus, the log collection reliability is improved.
Description
Technical field
The invention belongs to field of computer technology, more particularly to a kind of method and system of Real-time Collection rolling daily record.
Background technology
Flume is the most frequently used log collection technology in prior art, and Flume is one distributed, reliable efficient day
Will collection system, it allows User Defined data transfer model, and autgmentability is very strong, and with stronger fault-tolerant and recovery machine
System.However, Flume still just there are certain defect in actual use：Flume generally uses Spooling
Directory Source or Exec Source carry out collector journal, and wherein Spooling Directory Source are not supported
The content of the journal file to collecting is added, and when there is new log content to produce, can only pass through newly-built journal file
Mode is collecting new log content, if adopt roll in real time daily record to gather in this way, it will produce under Log Directory
The very many small documents of life, therefore Spooling Directory Source are not suitable for the Real-time Collection of rolling daily record；And
Although Exec Source can gather data in real time, which is usually the content that journal file is obtained from tail orders,
Tail reading of content can be carried out to fixed journal file, when journal file was carried out rolling write or occurred interim according to the time
During file renaming, it will collection failure occur, the situation for causing daily record to be lost, the reliability of its log collection are relatively low.
The content of the invention
The purpose of the embodiment of the present invention is to provide the method and system that a kind of Real-time Collection rolls daily record, it is intended in solution
Typically adopt Spooling Directory Source or Exec Source come collector journal in stating Flume, which does not apply to
In the relatively low problem of the Real-time Collection or collection reliability that roll daily record.
The embodiment of the present invention is achieved in that a kind of method that Real-time Collection rolls daily record, including：
Whether monitoring logfile directory there is predeterminable event, and the predeterminable event includes that the newly-built event of file and file are repaiied
Change event；
If generation predeterminable event, obtain the current read position filepointer of file pointer in this document and be somebody's turn to do
The file size filelength of file, according to the filepointer and this document of file pointer in the file
Filelength reads newly-increased file content, and the file content for reading is sent to the number of result collection system Flume
According to the message source Avro Source of serializing.
On the basis of above-mentioned technical proposal, if the generation predeterminable event, in acquisition this document, file pointer works as
The file size filelength of front reading position filepointer and this document, according to file pointer in the file
The filelength of filepointer and this document reads newly-increased file content, and by the file content for reading send to
The message source Avro Source of the Data Serialization of result collection system are specifically included：
If generation predeterminable event, the filename of the file is obtained, and the file is obtained according to the filename
Device number, file index number and filelength；
Whether there is the cache information of the file in judging the caching of result collection system；
If there is no the cache information of the file, the filepointer of file pointer in file is set to into 0,
File content is read by row in the range of [0, filelength], and the content for reading is sent to the Avro of result collection system
Source；
If there is the cache information of the file, according to the device number and the file index number from the caching
The filepointer of file pointer in reading this document, and the filepointer according to the file pointer, the file
The event type of filelength and the predeterminable event reads newly-increased file content, and the file content for reading is sent
To the Avro Source of result collection system.
It is on the basis of above-mentioned technical proposal, if the event type of the predeterminable event is the newly-built event of file, described
The event type of filepointer, the filelength of the file and the predeterminable event according to the file pointer
Newly-increased file content is read, the file content for reading is sent to the Avro Source of result collection system and is specifically included：
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, by row reading in the range of [filepointer, filelength]
File content is taken, and the file content for reading is sent to the Avro Source of result collection system, while deleting this document
Cache information；
If filepointer is more than or equal to filelength, the cache information of this document is directly removed.
On the basis of above-mentioned technical proposal, if the event type of the predeterminable event is file modification event, basis
The event type of the filepointer of the file pointer, the filelength of the file and the predeterminable event reads
Newly-increased file content, the file content for reading is sent to the Avro Source of result collection system and is specifically included：
Directly file content is read by row in the range of [filepointer, filelength], and by the file for reading
Content is sent to the Avro Source of result collection system.
On the basis of above-mentioned technical proposal, if also including after the cache information that there is the file：
The cache-time of the cache information is read, the difference of the cache-time and current time is calculated；
Judge the difference whether more than preset time threshold；
If being more than preset time threshold, the cache information is removed.
The another object of the embodiment of the present invention is to provide the system that a kind of Real-time Collection rolls daily record, including：
Event monitoring module, for monitoring whether logfile directory occurs predeterminable event, the predeterminable event includes text
The newly-built event of part and file modification event；
Log collection module, if for there is predeterminable event, obtaining the current read position of file pointer in this document
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content for reading is sent to result collection system
The message source Avro Source of the Data Serialization of Flume.
On the basis of above-mentioned technical proposal, log collection module is specifically included：
Fileinfo acquiring unit, if for there is predeterminable event, the filename of the file is obtained, and according to described
Filename obtains device number, file index number and the filelength of the file；
Caching judging unit, for judging the cache information in the caching of result collection system with the presence or absence of the file；
First log collection unit, if for the cache information that there is no the file, by file pointer in file
Filepointer is set to 0, reads file content by row, and the content for reading is sent out in the range of [0, filelength]
Deliver to the Avro Source of result collection system；
Second daily record collector unit, if for the cache information that there is the file, according to the device number and described
File index number reads the filepointer of file pointer in this document from the caching, and according to the file pointer
The event type of filepointer, the filelength of the file and the predeterminable event reads newly-increased file content,
The file content for reading is sent to the Avro Source of result collection system.
It is on the basis of above-mentioned technical proposal, if the event type of the predeterminable event is the newly-built event of file, described
Second daily record collector unit has to be used for：
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, by row reading in the range of [filepointer, filelength]
File content is taken, and the file content for reading is sent to the Avro Source of result collection system, while deleting this document
Cache information；
If filepointer is more than or equal to filelength, the cache information of this document is directly removed.
It is on the basis of above-mentioned technical proposal, if the event type of the predeterminable event is file modification event, described
Second daily record collector unit specifically for：
Directly file content is read by row in the range of [filepointer, filelength], and by the file for reading
Content is sent to the Avro Source of result collection system.
On the basis of above-mentioned technical proposal, the log collection module also includes：
Cache cleaner unit, for reading the cache-time of the cache information, calculate the cache-time with it is current when
Between difference；Judge the difference whether more than preset time threshold；If being more than preset time threshold, the caching letter is removed
Breath.
The method and system for implementing a kind of Real-time Collection rolling daily record provided in an embodiment of the present invention have following beneficial effect
Really：
By monitoring whether logfile directory occurs predeterminable event, the predeterminable event includes file to the embodiment of the present invention
Newly-built event and file modification event；If generation predeterminable event, the current read position of file pointer in this document is obtained
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content for reading is sent to result collection system
The message source Avro Source of the Data Serialization of Flume such that it is able to effectively eliminate new file wound under logfile directory
Build, file content is constantly changed and impact of the temporary file renaming situation to log collection, can realize rolling the reality of daily record
When gather, improve log collection reliability.
Description of the drawings
Fig. 1 is that method that a kind of Real-time Collection provided in an embodiment of the present invention rolls daily record implements flow chart；
Fig. 2 is that a kind of Real-time Collection provided in an embodiment of the present invention rolls implementing for step S102 in the method for daily record
Flow chart；
Fig. 3 is the schematic block diagram of the system that a kind of Real-time Collection provided in an embodiment of the present invention rolls daily record；
Fig. 4 is the signal that a kind of Real-time Collection provided in an embodiment of the present invention rolls log collection module in the system of daily record
Property block diagram.
Specific embodiment
In order that the objects, technical solutions and advantages of the present invention become more apparent, it is below in conjunction with drawings and Examples, right
The present invention is further elaborated.It should be appreciated that specific embodiment described herein is only to explain the present invention, and
It is not used in the restriction present invention.
Fig. 1 is that method that a kind of Real-time Collection provided in an embodiment of the present invention rolls daily record implements flow chart.Ginseng
As shown in Figure 1, the method that a kind of Real-time Collection that the present embodiment is provided rolls daily record, may comprise steps of：
In S101, whether monitoring logfile directory there is predeterminable event, and the predeterminable event includes the newly-built thing of file
Part and file modification event.
In the present embodiment, step S101 is specifically included：Monitoring service instance objects watcher are created, and for institute
Daily record text catalogue addition watcher is stated, is registration monitoring event by predeterminable event, it is then described by watcher monitoring
Whether logfile directory there is the predeterminable event.
In S102, the current read position of file pointer in this document if there is predeterminable event, is obtained
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content for reading is sent to result collection system
The message source Avro Source of the Data Serialization of Flume.
Fig. 2 shows the flow process that implements of step S102, shown in Figure 2, and step S102 can specifically include following
Step：
In S201, if there is predeterminable event, the filename of the file is obtained, and institute is obtained according to the filename
State device number, file index number and the filelength of file.
In the present embodiment, when there is predeterminable event, monitor event object can send signal, and be added to
In the queue of watcher, the monitoring thing for sending signal in the queue that result collection system can be monitored from watcher, is found
Part object, and then the filename of the monitor event object respective file is got, the text is obtained according to the filename then
The device number of part, file index number and filelength.
In S202, in judging the caching of result collection system, whether there is the cache information of the file.
In the present embodiment, searched whether in the caching of the result collection system by the filename of the file
The cache information of this document, if finding the cache information matched with file name, illustrates the caching of result collection system
In have the cache information of this document, conversely, there is no the cache information of this document in the caching of result collection system.
In S203, if there is no the cache information of the file, the filepointer of file pointer in file is set
0 is set to, and file content is read by row in the range of [0, filelength], and the content for reading is sent to log collection system
The Avro Source of system.
In the present embodiment, if there is no the cache information of the file in the caching of result collection system, illustrate this
Log content in file is newly-increased or amended content, and filepointer is set to 0 now, [0,
Filelength] in the range of the full content of this document is read by row, and the content for reading is sent to result collection system
Avro Source, while the read access time of the filepointer and cache information of more new file Chinese part pointer, and with
The device number and file index number of file as file pointer in the file after unique mark buffer update filepointer with
And the read access time of cache information.
In S204, if there is the cache information of the file, according to the device number and the file index number from
In the caching read this document in file pointer filepointer, and the filepointer according to the file pointer,
The event type of the filelength of the file and the predeterminable event reads newly-increased file content, by the text for reading
Part content is sent to the Avro Source of result collection system.
In one embodiment, it is if the event type of the predeterminable event is the newly-built event of file, described according to institute
The event type for stating the filepointer, the filelength of the file and the predeterminable event of file pointer reads newly
The file content of increasing, the file content for reading is sent to the Avro Source of result collection system and is specifically included：
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, by row reading in the range of [filepointer, filelength]
File content is taken, and the file content for reading is sent to the Avro Source of result collection system, while deleting this document
Cache information；
If filepointer is more than or equal to filelength, the cache information of this document is directly removed.
In the present embodiment, if the predeterminable event is the newly-built event of file, and in caching, there is the caching of this document
Information, then illustrate that the newly-built event of this document is triggered due to file renaming, if now filepointer is more than or equal to
Filelength, then directly remove the cache information of this document；If conversely, filepointer be less than filelength,
File content is read by row in the range of [filepointer, filelength], and the file content for reading is sent to daily record
The Avro Source of collection system, while deleting the cache information of this document.
In another specific embodiment, if the event type of the predeterminable event is file modification event, according to described
The event type of the filepointer of file pointer, the filelength of the file and the predeterminable event reads newly-increased
File content, the file content for reading is sent to the Avro Source of result collection system and is specifically included：
Directly file content is read by row in the range of [filepointer, filelength], and by the file for reading
Content is sent to the Avro Source of result collection system.
In the present embodiment, if the predeterminable event is file modification event, and in caching, there is the caching of this document
Information, then directly reading file content by row in the range of [filepointer, filelength], and by the text for reading
Part content is sent to the Avro Source of result collection system, and preserves the cache information of this document, while more new file is Chinese
The read access time of the filepointer and cache information of part pointer, and using the device number and file index of file as only
The filepointer and the read access time of cache information of file pointer in file after one mark buffer update.
Further, in step S204, if also including after the cache information that there is the file：
The cache-time of the cache information is read, the difference of the cache-time and current time is calculated；
Judge the difference whether more than preset time threshold；
If being more than preset time threshold, the cache information is removed.
In the present embodiment, due in real time according to the difference of the cache-time and current time of cache information to cache information
Cleared up such that it is able to which cache information cleaning caused by avoiding file renaming event does not occur because of logfile directory is obstructed
Problem occurs.
Above as can be seen that a kind of Real-time Collection that the present embodiment is provided rolls the method for daily record by monitoring journal file
Whether catalogue there is predeterminable event, and the predeterminable event includes the newly-built event of file and file modification event；If there is default thing
Part, then obtain the current read position filepointer and the file size of this document of file pointer in this document
Filelength, reads what is increased newly according to the filelength of the filepointer and this document of file pointer in the file
File content, and the file content for reading is sent the message source Avro of the Data Serialization to result collection system Flume
Source such that it is able to effectively eliminate new file establishment, file content under logfile directory and constantly change and temporary file
Impact of the renaming situation to log collection, can realize rolling the Real-time Collection of daily record, improve the reliability of log collection.
Fig. 3 is the schematic block diagram of the system that a kind of Real-time Collection provided in an embodiment of the present invention rolls daily record, the system
Method for running embodiment illustrated in fig. 1 offer.Part related to the present embodiment is illustrate only for convenience of description.
It is shown in Figure 3, the system that a kind of Real-time Collection that the present embodiment is provided rolls daily record, including：
Event monitoring module 1, for monitoring whether logfile directory occurs predeterminable event, the predeterminable event includes text
The newly-built event of part and file modification event；
Log collection module 2, if for there is predeterminable event, obtaining the current read position of file pointer in this document
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content for reading is sent to result collection system
The message source Avro Source of the Data Serialization of Flume.
Optionally, shown in Figure 4, the log collection module 2 is specifically included：
Fileinfo acquiring unit 21, if for there is predeterminable event, the filename of the file is obtained, and according to institute
State device number, file index number and filelength that filename obtains the file；
Caching judging unit 22, for judging that the caching in the caching of result collection system with the presence or absence of the file is believed
Breath；
First log collection unit 23, if for the cache information that there is no the file, by file pointer in file
Filepointer be set to 0, read file content by row in the range of [0, filelength], and by the content for reading
Send to the Avro Source of result collection system；
Second daily record collector unit 24, if for the cache information that there is the file, according to the device number and institute
The filepointer that file index number reads file pointer in this document from the caching is stated, and according to the file pointer
The event type of filepointer, the filelength of the file and the predeterminable event read in newly-increased file
Hold, the file content for reading is sent to the Avro Source of result collection system.
Optionally, if the event type of the predeterminable event is the newly-built event of file, the second daily record collector unit
24 have be used for：
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, by row reading in the range of [filepointer, filelength]
File content is taken, and the file content for reading is sent to the Avro Source of result collection system, while deleting this document
Cache information；
If filepointer is more than or equal to filelength, the cache information of this document is directly removed.
Optionally, if the event type of the predeterminable event is file modification event, the second daily record collector unit
24 specifically for：
Directly file content is read by row in the range of [filepointer, filelength], and by the file for reading
Content is sent to the Avro Source of result collection system, while deleting the cache information of this document.
Optionally, the log collection module also includes：
Cache cleaner unit 25, for reading the cache-time of the cache information, calculate the cache-time with it is current
The difference of time；Judge the difference whether more than preset time threshold；If being more than preset time threshold, the caching is removed
Information.
It should be noted that the modules in said system provided in an embodiment of the present invention, due to the inventive method
Embodiment is based on same design, and the technique effect which brings is identical with the inventive method embodiment, and particular content can be found in this
Narration in bright embodiment of the method, here is omitted.
Therefore, it can be seen that the system that a kind of Real-time Collection provided in an embodiment of the present invention rolls daily record equally can be effective
Elimination logfile directory under new file create, file content is constantly changed and temporary file renaming situation is to log collection
Impact, can realize roll daily record Real-time Collection, improve log collection reliability.
Presently preferred embodiments of the present invention is the foregoing is only, not to limit the present invention, all essences in the present invention
Any modification, equivalent and improvement made within god and principle etc., should be included within the scope of the present invention.
Claims (10)
1. a kind of method that Real-time Collection rolls daily record, it is characterised in that include：
Whether monitoring logfile directory there is predeterminable event, and the predeterminable event includes the newly-built event of file and file modification thing
Part；
If generation predeterminable event, the current read position filepointer and this document of file pointer in this document are obtained
File size filelength, according to the filelength of the filepointer and this document of file pointer in the file
Newly-increased file content is read, and the file content for reading is sent into the Data Serialization to result collection system Flume
Message source Avro Source.
2. the method that Real-time Collection as claimed in claim 1 rolls daily record, it is characterised in that if the generation predeterminable event,
The current read position filepointer and the file size of this document of file pointer in this document are obtained then
Filelength, reads what is increased newly according to the filelength of the filepointer and this document of file pointer in the file
File content, and the file content for reading is sent the message source Avro of the Data Serialization to result collection system
Source is specifically included：
If generation predeterminable event, the filename of the file is obtained, and the equipment of the file is obtained according to the filename
Number, file index number and filelength；
Whether there is the cache information of the file in judging the caching of result collection system；
If there is no the cache information of the file, the filepointer of file pointer in file is set to into 0, [0,
Filelength] in the range of read file content by row, and the content for reading is sent to the Avro of result collection system
Source；
If there is the cache information of the file, read from the caching according to the device number and the file index number
The filepointer of file pointer in this document, and the filepointer according to the file pointer, the file
The event type of filelength and the predeterminable event reads newly-increased file content, and the file content for reading is sent
To the Avro Source of result collection system.
3. the method that Real-time Collection as claimed in claim 2 rolls daily record, it is characterised in that if the event of the predeterminable event
Type is the newly-built event of file, then the filepointer according to the file pointer, the file filelength with
And the event type of the predeterminable event reads newly-increased file content, and the file content for reading is sent to log collection system
The Avro Source of system are specifically included：
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, text is read by row in the range of [filepointer, filelength]
Part content, and the file content for reading is sent to the Avro Source of result collection system, while deleting the slow of this document
Deposit information；
If filepointer is more than or equal to filelength, the cache information of this document is directly removed.
4. the method that Real-time Collection as claimed in claim 2 rolls daily record, it is characterised in that if the event of the predeterminable event
Type is file modification event, then filepointer, the filelength of the file according to the file pointer and institute
The event type for stating predeterminable event reads newly-increased file content, and the file content for reading is sent to result collection system
Avro Source are specifically included：
Directly file content is read by row in the range of [filepointer, filelength], and by the file content for reading
Send to the Avro Source of result collection system.
5. Real-time Collection as claimed in claim 2 rolls daily record method, it is characterised in that if described have the file
Also include after cache information：
The cache-time of the cache information is read, the difference of the cache-time and current time is calculated；
Judge the difference whether more than preset time threshold；
If being more than preset time threshold, the cache information is removed.
6. the system that a kind of Real-time Collection rolls daily record, it is characterised in that include：
Event monitoring module, for monitoring whether logfile directory occurs predeterminable event, the predeterminable event includes that file is new
Build event and file modification event；
Log collection module, if for there is predeterminable event, obtaining the current read position of file pointer in this document
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content for reading is sent to result collection system
The message source Avro Source of the Data Serialization of Flume.
7. the system that Real-time Collection as claimed in claim 6 rolls daily record, it is characterised in that log collection module is specifically wrapped
Include：
Fileinfo acquiring unit, if for there is predeterminable event, the filename of the file is obtained, and according to the file
Name obtains device number, file index number and the filelength of the file；
Caching judging unit, for judging the cache information in the caching of result collection system with the presence or absence of the file；
First log collection unit, if for the cache information that there is no the file, by file pointer in file
Filepointer is set to 0, reads file content by row, and the content for reading is sent out in the range of [0, filelength]
Deliver to the Avro Source of result collection system；
Second daily record collector unit, if for the cache information that there is the file, according to the device number and the file
Call number reads the filepointer of file pointer in this document from the caching, and according to the file pointer
The event type of filepointer, the filelength of the file and the predeterminable event reads newly-increased file content,
The file content for reading is sent to the AvroSource of result collection system.
8. the system that Real-time Collection as claimed in claim 7 rolls daily record, it is characterised in that if the event of the predeterminable event
Type is the newly-built event of file, then the second daily record collector unit has and is used for：
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, text is read by row in the range of [filepointer, filelength]
Part content, and the file content for reading is sent to the Avro Source of result collection system, while deleting the slow of this document
Deposit information；
If filepointer is more than or equal to filelength, the cache information of this document is directly removed.
9. the system that Real-time Collection as claimed in claim 7 rolls daily record, it is characterised in that if the event of the predeterminable event
Type be file modification event, then the second daily record collector unit specifically for：
Directly file content is read by row in the range of [filepointer, filelength], and by the file content for reading
Send to the Avro Source of result collection system.
10. the system that Real-time Collection as claimed in claim 7 rolls daily record, it is characterised in that the log collection module is also
Including：
Cache cleaner unit, for reading the cache-time of the cache information, calculates the cache-time with current time
Difference；Judge the difference whether more than preset time threshold；If being more than preset time threshold, the cache information is removed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610851865.8A CN106569936B (en) | 2016-09-26 | 2016-09-26 | A kind of real-time acquisition rolls the method and system of log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610851865.8A CN106569936B (en) | 2016-09-26 | 2016-09-26 | A kind of real-time acquisition rolls the method and system of log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106569936A true CN106569936A (en) | 2017-04-19 |
| CN106569936B CN106569936B (en) | 2019-05-03 |
Family
ID=58532446
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610851865.8A Active CN106569936B (en) | 2016-09-26 | 2016-09-26 | A kind of real-time acquisition rolls the method and system of log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106569936B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107341258A (en) * | 2017-07-12 | 2017-11-10 | 中国工商银行股份有限公司 | A kind of log data acquisition method and system |
| CN108304305A (en) * | 2018-01-11 | 2018-07-20 | 北京潘达互娱科技有限公司 | The method and apparatus that journal file is read |
| CN108733546A (en) * | 2018-04-02 | 2018-11-02 | 阿里巴巴集团控股有限公司 | A kind of log collection method, device and equipment |
| CN110309158A (en) * | 2018-04-13 | 2019-10-08 | 腾讯科技（深圳）有限公司 | Rolling abnormality judgment method, device and the readable medium of journal file |
| CN110825711A (en) * | 2019-10-17 | 2020-02-21 | 上海易点时空网络有限公司 | Method and device for transmitting data in quick partitioning mode based on Flume |
| CN112015798A (en) * | 2020-09-24 | 2020-12-01 | 中盈优创资讯科技有限公司 | Data processing method and device for ensuring data non-repetition and delay monitoring |
| CN114996335A (en) * | 2022-08-03 | 2022-09-02 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012208646A (en) * | 2011-03-29 | 2012-10-25 | Nec Corp | Monitoring system |
| CN103401934A (en) * | 2013-08-06 | 2013-11-20 | 广州唯品会信息科技有限公司 | Method and system for acquiring log data |
| CN103679002A (en) * | 2013-12-12 | 2014-03-26 | 小米科技有限责任公司 | Method and device for monitoring file change and server |
| CN104636494A (en) * | 2015-03-04 | 2015-05-20 | 浪潮电子信息产业股份有限公司 | Spark-based log auditing and reversed checking system for big data platforms |
| CN105631026A (en) * | 2015-12-30 | 2016-06-01 | 北京奇艺世纪科技有限公司 | Security data analysis system |
-
2016
- 2016-09-26 CN CN201610851865.8A patent/CN106569936B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012208646A (en) * | 2011-03-29 | 2012-10-25 | Nec Corp | Monitoring system |
| CN103401934A (en) * | 2013-08-06 | 2013-11-20 | 广州唯品会信息科技有限公司 | Method and system for acquiring log data |
| CN103679002A (en) * | 2013-12-12 | 2014-03-26 | 小米科技有限责任公司 | Method and device for monitoring file change and server |
| CN104636494A (en) * | 2015-03-04 | 2015-05-20 | 浪潮电子信息产业股份有限公司 | Spark-based log auditing and reversed checking system for big data platforms |
| CN105631026A (en) * | 2015-12-30 | 2016-06-01 | 北京奇艺世纪科技有限公司 | Security data analysis system |
Non-Patent Citations (1)
| Title |
|---|
| 胡庆宝等: "基于 Elasticsearch 的实时集群日志采集和分析系统实现", 《科研信息化技术与应用》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107341258A (en) * | 2017-07-12 | 2017-11-10 | 中国工商银行股份有限公司 | A kind of log data acquisition method and system |
| CN107341258B (en) * | 2017-07-12 | 2020-03-13 | 中国工商银行股份有限公司 | Log data acquisition method and system |
| CN108304305A (en) * | 2018-01-11 | 2018-07-20 | 北京潘达互娱科技有限公司 | The method and apparatus that journal file is read |
| CN108733546A (en) * | 2018-04-02 | 2018-11-02 | 阿里巴巴集团控股有限公司 | A kind of log collection method, device and equipment |
| CN110309158A (en) * | 2018-04-13 | 2019-10-08 | 腾讯科技（深圳）有限公司 | Rolling abnormality judgment method, device and the readable medium of journal file |
| CN110309158B (en) * | 2018-04-13 | 2023-08-01 | 腾讯科技（深圳）有限公司 | Method and device for judging rolling abnormality of log file and readable medium |
| CN110825711A (en) * | 2019-10-17 | 2020-02-21 | 上海易点时空网络有限公司 | Method and device for transmitting data in quick partitioning mode based on Flume |
| CN112015798A (en) * | 2020-09-24 | 2020-12-01 | 中盈优创资讯科技有限公司 | Data processing method and device for ensuring data non-repetition and delay monitoring |
| CN112015798B (en) * | 2020-09-24 | 2023-12-05 | 中盈优创资讯科技有限公司 | Data processing method and device for guaranteeing data non-repetition and delay monitoring |
| CN114996335A (en) * | 2022-08-03 | 2022-09-02 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
| CN114996335B (en) * | 2022-08-03 | 2022-11-01 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106569936B (en) | 2019-05-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106569936A (en) | Method and system for acquiring scrolling log in real time | |
| CN105824744B (en) | A kind of real-time logs capturing analysis method based on B2B platform | |
| US8418003B2 (en) | Log collecting apparatus, program, method, and system | |
| US8290994B2 (en) | Obtaining file system view in block-level data storage systems | |
| EP1501029A3 (en) | A system for management and inspection of an asset system and a method to be used in said system | |
| CN106789377B (en) | Service parameter updating method of network element cluster | |
| US9246859B2 (en) | Peer-to-peer collaboration of publishers in a publish-subscription environment | |
| GB0809162D0 (en) | Systems and methods of network operation and information processing, including data acquisitions, processing and provisions and/or interoperability features | |
| JP2013545159A5 (en) | ||
| US20140173414A1 (en) | Method and apparatus for saving dynamic web pages | |
| CN102567185B (en) | Monitoring method of application server | |
| EP2001229A3 (en) | Information processing system, collecting server, information processing method and program | |
| CN103905441B (en) | Data capture method and device | |
| EP1164767A3 (en) | Apparatus and method supporting operation of a server and corresponding recording medium, the method being based on access status to server | |
| CN102957550A (en) | System and method for alarming based on log detection | |
| CN104301304A (en) | Vulnerability detection system based on large ISP interconnection port and method thereof | |
| CN103428249B (en) | A kind of Collecting and dealing method of HTTP request bag, system and server | |
| CN104426838A (en) | Internet cache scheduling method and system | |
| US20150039715A1 (en) | Publisher-assisted, broker-based caching in a publish-subscription environment | |
| TW200619999A (en) | Web link management systems and methods | |
| CN115460251A (en) | Cloud edge cooperation-based equipment data acquisition method, device, equipment and medium | |
| CN103929339B (en) | A kind of web data acquisition method and system | |
| CN104601354B (en) | A kind of business diary storage method, apparatus and system | |
| CN102521076A (en) | Intelligent real-time file backup method | |
| CN104811418B (en) | The method and device of viral diagnosis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Guangdong Nanshan District street of Shenzhen city in Guangdong province 518000 software industry base in Shenzhen, fifth building room 505 Applicant after: Shenzhen box Mdt InfoTech Ltd Address before: Guangdong Nanshan District street of Shenzhen city in Guangdong province 518000 software industry base in Shenzhen, fifth building room 505 Applicant before: Shenzhen Iboxpay Information Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |