CN106569936B - A kind of real-time acquisition rolls the method and system of log - Google Patents
A kind of real-time acquisition rolls the method and system of log Download PDFInfo
- Publication number
- CN106569936B CN106569936B CN201610851865.8A CN201610851865A CN106569936B CN 106569936 B CN106569936 B CN 106569936B CN 201610851865 A CN201610851865 A CN 201610851865A CN 106569936 B CN106569936 B CN 106569936B
- Authority
- CN
- China
- Prior art keywords
- file
- filepointer
- filelength
- read
- event
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention provides the method and system that a kind of acquisition in real time rolls log, are related to field of computer technology.Wherein method includes: whether monitoring logfile directory occurs predeterminable event, and predeterminable event includes that file creates event and file modification event；If predeterminable event occurs, then obtain the current read position filepointer of file pointer in this document and file size filelength of this document, newly-increased file content is read according to the filelength of the filepointer of file pointer in file and this document, and the file content read is sent to the message source Avro Source of the Data Serialization of result collection system Flume.The present invention can effectively eliminate new document creation, file content under logfile directory and constantly modify and influence of the temporary file renaming situation to log collection, realize the real-time acquisition for rolling log, improve the reliability of log collection.
Description
Technical field
The invention belongs to field of computer technology, more particularly to a kind of acquisition in real time to roll the method and system of log.
Background technique
Flume is that most common log collection technology, Flume are a distributions in the prior art, reliable efficient day
Will collection system, it allows user's self-defining data mode, and scalability is very strong, and has stronger fault-tolerant and recovery machine
System.However, still just there are certain defects by Flume in actual use: Flume generally uses Spooling
Directory Source or Exec Source carry out collector journal, and wherein Spooling Directory Source is not supported
The content of the journal file of collection is added, it, can only be by creating journal file when there is new log content to generate
Mode collects new log content, if rolling log in real time in this way to acquire, it will produce under Log Directory
Raw very more small documents, therefore Spooling Directory Source is not suitable for rolling the real-time acquisition of log；And
Although Exec Source can real-time gather data, its be usually from tail order obtain journal file content,
Tail reading of content can be carried out to fixed journal file, when journal file carries out rolling write-in or occur interim according to the time
When file renames, it will the case where acquisition failure occur, log is caused to be lost, the reliability of log collection is lower.
Summary of the invention
The method and system for being designed to provide a kind of rolling log of acquisition in real time of the embodiment of the present invention, it is intended in solution
It states in Flume generally using Spooling Directory Source or Exec Source come collector journal, is not suitable for
In the lower problem of the real-time acquisition or acquisition reliability that roll log.
The embodiments of the present invention are implemented as follows, a kind of method that real-time acquisition rolls log, comprising:
Whether monitoring logfile directory occurs predeterminable event, and the predeterminable event includes that the newly-built event of file and file are repaired
Change event；
If predeterminable event occurs, obtains the current read position filepointer of file pointer in this document and be somebody's turn to do
The file size filelength of file, according to the filepointer of file pointer in the file and this document
Filelength reads newly-increased file content, and the file content read is sent to the number of result collection system Flume
According to the message source Avro Source of serializing.
Based on the above technical solution, if the generation predeterminable event, working as file pointer in this document is obtained
The file size filelength of preceding reading position filepointer and this document, according to file pointer in the file
The filelength of filepointer and this document reads newly-increased file content, and the file content read is sent to
The message source Avro Source of the Data Serialization of result collection system is specifically included:
If predeterminable event occurs, the filename of the file is obtained, and the file is obtained according to the filename
Device number, file index number and filelength；
Judge the cache information that whether there is the file in the caching of result collection system；
The cache information of the file if it does not exist then sets 0 for the filepointer of file pointer in file,
File content is read by row in [0, filelength] range, and the content read is sent to the Avro of result collection system
Source；
The cache information of the file if it exists, then according to the device number and the file index number from the caching
The filepointer of file pointer in this document is read, and according to the filepointer of the file pointer, the file
The event type of filelength and the predeterminable event reads newly-increased file content, and the file content read is sent
To the Avro Source of result collection system.
Based on the above technical solution, described if the event type of the predeterminable event is that file creates event
According to the event type of the filepointer of the file pointer, the filelength of the file and the predeterminable event
Newly-increased file content is read, the Avro Source that the file content read is sent to result collection system is specifically included:
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, read in [filepointer, filelength] range by row
File content is taken, and the file content read is sent to the Avro Source of result collection system, while deleting this document
Cache information；
If filepointer is greater than or equal to filelength, the cache information of this document is directly removed.
Based on the above technical solution, if the event type of the predeterminable event is file modification event, basis
The event type of the filepointer of the file pointer, the filelength of the file and the predeterminable event are read
Newly-increased file content specifically includes the Avro Source that the file content read is sent to result collection system:
File content, and the file that will be read are read by row directly in [filepointer, filelength] range
Content is sent to the Avro Source of result collection system.
Based on the above technical solution, after the cache information of the file described if it exists further include:
The cache-time for reading the cache information calculates the difference of the cache-time and current time；
Judge whether the difference is greater than preset time threshold；
If more than preset time threshold, then the cache information is removed.
The another object of the embodiment of the present invention is to provide a kind of system that acquisition rolls log in real time, comprising:
Event monitoring module, for monitoring whether logfile directory occurs predeterminable event, the predeterminable event includes text
Part creates event and file modification event；
Log collection module, if obtaining the current read position of file pointer in this document for predeterminable event to occur
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content read is sent to result collection system
The message source Avro Source of the Data Serialization of Flume.
Based on the above technical solution, log collection module specifically includes:
The file information acquiring unit, if obtaining the filename of the file, and according to described for predeterminable event to occur
Filename obtains the device number, file index number and filelength of the file；
Judging unit is cached, whether there is the cache information of the file in the caching for judging result collection system；
First log collection unit, for the cache information of the file if it does not exist, then by file pointer in file
Filepointer is set as 0, reads file content by row in [0, filelength] range, and the content read is sent out
It send to the Avro Source of result collection system；
Second log collector unit, for the cache information of the file if it exists, then according to the device number and described
File index number is from the filepointer for reading file pointer in this document in the caching, and according to the file pointer
The event type of filepointer, the filelength of the file and the predeterminable event read newly-increased file content,
The file content read is sent to the Avro Source of result collection system.
Based on the above technical solution, described if the event type of the predeterminable event is that file creates event
Second log collector unit, which has, to be used for:
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, read in [filepointer, filelength] range by row
File content is taken, and the file content read is sent to the Avro Source of result collection system, while deleting this document
Cache information；
If filepointer is greater than or equal to filelength, the cache information of this document is directly removed.
Based on the above technical solution, described if the event type of the predeterminable event is file modification event
Second log collector unit is specifically used for:
File content, and the file that will be read are read by row directly in [filepointer, filelength] range
Content is sent to the Avro Source of result collection system.
Based on the above technical solution, the log collection module further include:
Cache cleaner unit, for reading the cache-time of the cache information, calculate the cache-time and it is current when
Between difference；Judge whether the difference is greater than preset time threshold；If more than preset time threshold, then the caching letter is removed
Breath.
Implementing the method and system that a kind of acquisition in real time provided in an embodiment of the present invention rolls log has below beneficial to effect
Fruit:
For the embodiment of the present invention by monitoring whether logfile directory occurs predeterminable event, the predeterminable event includes file
Newly-built event and file modification event；If predeterminable event occurs, the current read position of file pointer in this document is obtained
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content read is sent to result collection system
The message source Avro Source of the Data Serialization of Flume, so as to effectively eliminate new file wound under logfile directory
Build, file content is constantly modified and influence of the temporary file renaming situation to log collection, can be realized the reality for rolling log
When acquire, improve the reliability of log collection.
Detailed description of the invention
Fig. 1 is the specific implementation flow chart for the method that a kind of acquisition in real time provided in an embodiment of the present invention rolls log；
Fig. 2 is the specific implementation that a kind of acquisition in real time provided in an embodiment of the present invention rolls step S102 in the method for log
Flow chart；
Fig. 3 is the schematic block diagram for the system that a kind of acquisition in real time provided in an embodiment of the present invention rolls log；
Fig. 4 is the signal that a kind of acquisition in real time provided in an embodiment of the present invention rolls log collection module in the system of log
Property block diagram.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.
Fig. 1 is the specific implementation flow chart for the method that a kind of acquisition in real time provided in an embodiment of the present invention rolls log.Ginseng
As shown in Figure 1, a kind of method that acquisition rolls log in real time provided in this embodiment, may comprise steps of:
In S101, whether monitoring logfile directory occurs predeterminable event, and the predeterminable event includes that file creates thing
Part and file modification event.
In the present embodiment, step S101 is specifically included: one monitoring service instance objects watcher of creation, and is institute
Log text catalogue addition watcher is stated, is registration monitoring event by predeterminable event, then by described in watcher monitoring
Whether logfile directory occurs the predeterminable event.
In S102, if predeterminable event occurs, the current read position of file pointer in this document is obtained
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content read is sent to result collection system
The message source Avro Source of the Data Serialization of Flume.
Shown in Figure 2 Fig. 2 shows the specific implementation flow of step S102, step S102 can specifically include following
Step:
In S201, if predeterminable event occurs, the filename of the file is obtained, and institute is obtained according to the filename
State the device number, file index number and filelength of file.
In the present embodiment, when predeterminable event occurs, monitor event object can issue signal, and be added to
In the queue of watcher, result collection system can find the monitoring thing for issuing signal from the queue that watcher is monitored
Part object, and then the filename of the monitor event object respective file is got, the text is then obtained according to the filename
Device number, file index number and the filelength of part.
In S202, the cache information that whether there is the file in the caching of result collection system is judged.
In the present embodiment, it has been searched whether in the caching of the result collection system by the filename of the file
The cache information of this document illustrates the caching of result collection system if finding the cache information to match with file name
In have the cache information of this document, conversely, there is no the cache information of this document in the caching of result collection system.
In S203, the cache information of the file, then set the filepointer of file pointer in file if it does not exist
It is set to 0, reads file content by row in [0, filelength] range, and the content read is sent to log collection system
The Avro Source of system.
In the present embodiment, if the cache information of the file is not present in the caching of result collection system, illustrate this
Log content in file is newly-increased or modified content, sets 0 for filepointer at this time, [0,
Filelength] the interior full content that this document is read by row of range, and the content read is sent to result collection system
Avro Source, while updating the read access time of the filepointer of file pointer and cache information in file, and with
The device number and file index number of file as file pointer in the file after unique identification buffer update filepointer with
And the read access time of cache information.
In S204, the cache information of the file if it exists, then according to the device number and the file index number from
In the caching read this document in file pointer filepointer, and according to the filepointer of the file pointer,
The event type of the filelength of the file and the predeterminable event reads newly-increased file content, the text that will be read
Part content is sent to the Avro Source of result collection system.
In one embodiment, described according to institute if the event type of the predeterminable event is that file creates event
The event type for stating the filepointer of file pointer, the filelength of the file and the predeterminable event is read newly
The file content of increasing specifically includes the Avro Source that the file content read is sent to result collection system:
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, read in [filepointer, filelength] range by row
File content is taken, and the file content read is sent to the Avro Source of result collection system, while deleting this document
Cache information；
If filepointer is greater than or equal to filelength, the cache information of this document is directly removed.
In the present embodiment, if the predeterminable event is that file creates event, and there are the cachings of this document in caching
Information then illustrates that the newly-built event of this document is triggered since file renames, if filepointer is greater than or equal at this time
Filelength then directly removes the cache information of this document；On the contrary, if filepointer is less than filelength,
File content is read by row in [filepointer, filelength] range, and the file content read is sent to log
The Avro Source of collection system, while deleting the cache information of this document.
In another embodiment, if the event type of the predeterminable event is file modification event, according to
The event type of the filepointer of file pointer, the filelength of the file and the predeterminable event read newly-increased
File content, the Avro Source that the file content read is sent to result collection system is specifically included:
File content, and the file that will be read are read by row directly in [filepointer, filelength] range
Content is sent to the Avro Source of result collection system.
In the present embodiment, if the predeterminable event is file modification event, and there are the cachings of this document in caching
Information is then directly reading file content, and the text that will be read by row in [filepointer, filelength] range
Part content is sent to the Avro Source of result collection system, and saves the cache information of this document, while updating file Chinese
The filepointer of part pointer and the read access time of cache information, and using the device number of file and file index number as only
The filepointer of file pointer and the read access time of cache information in file after one mark buffer update.
Further, in step S204, after the cache information of the file described if it exists further include:
The cache-time for reading the cache information calculates the difference of the cache-time and current time；
Judge whether the difference is greater than preset time threshold；
If more than preset time threshold, then the cache information is removed.
In the present embodiment, due in real time according to the difference of the cache-time of cache information and current time to cache information
Cleared up, so as to avoid not occurring because of logfile directory file renaming event caused by cache information cleaning be obstructed
Problem occurs.
Above as can be seen that the method that a kind of acquisition in real time provided in this embodiment rolls log passes through monitoring journal file
Whether catalogue occurs predeterminable event, and the predeterminable event includes that file creates event and file modification event；If default thing occurs
Part then obtains the current read position filepointer of file pointer in this document and the file size of this document
Filelength, newly-increased according to the filelength of the filepointer of file pointer and this document reading in the file
File content, and the file content read is sent to the message source Avro of the Data Serialization of result collection system Flume
Source is constantly modified and temporary file so as to effectively eliminate new document creation, file content under logfile directory
Influence of the situation to log collection is renamed, the real-time acquisition for rolling log is can be realized, improves the reliability of log collection.
Fig. 3 is the schematic block diagram for the system that a kind of acquisition in real time provided in an embodiment of the present invention rolls log, the system
Method for running embodiment illustrated in fig. 1 offer.It illustrate only part related to the present embodiment for ease of description.
It is shown in Figure 3, a kind of system that acquisition rolls log in real time provided in this embodiment, comprising:
Event monitoring module 1, for monitoring whether logfile directory occurs predeterminable event, the predeterminable event includes text
Part creates event and file modification event；
Log collection module 2, if obtaining the current read position of file pointer in this document for predeterminable event to occur
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content read is sent to result collection system
The message source Avro Source of the Data Serialization of Flume.
Optionally, shown in Figure 4, the log collection module 2 specifically includes:
The file information acquiring unit 21, if obtaining the filename of the file, and according to institute for predeterminable event to occur
State device number, file index number and filelength that filename obtains the file；
Judging unit 22 is cached, the caching that whether there is the file in the caching for judging result collection system is believed
Breath；
First log collection unit 23, for the cache information of the file if it does not exist, then by file pointer in file
Filepointer be set as 0, read file content, and the content that will be read by row in [0, filelength] range
It is sent to the Avro Source of result collection system；
Second log collector unit 24, for the cache information of the file if it exists, then according to the device number and institute
File index number is stated from the filepointer for reading file pointer in this document in the caching, and according to the file pointer
Filepointer, the filelength of the file and the event type of the predeterminable event read in newly-increased file
Hold, the file content read is sent to the Avro Source of result collection system.
Optionally, if the event type of the predeterminable event is that file creates event, the second log collector unit
24 have be used for:
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, read in [filepointer, filelength] range by row
File content is taken, and the file content read is sent to the Avro Source of result collection system, while deleting this document
Cache information；
If filepointer is greater than or equal to filelength, the cache information of this document is directly removed.
Optionally, if the event type of the predeterminable event is file modification event, the second log collector unit
24 are specifically used for:
File content, and the file that will be read are read by row directly in [filepointer, filelength] range
Content is sent to the Avro Source of result collection system, while deleting the cache information of this document.
Optionally, the log collection module further include:
Cache cleaner unit 25 calculates the cache-time and current for reading the cache-time of the cache information
The difference of time；Judge whether the difference is greater than preset time threshold；If more than preset time threshold, then the caching is removed
Information.
It should be noted that the modules in above system provided in an embodiment of the present invention, due to the method for the present invention
Embodiment is based on same design, and bring technical effect is identical as embodiment of the present invention method, and particular content can be found in this hair
Narration in bright embodiment of the method, details are not described herein again.
Therefore, it can be seen that the system that acquisition rolls log a kind of in real time provided in an embodiment of the present invention equally can be effective
Elimination logfile directory under new document creation, file content constantly modify and temporary file renames situation to log collection
Influence, can be realized roll log real-time acquisition, improve the reliability of log collection.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Made any modifications, equivalent replacements, and improvements etc., should all be included in the protection scope of the present invention within mind and principle.
Claims (8)
1. a kind of method that acquisition rolls log in real time characterized by comprising
Whether monitoring logfile directory occurs predeterminable event, and the predeterminable event includes that file creates event and file modification thing
Part；
If predeterminable event occurs, the current read position filepointer and this document of file pointer in this document are obtained
File size filelength, according to the filelength of the filepointer of file pointer and this document in the file
Newly-increased file content is read, and the file content read is sent to the Data Serialization of result collection system Flume
Message source Avro Source；
If the generation predeterminable event, obtains the current read position filepointer of file pointer in this document and be somebody's turn to do
The file size filelength of file, according to the filepointer of file pointer in the file and this document
Filelength reads newly-increased file content, and the file content read is sent to the data sequence of result collection system
The message source Avro Source of change is specifically included:
If predeterminable event occurs, the filename of the file, and the equipment that the file is obtained according to the filename are obtained
Number, file index number and filelength；
Judge the cache information that whether there is the file in the caching of result collection system；
The cache information of the file if it does not exist then sets 0 for the filepointer of file pointer in file, [0,
Filelength] the interior row of pressing of range reads file content, and the content read is sent to the Avro of result collection system
Source；
The cache information of the file if it exists is then read from the caching according to the device number and the file index number
The filepointer of file pointer in this document, and according to the filepointer of the file pointer, the file
The event type of filelength and the predeterminable event reads newly-increased file content, and the file content read is sent
To the Avro Source of result collection system.
2. the method as described in claim 1 that acquisition rolls log in real time, which is characterized in that if the event of the predeterminable event
Type is that file creates event, then the filepointer according to the file pointer, the file filelength with
And the event type of the predeterminable event reads newly-increased file content, and the file content read is sent to log collection system
The Avro Source of system is specifically included:
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, text is read by row in [filepointer, filelength] range
Part content, and the file content read is sent to the Avro Source of result collection system, while deleting the slow of this document
Deposit information；
If filepointer is greater than or equal to filelength, the cache information of this document is directly removed.
3. the method as described in claim 1 that acquisition rolls log in real time, which is characterized in that if the event of the predeterminable event
Type is file modification event, then according to the filepointer of the file pointer, the filelength of the file and institute
The event type for stating predeterminable event reads newly-increased file content, and the file content read is sent to result collection system
Avro Source is specifically included:
File content, and the file content that will be read are read by row directly in [filepointer, filelength] range
It is sent to the Avro Source of result collection system.
4. the method as described in claim 1 that acquisition rolls log in real time, which is characterized in that the file described if it exists
After cache information further include:
The cache-time for reading the cache information calculates the difference of the cache-time and current time；
Judge whether the difference is greater than preset time threshold；
If more than preset time threshold, then the cache information is removed.
5. a kind of system that acquisition rolls log in real time characterized by comprising
Event monitoring module, for monitoring whether logfile directory occurs predeterminable event, the predeterminable event includes that file is new
Build event and file modification event；
Log collection module, if obtaining the current read position of file pointer in this document for predeterminable event to occur
The file size filelength of filepointer and this document, according to the filepointer of file pointer in the file
Newly-increased file content is read with the filelength of this document, and the file content read is sent to result collection system
The message source Avro Source of the Data Serialization of Flume；
Log collection module specifically includes:
The file information acquiring unit, if obtaining the filename of the file, and according to the file for predeterminable event to occur
Name obtains device number, file index number and the filelength of the file；
Judging unit is cached, whether there is the cache information of the file in the caching for judging result collection system；
First log collection unit, for the cache information of the file if it does not exist, then by file pointer in file
Filepointer is set as 0, reads file content by row in [0, filelength] range, and the content read is sent out
It send to the Avro Source of result collection system；
Second log collector unit, for the cache information of the file if it exists, then according to the device number and the file
Call number is from the filepointer for reading file pointer in this document in the caching, and according to the file pointer
The event type of filepointer, the filelength of the file and the predeterminable event read newly-increased file content,
The file content read is sent to the Avro Source of result collection system.
6. the system as claimed in claim 5 that acquisition rolls log in real time, which is characterized in that if the event of the predeterminable event
Type is that file creates event, then the second log collector unit has and is used for:
Judge whether the filepointer of the file pointer is less than the filelength of the file；
If filepointer is less than filelength, text is read by row in [filepointer, filelength] range
Part content, and the file content read is sent to the Avro Source of result collection system, while deleting the slow of this document
Deposit information；
If filepointer is greater than or equal to filelength, the cache information of this document is directly removed.
7. the system as claimed in claim 5 that acquisition rolls log in real time, which is characterized in that if the event of the predeterminable event
Type is file modification event, then the second log collector unit is specifically used for:
File content, and the file content that will be read are read by row directly in [filepointer, filelength] range
It is sent to the Avro Source of result collection system.
8. the system as claimed in claim 5 that acquisition rolls log in real time, which is characterized in that the log collection module is also wrapped
It includes:
Cache cleaner unit calculates the cache-time and current time for reading the cache-time of the cache information
Difference；Judge whether the difference is greater than preset time threshold；If more than preset time threshold, then the cache information is removed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610851865.8A CN106569936B (en) | 2016-09-26 | 2016-09-26 | A kind of real-time acquisition rolls the method and system of log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610851865.8A CN106569936B (en) | 2016-09-26 | 2016-09-26 | A kind of real-time acquisition rolls the method and system of log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106569936A CN106569936A (en) | 2017-04-19 |
| CN106569936B true CN106569936B (en) | 2019-05-03 |
Family
ID=58532446
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610851865.8A Active CN106569936B (en) | 2016-09-26 | 2016-09-26 | A kind of real-time acquisition rolls the method and system of log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106569936B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107341258B (en) * | 2017-07-12 | 2020-03-13 | 中国工商银行股份有限公司 | Log data acquisition method and system |
| CN108304305A (en) * | 2018-01-11 | 2018-07-20 | 北京潘达互娱科技有限公司 | The method and apparatus that journal file is read |
| CN108733546A (en) * | 2018-04-02 | 2018-11-02 | 阿里巴巴集团控股有限公司 | A kind of log collection method, device and equipment |
| CN110309158B (en) * | 2018-04-13 | 2023-08-01 | 腾讯科技（深圳）有限公司 | Method and device for judging rolling abnormality of log file and readable medium |
| CN110825711A (en) * | 2019-10-17 | 2020-02-21 | 上海易点时空网络有限公司 | Method and device for transmitting data in quick partitioning mode based on Flume |
| CN112015798B (en) * | 2020-09-24 | 2023-12-05 | 中盈优创资讯科技有限公司 | Data processing method and device for guaranteeing data non-repetition and delay monitoring |
| CN114996335B (en) * | 2022-08-03 | 2022-11-01 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012208646A (en) * | 2011-03-29 | 2012-10-25 | Nec Corp | Monitoring system |
| CN103401934A (en) * | 2013-08-06 | 2013-11-20 | 广州唯品会信息科技有限公司 | Method and system for acquiring log data |
| CN103679002A (en) * | 2013-12-12 | 2014-03-26 | 小米科技有限责任公司 | Method and device for monitoring file change and server |
| CN104636494A (en) * | 2015-03-04 | 2015-05-20 | 浪潮电子信息产业股份有限公司 | Log audit checking system based on Spark big data platform |
| CN105631026A (en) * | 2015-12-30 | 2016-06-01 | 北京奇艺世纪科技有限公司 | Security data analysis system |
-
2016
- 2016-09-26 CN CN201610851865.8A patent/CN106569936B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012208646A (en) * | 2011-03-29 | 2012-10-25 | Nec Corp | Monitoring system |
| CN103401934A (en) * | 2013-08-06 | 2013-11-20 | 广州唯品会信息科技有限公司 | Method and system for acquiring log data |
| CN103679002A (en) * | 2013-12-12 | 2014-03-26 | 小米科技有限责任公司 | Method and device for monitoring file change and server |
| CN104636494A (en) * | 2015-03-04 | 2015-05-20 | 浪潮电子信息产业股份有限公司 | Log audit checking system based on Spark big data platform |
| CN105631026A (en) * | 2015-12-30 | 2016-06-01 | 北京奇艺世纪科技有限公司 | Security data analysis system |
Non-Patent Citations (1)
| Title |
|---|
| 基于 Elasticsearch 的实时集群日志采集和分析系统实现;胡庆宝等;《科研信息化技术与应用》;20160531;第13-18页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106569936A (en) | 2017-04-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106569936B (en) | A kind of real-time acquisition rolls the method and system of log | |
| CN110928718B (en) | Abnormality processing method, system, terminal and medium based on association analysis | |
| Wong et al. | Quantifying political leaning from tweets and retweets | |
| CN107273496B (en) | Method for detecting microblog network region emergency | |
| CN106815125A (en) | A kind of log audit method and platform | |
| Amini et al. | DENGRIS-Stream: A density-grid based clustering algorithm for evolving data streams over sliding window | |
| EP2472829A8 (en) | Methods, systems and devices for horizontally scalable high-availability dynamic context-based routing | |
| US9424743B2 (en) | Real-time traffic detection | |
| US8489694B2 (en) | Peer-to-peer collaboration of publishers in a publish-subscription environment | |
| JP2012198832A (en) | Duplicate file detection device | |
| CN109067787A (en) | Distributed Denial of Service (DDOS) attack detection method and device | |
| CN104301304A (en) | Vulnerability detection system based on large ISP interconnection port and method thereof | |
| CN109525448A (en) | Log data acquisition system and method | |
| CN111797104A (en) | Method and device for acquiring data change condition and electronic equipment | |
| CN118376839B (en) | Method, device and equipment for positioning newly-added peak point frequency based on DBSCAN algorithm | |
| CN108197112A (en) | A kind of method that event is extracted from news | |
| EP3009942A1 (en) | Social contact message monitoring method and device | |
| CN108073705B (en) | Distributed mass data aggregation acquisition method | |
| CN117768870A (en) | Equipment query method, electronic equipment and storage medium | |
| CN108966169A (en) | For the Analysis on heavy metal pollution early warning system of agricultural land soil | |
| US20120215858A1 (en) | Caching potentially repetitive message data in a publish-subscription environment | |
| CN112183161B (en) | Face database processing method, device and equipment | |
| CN107729206A (en) | Real-time analysis method, system and the computer-processing equipment of alarm log | |
| CN112907257A (en) | Risk threshold determining method, abnormality detecting device and electronic equipment | |
| US20200272936A1 (en) | System and Method for Managing Inventory of Artificial Intelligence Entities |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Guangdong Nanshan District street of Shenzhen city in Guangdong province 518000 software industry base in Shenzhen, fifth building room 505 Applicant after: Shenzhen box Mdt InfoTech Ltd Address before: Guangdong Nanshan District street of Shenzhen city in Guangdong province 518000 software industry base in Shenzhen, fifth building room 505 Applicant before: Shenzhen Iboxpay Information Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |