CN105159820A - Transmission method and device of system log data - Google Patents
Transmission method and device of system log data Download PDFInfo
- Publication number
- CN105159820A CN105159820A CN201510561048.4A CN201510561048A CN105159820A CN 105159820 A CN105159820 A CN 105159820A CN 201510561048 A CN201510561048 A CN 201510561048A CN 105159820 A CN105159820 A CN 105159820A
- Authority
- CN
- China
- Prior art keywords
- file
- log
- journal file
- old
- incremental
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a transmission method and device of system log data. The method comprises the following steps: fetching log data from an application server; according to the fetched log data, generating a new log file; comparing the new log file with an original log file in a log database, and extracting newly-increased content to serve as an increment log file; and writing the increment log file into the log database, and deleting the log data which is written into the log database from the application server. The transmission method and device of the system log data can fetch the log data for backup, and when the application server shuts down, the log which can not be sent out in time can not be lost. In addition, only one set of database is established in the log server, and stored resources are saved.
Description
Technical field
The present invention relates to syslog data technical field, particularly relate to a kind of syslog data transmission method and device.
Background technology
Daily record (log) can be used for storing and the details of register system ruuning situation and user's service condition, and these information check the Main Basis of system running state and the problem of investigation.Such as, in certain software there is collapse case in application program, so can find corresponding record from program event daily record, thus help person skilled analysis and deal with problems.
For inline system log information, log information can be used for recording service operation process on line and working procedure is monitored, can carry out pointing out, warning for the mistake occurred and may occur by the log information of analytic record, all right statistical service information, for the upgrading of the improved Haniltonian program of service logic provides reliable foundation.
Existing log transmission method, the log information that on application server, each application program module produces sends to daily record client by LIB storehouse, daily record client preserves the daily record received, again according to log server ip address and the associated port number of configuration, by received Log Sender to log server.Log server receives daily record, is saved in unified for received daily record in DB.
This scheme is machine once application server is delayed, and the daily record do not sent in time will be lost, and before log server processes daily record, is the daily record that cannot view current generation.In addition, this scheme will on the application server with log server sets up a set of database respectively, store two cover daily record datas, cause the waste of storage resources.
Summary of the invention
The object of this invention is to provide a kind of syslog data transmission method and device, object is to reduce the risk that daily record data is lost and the redundancy reducing storage.
For solving the problems of the technologies described above, the invention provides a kind of syslog data transmission method, comprising:
From application server crawl log data;
Daily record data according to capturing generates new journal file;
Original journal file in described new journal file and log database is compared, extracts newly-increased content as Incremental Log file;
By in described Incremental Log file write log database, and from described application server, delete the daily record data writing log database.
Alternatively, describedly to comprise from application server crawl log data:
Capture the full dose daily record data on the same day from application server according to predetermined time interval timing.
Alternatively, described original journal file in described new journal file and log database to be compared, extracts newly-increased content and comprise as Incremental Log file:
Judge in described log database, whether to there is the old journal file corresponding with described new journal file;
When there is described old journal file, described new journal file and old journal file being compared, extracting newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
Alternatively, original journal file in described new journal file and log database is being compared, is extracting newly-increased content and also comprise as after Incremental Log file:
Judge in described log database, whether to there is the old Incremental Log file corresponding with current delta journal file;
When there is described old Incremental Log file, current delta journal file is adopted to replace described old Incremental Log file.
Alternatively, described described Incremental Log file write log database to be comprised:
Make a mistake when Incremental Log file writes described log database, the daily record data of write is carried out rollback.
Present invention also offers a kind of syslog data transmitting device, comprising:
Handling module, for from application server crawl log data;
Generation module, for generating new journal file according to the daily record data captured;
Extraction module, for being compared by original journal file in described new journal file and log database, extracts newly-increased content as Incremental Log file;
Writing module, for being write in log database by described Incremental Log file, and deletes the daily record data writing log database from described application server.
Alternatively, described handling module is used for comprising from application server crawl log data:
Described handling module is specifically for capturing the full dose daily record data on the same day from application server according to predetermined time interval timing.
Alternatively, described extraction module comprises:
Whether judging unit, exist the old journal file corresponding with described new journal file for judging in described log database;
Extraction unit, for when there is described old journal file, compares described new journal file and old journal file, extracts newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
Alternatively, also comprise:
Replacement module, for original journal file in described new journal file and log database is being compared, extract newly-increased content as after Incremental Log file, judge in described log database, whether to there is the old Incremental Log file corresponding with current delta journal file; When there is described old Incremental Log file, current delta journal file is adopted to replace described old Incremental Log file.
Be used for described Incremental Log file write log database to comprise in, said write module alternatively:
The daily record data of write, specifically for making a mistake when Incremental Log file writes described log database, is carried out rollback by said write module.
Syslog data transmission method provided by the present invention and device, by crawl log data from application server, generate new journal file; Journal file original in new journal file and log database is compared, extracts newly-increased content as Incremental Log file; By in this Incremental Log file write log database, and from application server, delete the daily record data having write log database.Visible, syslog data transmission method provided by the present invention and device, can back up the daily record data grabbed, and to delay machine at application server, the daily record do not sent in time can not be lost.In addition, the present invention only establishes a set of database at log server, has saved the resource stored.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of a kind of embodiment of syslog data transmission method provided by the present invention;
Fig. 2 is the process flow diagram of the another kind of embodiment of syslog data transmission method provided by the present invention;
Fig. 3 is the structured flowchart of a kind of embodiment of syslog data transmitting device provided by the present invention.
Embodiment
In order to make those skilled in the art person understand the present invention program better, below in conjunction with the drawings and specific embodiments, the present invention is described in further detail.Obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
As shown in Figure 1, the method comprises the process flow diagram of a kind of embodiment of syslog data transmission method provided by the present invention:
Step S101: from application server crawl log data;
Step S102: the daily record data according to capturing generates new journal file;
Step S103: compared by original journal file in described new journal file and log database, extracts newly-increased content as Incremental Log file;
Step S104: by described Incremental Log file write log database, and delete the daily record data writing log database from described application server.
Syslog data transmission method provided by the present invention, by crawl log data from application server, generates new journal file; Journal file original in new journal file and log database is compared, extracts newly-increased content as Incremental Log file; By in this Incremental Log file write log database, and from application server, delete the daily record data having write log database.Visible, syslog data transmission method provided by the present invention, can back up the daily record data grabbed, and to delay machine at application server, the daily record do not sent in time can not be lost.In addition, the present invention only establishes a set of database at log server, has saved the resource stored.
As shown in Figure 2, the method comprises the process flow diagram of the another kind of embodiment of syslog data transmission method provided by the present invention:
Step S201: the full dose daily record data capturing the same day according to predetermined time interval timing from application server;
Step S202: the daily record data according to capturing generates new journal file;
Step S203: judge whether there is the old journal file corresponding with new journal file in log database;
Step S204: when there is old journal file, compares new journal file and old journal file, extracts newly-increased content as Incremental Log file; When there is not old journal file, creating an empty file as old journal file, extracting the content of new journal file as Incremental Log file;
Step S205: by Incremental Log file write log database, and delete the daily record data having write log database from application server;
Step S206: judge whether there is the old Incremental Log file corresponding with current delta journal file in log database; When there is old Incremental Log file, current delta journal file is adopted to replace old Incremental Log file.
Alternatively, described Incremental Log file writes in the process of log database above-mentioned by the syslog data transmission method that the present embodiment provides, make a mistake when Incremental Log file writes described log database, then the daily record data of write is carried out rollback.
Syslog data transmission method provided by the present invention can reduce the risk that daily record data is lost, and reduces the redundancy stored.
For making those skilled in the art understand the present invention better, set forth the processing procedure of the embodiment of the present invention better below by way of a concrete example:
Step S301: set per half an hour log server from crawl log file application server on log server, such as today is on August 31st, 2012, log server just can per half an hour, captures the full dose daily record data of today, such as: GatewayServer_INFO_log.20120831.
Step S302: after log server grabs daily record data, carries out rename to daily record data, the file that called after ends up with new, such as GatewayServer_INFO_log.20120831.new, represents that this file is new journal file.
Step S303: judge whether there is GatewayServer-_INFO_log.20120831.old file in log server, if so, then performs step S305; If not, then step S304 is performed:.
Step S304: log server creates an empty ancient deed, called after GatewayServer-_INFO_log.20120831.old file, goes to step S305.
Step S305: use diff order GatewayServer_INFO_log.20120831.new file and GatewayServer_INFO_log.20120831.old file to be contrasted; Extract new content and generate new comparison document GatewayServer_INFO_log.20120831.temp file.
Step S306: adopt new comparison document GatewayServer_INFO_log.20120831.temp replacement old comparison document GatewayServer_INFO_log.20120831.temp formerly.
Step S307: use sed order to delete the respective explanations information produced by diff order, generates delta file GatewayServer_INFO_log.20120831.final file.
Step S308: adopt described new increment journal file GatewayServer_INFO_log.20120831.final replacement old Incremental Log file GatewayServer_INFO_log.20120831.final formerly.
Step S309: delta file GatewayServer_INFO_log.20120831.final file is formatd and imports database, such as, a line content of this file is [20518] [1190676800] [0AA0000FE33C52012083100155221307] 2012083000:22:08.063INFOUpdateTransLogSuccess, corresponding is process number, thread number, serial number, timestamp, rank, separator, content, 7 row contents, but system directive such as AWK (text processing facilities) etc. can think that foregoing is 8 row, because system can only judge columns according to space for above content, but between the date and time of time row, there is a space, can think that it is two row, need to carry out merging treatment to these two row, form row.Then to go out the literary name section that 6 row contents of decollator are database, generate SQL (Structured Query Language (SQL)), insert log database, often read a line, process once, until run through all files.
Step S310: adopt new journal file GatewayServer_INFO_log.20120831.new to replace old journal file GatewayServer_INFO_log.20120831.old.
Step S311: if making a mistake in Incremental Log file write log database, then the daily record data of write is carried out rollback.
It should be noted that, for embodiment of the method, in order to simple description, therefore it is all expressed as a series of combination of actions, but those skilled in the art should know, the present invention is not by the restriction of described sequence of movement, because according to the present invention, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to preferred embodiment, and involved action and module might not be that the present invention is necessary.
The structured flowchart of a kind of embodiment of syslog data transmitting device provided by the present invention as shown in Figure 3, comprising:
Handling module 100, for from application server crawl log data;
Generation module 200, for generating new journal file according to the daily record data captured;
Extraction module 300, for being compared by original journal file in described new journal file and log database, extracts newly-increased content as Incremental Log file;
Writing module 400, for being write in log database by described Incremental Log file, and deletes the daily record data writing log database from described application server.
Syslog data transmitting device provided by the present invention, by crawl log data from application server, generates new journal file; Journal file original in new journal file and log database is compared, extracts newly-increased content as Incremental Log file; By in this Incremental Log file write log database, and from application server, delete the daily record data having write log database.Visible, syslog data transmitting device provided by the present invention, can back up the daily record data grabbed, and to delay machine at application server, the daily record do not sent in time can not be lost.In addition, the present invention only establishes a set of database at log server, has saved the resource stored.
Wherein, said extracted module may further include:
Whether judging unit, exist the old journal file corresponding with described new journal file for judging in described log database;
Extraction unit, for when there is described old journal file, compares described new journal file and old journal file, extracts newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
Syslog data transmitting device provided by the present invention is corresponding with said method, does not repeat them here.
In this instructions, each embodiment adopts the mode of going forward one by one to describe, and what each embodiment stressed is the difference with other embodiment, between each embodiment same or similar part mutually see.
To the above-mentioned explanation of the disclosed embodiments, professional and technical personnel in the field are realized or uses the present invention.To be apparent for those skilled in the art to the multiple amendment of these embodiments, General Principle as defined herein can without departing from the spirit or scope of the present invention, realize in other embodiments.Therefore, the present invention can not be restricted to these embodiments shown in this article, but will meet the widest scope consistent with principle disclosed herein and features of novelty.
Claims (10)
1. a syslog data transmission method, is characterized in that, comprising:
From application server crawl log data;
Daily record data according to capturing generates new journal file;
Original journal file in described new journal file and log database is compared, extracts newly-increased content as Incremental Log file;
By in described Incremental Log file write log database, and from described application server, delete the daily record data writing log database.
2. the system as claimed in claim 1 daily record data transmission method, is characterized in that, describedly comprises from application server crawl log data:
Capture the full dose daily record data on the same day from application server according to predetermined time interval timing.
3. the system as claimed in claim 1 daily record data transmission method, is characterized in that, is describedly compared by original journal file in described new journal file and log database, extracts newly-increased content and comprises as Incremental Log file:
Judge in described log database, whether to there is the old journal file corresponding with described new journal file;
When there is described old journal file, described new journal file and old journal file being compared, extracting newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
4. syslog data transmission method as claimed in claim 3, is characterized in that, compared by original journal file in described new journal file and log database, extracts newly-increased content and also comprises as after Incremental Log file:
Judge in described log database, whether to there is the old Incremental Log file corresponding with current delta journal file;
When there is described old Incremental Log file, current delta journal file is adopted to replace described old Incremental Log file.
5. the syslog data transmission method as described in any one of Claims 1-4, is characterized in that, is describedly comprised by described Incremental Log file write log database:
Make a mistake when Incremental Log file writes described log database, the daily record data of write is carried out rollback.
6. a syslog data transmitting device, is characterized in that, comprising:
Handling module, for from application server crawl log data;
Generation module, for generating new journal file according to the daily record data captured;
Extraction module, for being compared by original journal file in described new journal file and log database, extracts newly-increased content as Incremental Log file;
Writing module, for being write in log database by described Incremental Log file, and deletes the daily record data writing log database from described application server.
7. syslog data transmitting device as claimed in claim 6, is characterized in that, described handling module is used for comprising from application server crawl log data:
Described handling module is specifically for capturing the full dose daily record data on the same day from application server according to predetermined time interval timing.
8. syslog data transmitting device as claimed in claim 6, it is characterized in that, described extraction module comprises:
Whether judging unit, exist the old journal file corresponding with described new journal file for judging in described log database;
Extraction unit, for when there is described old journal file, compares described new journal file and old journal file, extracts newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
9. syslog data transmitting device as claimed in claim 8, is characterized in that, also comprise:
Replacement module, for original journal file in described new journal file and log database is being compared, extract newly-increased content as after Incremental Log file, judge in described log database, whether to there is the old Incremental Log file corresponding with current delta journal file; When there is described old Incremental Log file, current delta journal file is adopted to replace described old Incremental Log file.
10. the syslog data transmitting device as described in any one of claim 6 to 9, is characterized in that, said write module is used for described Incremental Log file write log database to comprise:
The daily record data of write, specifically for making a mistake when Incremental Log file writes described log database, is carried out rollback by said write module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510561048.4A CN105159820A (en) | 2015-09-06 | 2015-09-06 | Transmission method and device of system log data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510561048.4A CN105159820A (en) | 2015-09-06 | 2015-09-06 | Transmission method and device of system log data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105159820A true CN105159820A (en) | 2015-12-16 |
Family
ID=54800682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510561048.4A Pending CN105159820A (en) | 2015-09-06 | 2015-09-06 | Transmission method and device of system log data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105159820A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105516349A (en) * | 2016-01-04 | 2016-04-20 | 陈华锋 | File transmission method and system |
| CN107229554A (en) * | 2016-03-25 | 2017-10-03 | 展讯通信(天津)有限公司 | Mobile terminal stability test method and device |
| CN107247767A (en) * | 2017-06-05 | 2017-10-13 | 山东浪潮通软信息科技有限公司 | A kind of method and device that database is imported by formatted data files |
| CN107562882A (en) * | 2017-09-04 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of method of data synchronization and device based on log analysis |
| CN110389883A (en) * | 2019-06-27 | 2019-10-29 | 西安联乘智能科技有限公司 | A kind of module log real-time monitoring system based on multithreading |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080059542A1 (en) * | 2006-08-30 | 2008-03-06 | Inmage Systems, Inc. | Ensuring data persistence and consistency in enterprise storage backup systems |
| CN102937931A (en) * | 2012-09-25 | 2013-02-20 | 北京奇虎科技有限公司 | Method and system of log data transmission |
| CN104699592A (en) * | 2012-09-25 | 2015-06-10 | 北京奇虎科技有限公司 | Log data transmission method and log data transmission system |
| CN104714880A (en) * | 2012-09-25 | 2015-06-17 | 北京奇虎科技有限公司 | Log data transmission method and system as well as log server |
-
2015
- 2015-09-06 CN CN201510561048.4A patent/CN105159820A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080059542A1 (en) * | 2006-08-30 | 2008-03-06 | Inmage Systems, Inc. | Ensuring data persistence and consistency in enterprise storage backup systems |
| CN102937931A (en) * | 2012-09-25 | 2013-02-20 | 北京奇虎科技有限公司 | Method and system of log data transmission |
| CN104699592A (en) * | 2012-09-25 | 2015-06-10 | 北京奇虎科技有限公司 | Log data transmission method and log data transmission system |
| CN104714880A (en) * | 2012-09-25 | 2015-06-17 | 北京奇虎科技有限公司 | Log data transmission method and system as well as log server |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105516349A (en) * | 2016-01-04 | 2016-04-20 | 陈华锋 | File transmission method and system |
| CN107229554A (en) * | 2016-03-25 | 2017-10-03 | 展讯通信(天津)有限公司 | Mobile terminal stability test method and device |
| CN107247767A (en) * | 2017-06-05 | 2017-10-13 | 山东浪潮通软信息科技有限公司 | A kind of method and device that database is imported by formatted data files |
| CN107247767B (en) * | 2017-06-05 | 2020-08-04 | 浪潮通用软件有限公司 | Method and device for importing formatted data file into database |
| CN107562882A (en) * | 2017-09-04 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of method of data synchronization and device based on log analysis |
| CN110389883A (en) * | 2019-06-27 | 2019-10-29 | 西安联乘智能科技有限公司 | A kind of module log real-time monitoring system based on multithreading |
| CN110389883B (en) * | 2019-06-27 | 2023-08-01 | 西安联乘智能科技有限公司 | Module log real-time monitoring system based on multithreading |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109034993B (en) | Account checking method, account checking equipment, account checking system and computer readable storage medium | |
| US11455217B2 (en) | Transaction consistency query support for replicated data from recovery log to external data stores | |
| US9787706B1 (en) | Modular architecture for analysis database | |
| CN104166600B (en) | Backup and recovery method and device | |
| CN102937931B (en) | Method and system of log data transmission | |
| CN105159820A (en) | Transmission method and device of system log data | |
| US9361343B2 (en) | Method for parallel mining of temporal relations in large event file | |
| US20170344433A1 (en) | Apparatus and method for data migration | |
| CN103699580A (en) | Database synchronization method and database synchronization device | |
| CN110019267A (en) | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium | |
| CN102970158A (en) | Log storage and processing method and log server | |
| CN104809201A (en) | Database synchronization method and device | |
| CN106354817B (en) | Log processing method and device | |
| CN102456076A (en) | Massive fragment data aggregation system and method | |
| CN110019873B (en) | Face data processing method, device and equipment | |
| CN103701906A (en) | Distributed real-time calculation system and data processing method thereof | |
| CN104077420A (en) | Method and device for importing data into HBase database | |
| CN104102701A (en) | Hive-based method for filing and inquiring historical data | |
| CN109145040A (en) | A kind of data administering method based on double message queues | |
| CN104699592A (en) | Log data transmission method and log data transmission system | |
| CN103716384A (en) | Method and device for realizing cloud storage data synchronization in cross-data-center manner | |
| CN109992469B (en) | Method and device for merging logs | |
| CN104965835A (en) | Method and apparatus for reading and writing files of a distributed file system | |
| CN103678314A (en) | Mass data processing system, equipment and method based on association rule extraction | |
| CN104714880A (en) | Log data transmission method and system as well as log server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20151216 |