CN105159820A - Transmission method and device of system log data - Google Patents

Transmission method and device of system log data Download PDF

Info

Publication number
CN105159820A
CN105159820A CN201510561048.4A CN201510561048A CN105159820A CN 105159820 A CN105159820 A CN 105159820A CN 201510561048 A CN201510561048 A CN 201510561048A CN 105159820 A CN105159820 A CN 105159820A
Authority
CN
China
Prior art keywords
file
log
journal file
old
incremental
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201510561048.4A
Other languages
Chinese (zh)
Inventor
张壮
陈良华
杜彦魁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Beijing Electronic Information Industry Co Ltd
Original Assignee
Inspur Beijing Electronic Information Industry Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Beijing Electronic Information Industry Co Ltd filed Critical Inspur Beijing Electronic Information Industry Co Ltd
Priority to CN201510561048.4A priority Critical patent/CN105159820A/en
Publication of CN105159820A publication Critical patent/CN105159820A/en
Pending legal-status Critical Current

Links

Abstract

The invention discloses a transmission method and device of system log data. The method comprises the following steps: fetching log data from an application server; according to the fetched log data, generating a new log file; comparing the new log file with an original log file in a log database, and extracting newly-increased content to serve as an increment log file; and writing the increment log file into the log database, and deleting the log data which is written into the log database from the application server. The transmission method and device of the system log data can fetch the log data for backup, and when the application server shuts down, the log which can not be sent out in time can not be lost. In addition, only one set of database is established in the log server, and stored resources are saved.

Description

A kind of syslog data transmission method and device
Technical field
The present invention relates to syslog data technical field, particularly relate to a kind of syslog data transmission method and device.
Background technology
Daily record (log) can be used for storing and the details of register system ruuning situation and user's service condition, and these information check the Main Basis of system running state and the problem of investigation.Such as, in certain software there is collapse case in application program, so can find corresponding record from program event daily record, thus help person skilled analysis and deal with problems.
For inline system log information, log information can be used for recording service operation process on line and working procedure is monitored, can carry out pointing out, warning for the mistake occurred and may occur by the log information of analytic record, all right statistical service information, for the upgrading of the improved Haniltonian program of service logic provides reliable foundation.
Existing log transmission method, the log information that on application server, each application program module produces sends to daily record client by LIB storehouse, daily record client preserves the daily record received, again according to log server ip address and the associated port number of configuration, by received Log Sender to log server.Log server receives daily record, is saved in unified for received daily record in DB.
This scheme is machine once application server is delayed, and the daily record do not sent in time will be lost, and before log server processes daily record, is the daily record that cannot view current generation.In addition, this scheme will on the application server with log server sets up a set of database respectively, store two cover daily record datas, cause the waste of storage resources.
Summary of the invention
The object of this invention is to provide a kind of syslog data transmission method and device, object is to reduce the risk that daily record data is lost and the redundancy reducing storage.
For solving the problems of the technologies described above, the invention provides a kind of syslog data transmission method, comprising:
From application server crawl log data;
Daily record data according to capturing generates new journal file;
Original journal file in described new journal file and log database is compared, extracts newly-increased content as Incremental Log file;
By in described Incremental Log file write log database, and from described application server, delete the daily record data writing log database.
Alternatively, describedly to comprise from application server crawl log data:
Capture the full dose daily record data on the same day from application server according to predetermined time interval timing.
Alternatively, described original journal file in described new journal file and log database to be compared, extracts newly-increased content and comprise as Incremental Log file:
Judge in described log database, whether to there is the old journal file corresponding with described new journal file;
When there is described old journal file, described new journal file and old journal file being compared, extracting newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
Alternatively, original journal file in described new journal file and log database is being compared, is extracting newly-increased content and also comprise as after Incremental Log file:
Judge in described log database, whether to there is the old Incremental Log file corresponding with current delta journal file;
When there is described old Incremental Log file, current delta journal file is adopted to replace described old Incremental Log file.
Alternatively, described described Incremental Log file write log database to be comprised:
Make a mistake when Incremental Log file writes described log database, the daily record data of write is carried out rollback.
Present invention also offers a kind of syslog data transmitting device, comprising:
Handling module, for from application server crawl log data;
Generation module, for generating new journal file according to the daily record data captured;
Extraction module, for being compared by original journal file in described new journal file and log database, extracts newly-increased content as Incremental Log file;
Writing module, for being write in log database by described Incremental Log file, and deletes the daily record data writing log database from described application server.
Alternatively, described handling module is used for comprising from application server crawl log data:
Described handling module is specifically for capturing the full dose daily record data on the same day from application server according to predetermined time interval timing.
Alternatively, described extraction module comprises:
Whether judging unit, exist the old journal file corresponding with described new journal file for judging in described log database;
Extraction unit, for when there is described old journal file, compares described new journal file and old journal file, extracts newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
Alternatively, also comprise:
Replacement module, for original journal file in described new journal file and log database is being compared, extract newly-increased content as after Incremental Log file, judge in described log database, whether to there is the old Incremental Log file corresponding with current delta journal file; When there is described old Incremental Log file, current delta journal file is adopted to replace described old Incremental Log file.
Be used for described Incremental Log file write log database to comprise in, said write module alternatively:
The daily record data of write, specifically for making a mistake when Incremental Log file writes described log database, is carried out rollback by said write module.
Syslog data transmission method provided by the present invention and device, by crawl log data from application server, generate new journal file; Journal file original in new journal file and log database is compared, extracts newly-increased content as Incremental Log file; By in this Incremental Log file write log database, and from application server, delete the daily record data having write log database.Visible, syslog data transmission method provided by the present invention and device, can back up the daily record data grabbed, and to delay machine at application server, the daily record do not sent in time can not be lost.In addition, the present invention only establishes a set of database at log server, has saved the resource stored.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of a kind of embodiment of syslog data transmission method provided by the present invention;
Fig. 2 is the process flow diagram of the another kind of embodiment of syslog data transmission method provided by the present invention;
Fig. 3 is the structured flowchart of a kind of embodiment of syslog data transmitting device provided by the present invention.
Embodiment
In order to make those skilled in the art person understand the present invention program better, below in conjunction with the drawings and specific embodiments, the present invention is described in further detail.Obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
As shown in Figure 1, the method comprises the process flow diagram of a kind of embodiment of syslog data transmission method provided by the present invention:
Step S101: from application server crawl log data;
Step S102: the daily record data according to capturing generates new journal file;
Step S103: compared by original journal file in described new journal file and log database, extracts newly-increased content as Incremental Log file;
Step S104: by described Incremental Log file write log database, and delete the daily record data writing log database from described application server.
Syslog data transmission method provided by the present invention, by crawl log data from application server, generates new journal file; Journal file original in new journal file and log database is compared, extracts newly-increased content as Incremental Log file; By in this Incremental Log file write log database, and from application server, delete the daily record data having write log database.Visible, syslog data transmission method provided by the present invention, can back up the daily record data grabbed, and to delay machine at application server, the daily record do not sent in time can not be lost.In addition, the present invention only establishes a set of database at log server, has saved the resource stored.
As shown in Figure 2, the method comprises the process flow diagram of the another kind of embodiment of syslog data transmission method provided by the present invention:
Step S201: the full dose daily record data capturing the same day according to predetermined time interval timing from application server;
Step S202: the daily record data according to capturing generates new journal file;
Step S203: judge whether there is the old journal file corresponding with new journal file in log database;
Step S204: when there is old journal file, compares new journal file and old journal file, extracts newly-increased content as Incremental Log file; When there is not old journal file, creating an empty file as old journal file, extracting the content of new journal file as Incremental Log file;
Step S205: by Incremental Log file write log database, and delete the daily record data having write log database from application server;
Step S206: judge whether there is the old Incremental Log file corresponding with current delta journal file in log database; When there is old Incremental Log file, current delta journal file is adopted to replace old Incremental Log file.
Alternatively, described Incremental Log file writes in the process of log database above-mentioned by the syslog data transmission method that the present embodiment provides, make a mistake when Incremental Log file writes described log database, then the daily record data of write is carried out rollback.
Syslog data transmission method provided by the present invention can reduce the risk that daily record data is lost, and reduces the redundancy stored.
For making those skilled in the art understand the present invention better, set forth the processing procedure of the embodiment of the present invention better below by way of a concrete example:
Step S301: set per half an hour log server from crawl log file application server on log server, such as today is on August 31st, 2012, log server just can per half an hour, captures the full dose daily record data of today, such as: GatewayServer_INFO_log.20120831.
Step S302: after log server grabs daily record data, carries out rename to daily record data, the file that called after ends up with new, such as GatewayServer_INFO_log.20120831.new, represents that this file is new journal file.
Step S303: judge whether there is GatewayServer-_INFO_log.20120831.old file in log server, if so, then performs step S305; If not, then step S304 is performed:.
Step S304: log server creates an empty ancient deed, called after GatewayServer-_INFO_log.20120831.old file, goes to step S305.
Step S305: use diff order GatewayServer_INFO_log.20120831.new file and GatewayServer_INFO_log.20120831.old file to be contrasted; Extract new content and generate new comparison document GatewayServer_INFO_log.20120831.temp file.
Step S306: adopt new comparison document GatewayServer_INFO_log.20120831.temp replacement old comparison document GatewayServer_INFO_log.20120831.temp formerly.
Step S307: use sed order to delete the respective explanations information produced by diff order, generates delta file GatewayServer_INFO_log.20120831.final file.
Step S308: adopt described new increment journal file GatewayServer_INFO_log.20120831.final replacement old Incremental Log file GatewayServer_INFO_log.20120831.final formerly.
Step S309: delta file GatewayServer_INFO_log.20120831.final file is formatd and imports database, such as, a line content of this file is [20518] [1190676800] [0AA0000FE33C52012083100155221307] 2012083000:22:08.063INFOUpdateTransLogSuccess, corresponding is process number, thread number, serial number, timestamp, rank, separator, content, 7 row contents, but system directive such as AWK (text processing facilities) etc. can think that foregoing is 8 row, because system can only judge columns according to space for above content, but between the date and time of time row, there is a space, can think that it is two row, need to carry out merging treatment to these two row, form row.Then to go out the literary name section that 6 row contents of decollator are database, generate SQL (Structured Query Language (SQL)), insert log database, often read a line, process once, until run through all files.
Step S310: adopt new journal file GatewayServer_INFO_log.20120831.new to replace old journal file GatewayServer_INFO_log.20120831.old.
Step S311: if making a mistake in Incremental Log file write log database, then the daily record data of write is carried out rollback.
It should be noted that, for embodiment of the method, in order to simple description, therefore it is all expressed as a series of combination of actions, but those skilled in the art should know, the present invention is not by the restriction of described sequence of movement, because according to the present invention, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to preferred embodiment, and involved action and module might not be that the present invention is necessary.
The structured flowchart of a kind of embodiment of syslog data transmitting device provided by the present invention as shown in Figure 3, comprising:
Handling module 100, for from application server crawl log data;
Generation module 200, for generating new journal file according to the daily record data captured;
Extraction module 300, for being compared by original journal file in described new journal file and log database, extracts newly-increased content as Incremental Log file;
Writing module 400, for being write in log database by described Incremental Log file, and deletes the daily record data writing log database from described application server.
Syslog data transmitting device provided by the present invention, by crawl log data from application server, generates new journal file; Journal file original in new journal file and log database is compared, extracts newly-increased content as Incremental Log file; By in this Incremental Log file write log database, and from application server, delete the daily record data having write log database.Visible, syslog data transmitting device provided by the present invention, can back up the daily record data grabbed, and to delay machine at application server, the daily record do not sent in time can not be lost.In addition, the present invention only establishes a set of database at log server, has saved the resource stored.
Wherein, said extracted module may further include:
Whether judging unit, exist the old journal file corresponding with described new journal file for judging in described log database;
Extraction unit, for when there is described old journal file, compares described new journal file and old journal file, extracts newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
Syslog data transmitting device provided by the present invention is corresponding with said method, does not repeat them here.
In this instructions, each embodiment adopts the mode of going forward one by one to describe, and what each embodiment stressed is the difference with other embodiment, between each embodiment same or similar part mutually see.
To the above-mentioned explanation of the disclosed embodiments, professional and technical personnel in the field are realized or uses the present invention.To be apparent for those skilled in the art to the multiple amendment of these embodiments, General Principle as defined herein can without departing from the spirit or scope of the present invention, realize in other embodiments.Therefore, the present invention can not be restricted to these embodiments shown in this article, but will meet the widest scope consistent with principle disclosed herein and features of novelty.

Claims (10)

1. a syslog data transmission method, is characterized in that, comprising:
From application server crawl log data;
Daily record data according to capturing generates new journal file;
Original journal file in described new journal file and log database is compared, extracts newly-increased content as Incremental Log file;
By in described Incremental Log file write log database, and from described application server, delete the daily record data writing log database.
2. the system as claimed in claim 1 daily record data transmission method, is characterized in that, describedly comprises from application server crawl log data:
Capture the full dose daily record data on the same day from application server according to predetermined time interval timing.
3. the system as claimed in claim 1 daily record data transmission method, is characterized in that, is describedly compared by original journal file in described new journal file and log database, extracts newly-increased content and comprises as Incremental Log file:
Judge in described log database, whether to there is the old journal file corresponding with described new journal file;
When there is described old journal file, described new journal file and old journal file being compared, extracting newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
4. syslog data transmission method as claimed in claim 3, is characterized in that, compared by original journal file in described new journal file and log database, extracts newly-increased content and also comprises as after Incremental Log file:
Judge in described log database, whether to there is the old Incremental Log file corresponding with current delta journal file;
When there is described old Incremental Log file, current delta journal file is adopted to replace described old Incremental Log file.
5. the syslog data transmission method as described in any one of Claims 1-4, is characterized in that, is describedly comprised by described Incremental Log file write log database:
Make a mistake when Incremental Log file writes described log database, the daily record data of write is carried out rollback.
6. a syslog data transmitting device, is characterized in that, comprising:
Handling module, for from application server crawl log data;
Generation module, for generating new journal file according to the daily record data captured;
Extraction module, for being compared by original journal file in described new journal file and log database, extracts newly-increased content as Incremental Log file;
Writing module, for being write in log database by described Incremental Log file, and deletes the daily record data writing log database from described application server.
7. syslog data transmitting device as claimed in claim 6, is characterized in that, described handling module is used for comprising from application server crawl log data:
Described handling module is specifically for capturing the full dose daily record data on the same day from application server according to predetermined time interval timing.
8. syslog data transmitting device as claimed in claim 6, it is characterized in that, described extraction module comprises:
Whether judging unit, exist the old journal file corresponding with described new journal file for judging in described log database;
Extraction unit, for when there is described old journal file, compares described new journal file and old journal file, extracts newly-increased content as Incremental Log file; When there is not described old journal file, creating an empty file as old journal file, extracting the content of described new journal file as Incremental Log file.
9. syslog data transmitting device as claimed in claim 8, is characterized in that, also comprise:
Replacement module, for original journal file in described new journal file and log database is being compared, extract newly-increased content as after Incremental Log file, judge in described log database, whether to there is the old Incremental Log file corresponding with current delta journal file; When there is described old Incremental Log file, current delta journal file is adopted to replace described old Incremental Log file.
10. the syslog data transmitting device as described in any one of claim 6 to 9, is characterized in that, said write module is used for described Incremental Log file write log database to comprise:
The daily record data of write, specifically for making a mistake when Incremental Log file writes described log database, is carried out rollback by said write module.
CN201510561048.4A 2015-09-06 2015-09-06 Transmission method and device of system log data Pending CN105159820A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510561048.4A CN105159820A (en) 2015-09-06 2015-09-06 Transmission method and device of system log data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510561048.4A CN105159820A (en) 2015-09-06 2015-09-06 Transmission method and device of system log data

Publications (1)

Publication Number Publication Date
CN105159820A true CN105159820A (en) 2015-12-16

Family

ID=54800682

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510561048.4A Pending CN105159820A (en) 2015-09-06 2015-09-06 Transmission method and device of system log data

Country Status (1)

Country Link
CN (1) CN105159820A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105516349A (en) * 2016-01-04 2016-04-20 陈华锋 File transmission method and system
CN107229554A (en) * 2016-03-25 2017-10-03 展讯通信(天津)有限公司 Mobile terminal stability test method and device
CN107247767A (en) * 2017-06-05 2017-10-13 山东浪潮通软信息科技有限公司 A kind of method and device that database is imported by formatted data files
CN107562882A (en) * 2017-09-04 2018-01-09 郑州云海信息技术有限公司 A kind of method of data synchronization and device based on log analysis

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080059542A1 (en) * 2006-08-30 2008-03-06 Inmage Systems, Inc. Ensuring data persistence and consistency in enterprise storage backup systems
CN102937931A (en) * 2012-09-25 2013-02-20 北京奇虎科技有限公司 Method and system of log data transmission
CN104699592A (en) * 2012-09-25 2015-06-10 北京奇虎科技有限公司 Log data transmission method and log data transmission system
CN104714880A (en) * 2012-09-25 2015-06-17 北京奇虎科技有限公司 Log data transmission method and system as well as log server

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080059542A1 (en) * 2006-08-30 2008-03-06 Inmage Systems, Inc. Ensuring data persistence and consistency in enterprise storage backup systems
CN102937931A (en) * 2012-09-25 2013-02-20 北京奇虎科技有限公司 Method and system of log data transmission
CN104699592A (en) * 2012-09-25 2015-06-10 北京奇虎科技有限公司 Log data transmission method and log data transmission system
CN104714880A (en) * 2012-09-25 2015-06-17 北京奇虎科技有限公司 Log data transmission method and system as well as log server

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105516349A (en) * 2016-01-04 2016-04-20 陈华锋 File transmission method and system
CN107229554A (en) * 2016-03-25 2017-10-03 展讯通信(天津)有限公司 Mobile terminal stability test method and device
CN107247767A (en) * 2017-06-05 2017-10-13 山东浪潮通软信息科技有限公司 A kind of method and device that database is imported by formatted data files
CN107247767B (en) * 2017-06-05 2020-08-04 浪潮通用软件有限公司 Method and device for importing formatted data file into database
CN107562882A (en) * 2017-09-04 2018-01-09 郑州云海信息技术有限公司 A kind of method of data synchronization and device based on log analysis

Similar Documents

Publication Publication Date Title
US9787706B1 (en) Modular architecture for analysis database
CN102937931B (en) Method and system of log data transmission
CN104166600B (en) Backup and recovery method and device
CN105159820A (en) Transmission method and device of system log data
CN109656934B (en) Source Oracle database DDL synchronization method and device based on log analysis
US9361343B2 (en) Method for parallel mining of temporal relations in large event file
CN103699580A (en) Database synchronization method and database synchronization device
CN102970158A (en) Log storage and processing method and log server
US20170351585A1 (en) Transaction consistency query support for replicated data from recovery log to external data stores
CN109034993B (en) Account checking method, account checking equipment, account checking system and computer readable storage medium
CN106354817B (en) Log processing method and device
US20170344433A1 (en) Apparatus and method for data migration
CN104809201A (en) Database synchronization method and device
CN102456076A (en) Massive fragment data aggregation system and method
CN104077420A (en) Method and device for importing data into HBase database
Hurst et al. Social streams blog crawler
CN104699592A (en) Log data transmission method and log data transmission system
CN103716384A (en) Method and device for realizing cloud storage data synchronization in cross-data-center manner
CN104123280A (en) File comparison method and device
CN104965835A (en) Method and apparatus for reading and writing files of a distributed file system
CN104102701A (en) Hive-based method for filing and inquiring historical data
CN104679493A (en) Improved method for process event handling mechanism
CN101257461A (en) Method and apparatus for filtering content based on classification
CN104714880A (en) Log data transmission method and system as well as log server
CN103577604A (en) Image indexing structure for Hadoop distributed type environment

Legal Events

Date Code Title Description
PB01 Publication
C06 Publication
SE01 Entry into force of request for substantive examination
C10 Entry into substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20151216

RJ01 Rejection of invention patent application after publication