CN105306261A - Method, device and system for collecting logs - Google Patents
Method, device and system for collecting logs Download PDFInfo
- Publication number
- CN105306261A CN105306261A CN201510632220.0A CN201510632220A CN105306261A CN 105306261 A CN105306261 A CN 105306261A CN 201510632220 A CN201510632220 A CN 201510632220A CN 105306261 A CN105306261 A CN 105306261A
- Authority
- CN
- China
- Prior art keywords
- node
- log
- daily record
- log collection
- center
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/085—Retrieval of network configuration; Tracking network configuration history
- H04L41/0853—Retrieval of network configuration; Tracking network configuration history by actively collecting configuration information or by backing up configuration information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/085—Retrieval of network configuration; Tracking network configuration history
- H04L41/0853—Retrieval of network configuration; Tracking network configuration history by actively collecting configuration information or by backing up configuration information
- H04L41/0856—Retrieval of network configuration; Tracking network configuration history by actively collecting configuration information or by backing up configuration information by backing up or archiving configuration information
Abstract
The embodiments of the invention provide a method, device and system for collecting logs. The method comprises the following steps: each log collecting node acquires a preset target center node list; after obtaining a to-be-uploaded log, each log collecting node takes a try one by one according to indentifying information in the target center node list so as to upload the to-be-uploaded log to a target center node in the target center node list until the to-be-uploaded log is successfully uploaded to a target center node in the target center node list; each log collecting sub-center node forwards the collected log to a log collecting center node; and the log collecting center node stores logs collected from all the log collecting nodes and all the log collecting sub-center nodes. With adoption of the embodiments of the invention, the possibility of successfully collecting logs can be improved greatly and the quantity of omitted logs can be reduced.
Description
Technical field
The present invention relates to networking technology area, particularly relate to a kind of method of collector journal, Apparatus and system.
Background technology
With the develop rapidly of Internet technology, user, by carrying out statistical analysis to basic data, can obtain required information.Usually, basic data obtains by collecting the daily record produced in network.Visible, the daily record situation collected by result collection system is related to the reliability carrying out analyzing acquired results for basic data.That is, log collection is a very important link.
Fig. 1 is the configuration diagram of the system of a kind of collector journal of the prior art, within the system, comprises multiple log collection node, multiple log collection branch center node and 1 log collection Centroid.For the system architecture shown in Fig. 1, there is a kind of like this mode of collector journal: each collector node in result collection system binds 1 target's center's node in advance, in the process of log collection, the daily record of generation is uploaded to the target's center's node bound in advance by each collector node, and stores the daily record of having uploaded.Wherein, target's center's node can comprise: log collection Centroid or log collection branch center node.
Each collector node and target's center's node binding in Fig. 1, therefore, once the target's center's nodes break down corresponding to collector node, just produced daily record cannot be collected in time, cause there is time delay during collector journal, daily record also can be caused time serious to lose, and user needs manually to change target's center's node to recover the collection of this collector node to daily record.
Summary of the invention
The object of the embodiment of the present invention is the method, the Apparatus and system that provide a kind of collector journal, to realize the possibility improving daily record successful collection, reduces the quantity of the omission daily record produced.
For achieving the above object, the embodiment of the invention discloses a kind of method of collector journal, be applied in the result collection system comprising multiple log collection node, multiple log collection branch center node and 1 log collection Centroid, described method comprises step:
Each log collection node obtains the target's center's node listing preset; Described target's center node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node;
Each log collection node is after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing;
The daily record received is forwarded to log collection Centroid by each log collection branch center node;
Described log collection Centroid is by the daily record received from each log collection node and receive daily record from log collection branch center node and store.
Preferably, described each log collection node obtains the step of the target's center's node listing preset, for:
Each log collection node obtains the target's center's node listing preset self stored; Or,
Described result collection system also comprises: log interface node, and described log interface node stores default target's center's node listing; Each log collection node obtains default target's center's node listing from described log interface node.
Preferably, described result collection system also comprises: log-file information memory node and daily record memory node;
Described log packet to be uploaded contains: journal file and log-file information;
Described log collection Centroid receives by the daily record received from each log collection node with from log collection branch center node the step that daily record carries out storing, and comprises:
The daily record received from each log collection node and receive daily record from log collection branch center node, extracts described log-file information and is sent to log-file information memory node and stores, and described journal file is stored to daily record memory node.
Preferably, the label information whether successfully uploaded is comprised in described log-file information;
Described log interface node obtains the log-file information that each log collection node is uploaded further from described log-file information memory node;
And according to the label information whether successfully uploaded described in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded;
When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
Preferably, described result collection system also comprises: log analysis node;
Described described journal file is stored to daily record memory node after, also comprise:
The log-file information that log analysis node stores according to log-file information memory node, obtains the quantity N1 of the journal file that log collection Centroid receives every day;
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
For achieving the above object, the embodiment of the invention discloses a kind of device of collector journal, be applied in the result collection system comprising multiple log collection node, multiple log collection branch center node and 1 log collection Centroid, described device comprises:
List obtains module, is arranged in each log collection node, for obtaining default target's center's node listing; Described target's center node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node;
Transmission module in daily record, be arranged in each log collection node, for after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing;
Daily record forwarding module, is arranged in the node of each log collection branch center, for the daily record received is forwarded to log collection Centroid;
Daily record memory module, is arranged in described log collection Centroid, for by the daily record received from each log collection node with receive daily record from log collection branch center node and store.
Preferably, described list obtains module, specifically for:
Each log collection node obtains the target's center's node listing preset self stored; Or,
Each log collection node obtains default target's center's node listing from the log interface node preset.
Preferably: described log packet to be uploaded contains: journal file and log-file information;
Described daily record memory module, specifically for the daily record that receives from each log collection node with receive daily record from log collection branch center node, extract described log-file information to be sent to default log-file information memory node and to store, and described journal file is stored to default daily record memory node.
Preferably, omission daily record retransmission module is also comprised;
The label information whether successfully uploaded is comprised in described log-file information;
Described omission daily record retransmission module, is arranged in described log interface node, for: from described log-file information memory node, obtain the log-file information that each log collection node is uploaded; And according to the label information whether successfully uploaded described in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded; When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
Preferably, also comprise: integrity degree computing module;
Described integrity degree computing module, be arranged in default log analysis node, for: described described journal file is stored to daily record memory node after, according to the log-file information that log-file information memory node stores, obtain the quantity N1 of the journal file that log collection Centroid receives every day;
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
For achieving the above object, the embodiment of the invention discloses a kind of result collection system, comprising multiple log collection node, multiple log collection branch center node and 1 log collection Centroid: also comprise log interface node;
Described each log collection node obtains the target's center's node listing preset; Described target's center node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node; And each log collection node is after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing;
Described log interface node, for storing default target's center's node listing, makes each log collection node obtain default target's center's node listing from described log interface node;
The daily record received is forwarded to log collection Centroid by described multiple log collection branch centers node;
Described log collection Centroid is by the daily record received from each log collection node and receive daily record from log collection branch center node and store.
Preferably, also comprise: log-file information memory node and daily record memory node;
Described log packet to be uploaded contains: journal file and log-file information;
The daily record that described log collection Centroid receives from each log collection node and receive daily record from log collection branch center node, extract described log-file information to be sent to log-file information memory node and to store, and described journal file is stored to described daily record memory node.
Preferably, the label information whether successfully uploaded is comprised in described log-file information;
Described log interface node, obtains the log-file information that each log collection node is uploaded further from described log-file information memory node; And according to the label information whether successfully uploaded described in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded; When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
Preferably, this system also comprises: log analysis node;
Described log analysis node, for: described described journal file is stored to daily record memory node after, according to the log-file information that log-file information memory node stores, obtain the quantity N1 of the journal file that log collection Centroid receives every day;
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
Method, the Apparatus and system of a kind of collector journal that the embodiment of the present invention provides, can in advance to each log collection Joint Enterprise target's center's node listing, each log collection node can according to the order of the target's center's node in this list, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing.Like this, greatly can improve daily record by the possibility of successful collection, certainly also reduce the quantity of produced omission daily record, can collect the daily record that each log collection node produces every day in time.Certainly, arbitrary product of the present invention is implemented or method must not necessarily need to reach above-described all advantages simultaneously.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the configuration diagram of the system of a kind of collector journal of the prior art;
The schematic flow sheet of the method for a kind of collector journal that Fig. 2 provides for the embodiment of the present invention;
The structural representation of the device of a kind of collector journal that Fig. 3 provides for the embodiment of the present invention;
The configuration diagram of the system of a kind of collector journal that Fig. 4 provides for the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
The schematic flow sheet of the method for a kind of collector journal that Fig. 2 provides for the embodiment of the present invention, be applied in the result collection system comprising multiple log collection node, multiple log collection branch center node and 1 log collection Centroid, the method can comprise the following steps:
Step S101: each log collection node obtains the target's center's node listing preset; Described target's center node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node.
Wherein, each log collection node obtains the step of the target's center's node listing preset, and can comprise following two kinds of modes:
Mode one: each log collection node obtains the target's center's node listing preset self stored;
Mode two: this result collection system also comprises: log interface node, this log interface node stores default target's center's node listing; Each log collection node obtains default target's center's node listing from this log interface node.
It should be noted that, here " identification information " that relate to can be the information such as IP address, MAC Address of log collection Centroid and/or multiple log collection branch centers node, the present invention does not need to limit this identification information, and the identification information be applicable to determined by those skilled in the art's needs according to practical application.
Step S102: each log collection node is after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing.
In one preferred embodiment of the invention, upload probability of success situation according to the log collection node added up in advance and target's center's node, the target's center's node in target's center's node listing is arranged according to order from high to low; At each log collection node after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing.
It should be noted that, with the of the prior art man-to-man single node network pattern shown in Fig. 1 unlike, log collection node in this step and be a kind of network schemer of one-to-many between target's center's node, log collection node can attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing successively, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing.Like this, add daily record by the possibility successfully uploaded, reduce the quantity of omitting daily record, also improve daily record by the possibility of collecting in time.
In actual applications, the whole log collection produced by the log collection node in result collection system the previous day before being usually defined in 6 o'clock to 12 o'clock of next day are complete.The method that the embodiment of the present invention provides, daily record can be added by the possibility successfully uploaded, reduce the quantity of omitting daily record, therefore, it is possible to improve embodiment illustrated in fig. 1 in the result collection system of man-to-man single node network pattern can not at the appointed time by the problem of log collection to daily record memory node.
Also it should be noted that, upload for the target's center's node in target's center's list at log collection node, when the uploading speed of the target's center's node attempted is too slow or when uploading time-out, will attempt next target's center's node of this target's center's node.If each log collection node the target's center's node in the target's center's list corresponded all is attempted complete and all daily record to be uploaded successfully cannot be uploaded time, first this daily record to be uploaded can be saved to this log collection node, this omission daily record is uploaded to target's center's node when next time uploads as omission daily record.
Step S103: the daily record received is forwarded to log collection Centroid by each log collection branch center node.
Step S104: described log collection Centroid is by the daily record received from each log collection node and receive daily record from log collection branch center node and store.
In a kind of specific embodiment of the present invention, this result collection system can also comprise: log-file information memory node and daily record memory node;
Wherein, this daily record to be uploaded can comprise: journal file and log-file information;
This log collection Centroid receives by the daily record received from each log collection node with from log collection branch center node the step that daily record carries out storing, and can comprise:
The daily record received from each log collection node and receive daily record from log collection branch center node, extracts this log-file information and is sent to log-file information memory node and stores, and this journal file is stored to daily record memory node.
Further, the label information whether successfully uploaded can be comprised in this log-file information;
When comprising log interface node in this result collection system, this log interface node obtains the log-file information that each log collection node is uploaded further from this log-file information memory node;
And according to the label information whether successfully uploaded comprised in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded;
When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
In addition, this result collection system can also comprise: log analysis node;
After this journal file is stored to daily record memory node, can also comprise the following steps:
The log-file information that log analysis node stores according to log-file information memory node, obtains the quantity N1 of the journal file that log collection Centroid receives every day;
It should be noted that, in this log-file information, can also comprise: the file name of daily record, collect the identification information of target's center's node of this daily record.Concrete, log analysis node here can according to the file name etc. in the log-file information stored in this log-file information memory node, obtains the quantity N1 of the journal file that this log collection Centroid receives every day.
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
For example, here the daily record generation rule preset mentioned, can be: within every 5 minutes, produce a journal file that so, the quantity N2 ＇ of the journal file that each log collection node in this Log Collect System can produce every day is: (24*60)/5=288.Easy understand, the quantity N2 of the journal file that all log collection node common properties every day is raw is the quantity N2 ＇ sum of the journal file that all log collection nodes can produce every day, suppose, the quantity of the log collection node in this result collection system is 100, so, the quantity N2 of the journal file that this result collection system common property every day is raw is calculated as 28800.
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
The application embodiment of the present invention, can in advance to each log collection Joint Enterprise target's center's node listing, each log collection node can according to the order of the target's center's node in this list, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing.Compared with respectively daily record being uploaded to fixing target's center's node with log collection node each in prior art, greatly can improving daily record by the possibility of successful collection, certainly also reduce the quantity of produced omission daily record.
In addition, can also by calculate every day log collection the collection degree of integrity degree to the daily record that each log collection node produces weigh, so that according to the collection degree of daily record further to the adjustment that this result collection system is correlated with.
Certainly, arbitrary product of the present invention is implemented or method must not necessarily need to reach above-described all advantages simultaneously.
The structural representation of the device of a kind of collector journal that Fig. 3 provides for the embodiment of the present invention, be applied in the result collection system comprising multiple log collection node, multiple log collection branch center node and 1 log collection Centroid, this device can comprise with lower module: list obtains transmission module 220, daily record forwarding module 230 and daily record memory module 240 in module 210, daily record; Wherein,
List obtains module 210, is arranged in each log collection node, for obtaining default target's center's node listing; Described target's center node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node;
Transmission module 220 in daily record, be arranged in each log collection node, for after list acquisition module 210 obtains daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing;
Daily record forwarding module 230, is arranged in the node of each log collection branch center, for the daily record received is forwarded to log collection Centroid;
Daily record memory module 240, is arranged in described log collection Centroid, for by the daily record received from each log collection node with receive daily record from log collection branch center node and store.
Wherein, list obtains module 210, specifically for:
Each log collection node obtains the target's center's node listing preset self stored; Or,
Each log collection node obtains default target's center's node listing from the log interface node preset.
In a kind of specific embodiment of the present invention, this daily record to be uploaded that in daily record, transmission module 220 is uploaded can comprise: journal file and log-file information;
Daily record memory module 240, specifically for the daily record that receives from each log collection node with receive daily record from log collection branch center node, extract described log-file information to be sent to default log-file information memory node and to store, and described journal file is stored to default daily record memory node.
In another kind of specific embodiment of the present invention, can also comprise in this result collection system and omit daily record retransmission module;
The label information whether successfully uploaded can be comprised in this log-file information;
This omission daily record retransmission module, is arranged in this log interface node, for: from this log-file information memory node, obtain the log-file information that each log collection node is uploaded; And according to the label information whether successfully uploaded described in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded; When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
In another kind of specific embodiment of the present invention, in this result collection system, integrity degree computing module can also be comprised;
Described integrity degree computing module, be arranged in default log analysis node, for: after this journal file is stored to daily record memory node, according to the log-file information that log-file information memory node stores, obtain the quantity N1 of the journal file that log collection Centroid receives every day;
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
It should be noted that, above-mentioned integrity degree computing module 260, can also be arranged in the independent server of in this result collection system one or in node, the embodiment of the present invention does not need to limit the position set by this integrity degree computing module, and any possible implementation all can be applied to the present invention.
The application embodiment of the present invention, can in advance to each log collection Joint Enterprise target's center's node listing, each log collection node can according to the order of the target's center's node in this list, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing.Compared with respectively daily record being uploaded to fixing target's center's node with log collection node each in prior art, greatly can improving daily record by the possibility of successful collection, certainly also reduce the quantity of produced omission daily record.
In addition, can also by calculate every day log collection the collection degree of integrity degree to the daily record that each log collection node produces weigh, so that according to the collection degree of daily record further to the adjustment that this result collection system is correlated with.
Certainly, arbitrary product of the present invention is implemented or method must not necessarily need to reach above-described all advantages simultaneously.
The configuration diagram of the system of a kind of collector journal that Fig. 4 provides for the embodiment of the present invention, be applied in the result collection system comprising multiple log collection node, multiple log collection branch center node and 1 log collection Centroid, this system can also comprise: log interface node;
Each log collection node obtains the target's center's node listing preset; This target's center's node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node; And each log collection node is after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing;
This log interface node, for storing default target's center's node listing, makes each log collection node obtain default target's center's node listing from described log interface node;
The daily record received is forwarded to log collection Centroid by multiple log collection branch centers node;
Log collection Centroid is by the daily record received from each log collection node and receive daily record from log collection branch center node and store.
In a kind of specific embodiment of the present invention, this result collection system can also comprise: log-file information memory node and daily record memory node;
This log packet to be uploaded contains: journal file and log-file information;
The daily record that this log collection Centroid receives from each log collection node and receive daily record from log collection branch center node, extract described log-file information to be sent to log-file information memory node and to store, and described journal file is stored to described daily record memory node.
In another kind of specific embodiment of the present invention, the label information whether successfully uploaded in this log-file information, can be comprised;
This log interface node, obtains the log-file information that each log collection node is uploaded further from this log-file information memory node; And according to the label information whether successfully uploaded described in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded; When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
In another kind of specific embodiment of the present invention, can also comprise in this result collection system: log analysis node;
Above-mentioned log analysis node, for: after this journal file is stored to daily record memory node, according to the log-file information that log-file information memory node stores, obtain the quantity N1 of the journal file that log collection Centroid receives every day;
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
The application embodiment of the present invention, can in advance to each log collection Joint Enterprise target's center's node listing, each log collection node can according to the order of the target's center's node in this list, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing.Compared with respectively daily record being uploaded to fixing target's center's node with log collection node each in prior art, greatly can improving daily record by the possibility of successful collection, certainly also reduce the quantity of produced omission daily record.
In addition, can also by calculate every day log collection the collection degree of integrity degree to the daily record that each log collection node produces weigh, so that according to the collection degree of daily record further to the adjustment that this result collection system is correlated with.
Certainly, arbitrary product of the present invention is implemented or method must not necessarily need to reach above-described all advantages simultaneously.
For system or device embodiment, because it is substantially similar to embodiment of the method, so description is fairly simple, relevant part illustrates see the part of embodiment of the method.
It should be noted that, in this article, the such as relational terms of first and second grades and so on is only used for an entity or operation to separate with another entity or operating space, and not necessarily requires or imply the relation that there is any this reality between these entities or operation or sequentially.And, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment comprising described key element and also there is other identical element.
One of ordinary skill in the art will appreciate that all or part of step realized in said method execution mode is that the hardware that can carry out instruction relevant by program has come, described program can be stored in computer read/write memory medium, here the alleged storage medium obtained, as: ROM/RAM, magnetic disc, CD etc.
The foregoing is only preferred embodiment of the present invention, be not intended to limit protection scope of the present invention.All any amendments done within the spirit and principles in the present invention, equivalent replacement, improvement etc., be all included in protection scope of the present invention.
Claims (14)
1. a method for collector journal, be applied in the result collection system comprising multiple log collection node, multiple log collection branch center node and 1 log collection Centroid, it is characterized in that, described method comprises step:
Each log collection node obtains the target's center's node listing preset; Described target's center node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node;
Each log collection node is after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing;
The daily record received is forwarded to log collection Centroid by each log collection branch center node;
Described log collection Centroid is by the daily record received from each log collection node and receive daily record from log collection branch center node and store.
2. method according to claim 1, is characterized in that, described each log collection node obtains the step of the target's center's node listing preset, for:
Each log collection node obtains the target's center's node listing preset self stored; Or,
Described result collection system also comprises: log interface node, and described log interface node stores default target's center's node listing; Each log collection node obtains default target's center's node listing from described log interface node.
3. method according to claim 2, is characterized in that, described result collection system also comprises: log-file information memory node and daily record memory node;
Described log packet to be uploaded contains: journal file and log-file information;
Described log collection Centroid receives by the daily record received from each log collection node with from log collection branch center node the step that daily record carries out storing, and comprises:
The daily record received from each log collection node and receive daily record from log collection branch center node, extracts described log-file information and is sent to log-file information memory node and stores, and described journal file is stored to daily record memory node.
4. method according to claim 3, is characterized in that, comprises the label information whether successfully uploaded in described log-file information;
Described log interface node obtains the log-file information that each log collection node is uploaded further from described log-file information memory node;
And according to the label information whether successfully uploaded described in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded;
When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
5. method according to claim 3, is characterized in that, described result collection system also comprises: log analysis node;
Described described journal file is stored to daily record memory node after, also comprise:
The log-file information that log analysis node stores according to log-file information memory node, obtains the quantity N1 of the journal file that log collection Centroid receives every day;
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
6. a device for collector journal, be applied in the result collection system comprising multiple log collection node, multiple log collection branch center node and 1 log collection Centroid, it is characterized in that, described device comprises:
List obtains module, is arranged in each log collection node, for obtaining default target's center's node listing; Described target's center node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node;
Transmission module in daily record, be arranged in each log collection node, for after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing;
Daily record forwarding module, is arranged in the node of each log collection branch center, for the daily record received is forwarded to log collection Centroid;
Daily record memory module, is arranged in described log collection Centroid, for by the daily record received from each log collection node with receive daily record from log collection branch center node and store.
7. device according to claim 6, is characterized in that, described list obtains module, specifically for:
Each log collection node obtains the target's center's node listing preset self stored; Or,
Each log collection node obtains default target's center's node listing from the log interface node preset.
8. device according to claim 7, is characterized in that:
Described log packet to be uploaded contains: journal file and log-file information;
Described daily record memory module, specifically for the daily record that receives from each log collection node with receive daily record from log collection branch center node, extract described log-file information to be sent to default log-file information memory node and to store, and described journal file is stored to default daily record memory node.
9. device according to claim 8, is characterized in that, also comprises and omits daily record retransmission module;
The label information whether successfully uploaded is comprised in described log-file information;
Described omission daily record retransmission module, is arranged in described log interface node, for: from described log-file information memory node, obtain the log-file information that each log collection node is uploaded; And according to the label information whether successfully uploaded described in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded; When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
10. device according to claim 8, is characterized in that, also comprises: integrity degree computing module;
Described integrity degree computing module, be arranged in default log analysis node, for: described described journal file is stored to daily record memory node after, according to the log-file information that log-file information memory node stores, obtain the quantity N1 of the journal file that log collection Centroid receives every day;
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
11. 1 kinds of result collection systems, comprise multiple log collection node, multiple log collection branch center node and 1 log collection Centroid, it is characterized in that: also comprise log interface node;
Described each log collection node obtains the target's center's node listing preset; Described target's center node listing comprises the identification information of log collection Centroid and/or multiple log collection branch centers node; And each log collection node is after acquisition daily record to be uploaded, respectively according to the described identification information in obtained described target's center node listing, attempt target's center's node daily record to be uploaded be uploaded in target's center's node listing one by one, until successfully daily record to be uploaded be uploaded in target's center's node in target's center's node listing;
Described log interface node, for storing default target's center's node listing, makes each log collection node obtain default target's center's node listing from described log interface node;
The daily record received is forwarded to log collection Centroid by described multiple log collection branch centers node;
Described log collection Centroid is by the daily record received from each log collection node and receive daily record from log collection branch center node and store.
12. systems according to claim 11, is characterized in that, also comprise: log-file information memory node and daily record memory node;
Described log packet to be uploaded contains: journal file and log-file information;
The daily record that described log collection Centroid receives from each log collection node and receive daily record from log collection branch center node, extract described log-file information to be sent to log-file information memory node and to store, and described journal file is stored to described daily record memory node.
13. systems according to claim 9, is characterized in that, comprise the label information whether successfully uploaded in described log-file information;
Described log interface node, obtains the log-file information that each log collection node is uploaded further from described log-file information memory node; And according to the label information whether successfully uploaded described in log-file information, judge whether there is omission daily record in the daily record that each log collection node is uploaded; When the daily record that daily record collector node is uploaded exists omission daily record, notify that this log collection node retransmits and omit daily record.
14. systems according to claim 10, is characterized in that, this system also comprises: log analysis node;
Described log analysis node, for: described described journal file is stored to daily record memory node after, according to the log-file information that log-file information memory node stores, obtain the quantity N1 of the journal file that log collection Centroid receives every day;
According to the daily record generation rule preset, obtain the quantity N2 of the raw journal file of all log collection node common properties every day;
According to formula: (N1/N2) * 100%, calculate the integrity degree of log collection every day.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510632220.0A CN105306261A (en) | 2015-09-29 | 2015-09-29 | Method, device and system for collecting logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510632220.0A CN105306261A (en) | 2015-09-29 | 2015-09-29 | Method, device and system for collecting logs |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105306261A true CN105306261A (en) | 2016-02-03 |
Family
ID=55203045
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510632220.0A Pending CN105306261A (en) | 2015-09-29 | 2015-09-29 | Method, device and system for collecting logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105306261A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106991177A (en) * | 2017-04-06 | 2017-07-28 | 广东亿迅科技有限公司 | The nondestructive collection method and its system of real-time logs under a kind of distributed environment |
| CN107222541A (en) * | 2017-06-06 | 2017-09-29 | 南京南瑞继保电气有限公司 | Delivery method and system on a kind of new energy power prediction file |
| CN107360242A (en) * | 2017-07-27 | 2017-11-17 | 郑州云海信息技术有限公司 | The method for down loading and system of a kind of journal file |
| CN107423188A (en) * | 2016-03-07 | 2017-12-01 | 阿里巴巴集团控股有限公司 | Log processing method and equipment |
| CN109104327A (en) * | 2018-06-08 | 2018-12-28 | 阿里巴巴集团控股有限公司 | A kind of business diary generation method, device and equipment |
| CN109213742A (en) * | 2018-11-02 | 2019-01-15 | 大唐网络有限公司 | Log collection method and device |
| CN110716841A (en) * | 2019-09-17 | 2020-01-21 | 香港乐蜜有限公司 | Monitoring data collection method, device and equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN102750326A (en) * | 2012-05-30 | 2012-10-24 | 浪潮电子信息产业股份有限公司 | Log management optimization method of cluster system based on downsizing strategy |
| CN102780777A (en) * | 2012-07-19 | 2012-11-14 | 北京蓝汛通信技术有限责任公司 | Log collecting method and log collecting system |
| CN103856354A (en) * | 2014-03-07 | 2014-06-11 | 浪潮电子信息产业股份有限公司 | Method for achieving unified management of logs of cluster storage system |
| CN104023083A (en) * | 2014-06-23 | 2014-09-03 | 广东睿江科技有限公司 | Method and device for log collection cluster load balance |
| CN104579777A (en) * | 2015-01-06 | 2015-04-29 | 浪潮软件股份有限公司 | Log management system and implementation method |
-
2015
- 2015-09-29 CN CN201510632220.0A patent/CN105306261A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| CN102750326A (en) * | 2012-05-30 | 2012-10-24 | 浪潮电子信息产业股份有限公司 | Log management optimization method of cluster system based on downsizing strategy |
| CN102780777A (en) * | 2012-07-19 | 2012-11-14 | 北京蓝汛通信技术有限责任公司 | Log collecting method and log collecting system |
| CN103856354A (en) * | 2014-03-07 | 2014-06-11 | 浪潮电子信息产业股份有限公司 | Method for achieving unified management of logs of cluster storage system |
| CN104023083A (en) * | 2014-06-23 | 2014-09-03 | 广东睿江科技有限公司 | Method and device for log collection cluster load balance |
| CN104579777A (en) * | 2015-01-06 | 2015-04-29 | 浪潮软件股份有限公司 | Log management system and implementation method |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107423188A (en) * | 2016-03-07 | 2017-12-01 | 阿里巴巴集团控股有限公司 | Log processing method and equipment |
| CN107423188B (en) * | 2016-03-07 | 2021-05-07 | 阿里巴巴集团控股有限公司 | Log processing method and device |
| CN106991177A (en) * | 2017-04-06 | 2017-07-28 | 广东亿迅科技有限公司 | The nondestructive collection method and its system of real-time logs under a kind of distributed environment |
| CN106991177B (en) * | 2017-04-06 | 2020-11-13 | 广东亿迅科技有限公司 | Lossless acquisition method and system for real-time logs in distributed environment |
| CN107222541A (en) * | 2017-06-06 | 2017-09-29 | 南京南瑞继保电气有限公司 | Delivery method and system on a kind of new energy power prediction file |
| CN107222541B (en) * | 2017-06-06 | 2020-10-16 | 南京南瑞继保电气有限公司 | New energy power prediction file uploading method and system |
| CN107360242A (en) * | 2017-07-27 | 2017-11-17 | 郑州云海信息技术有限公司 | The method for down loading and system of a kind of journal file |
| CN109104327A (en) * | 2018-06-08 | 2018-12-28 | 阿里巴巴集团控股有限公司 | A kind of business diary generation method, device and equipment |
| CN109213742A (en) * | 2018-11-02 | 2019-01-15 | 大唐网络有限公司 | Log collection method and device |
| CN110716841A (en) * | 2019-09-17 | 2020-01-21 | 香港乐蜜有限公司 | Monitoring data collection method, device and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105306261A (en) | Method, device and system for collecting logs | |
| US7827299B2 (en) | Transitioning between historical and real time data streams in the processing of data change messages | |
| CN107977473B (en) | Logback-based distributed system log retrieval method and system | |
| CN111338893B (en) | Process log processing method, device, computer equipment and storage medium | |
| CN109302495A (en) | A kind of date storage method and device | |
| CN104584524A (en) | Aggregating data in a mediation system | |
| CN110995513A (en) | Data sending and receiving method in Internet of things system, Internet of things equipment and platform | |
| CN104202387A (en) | Metadata recovery method and related device | |
| CN103346960A (en) | Recognizable figure-based data transmission method across networks | |
| CN104239353B (en) | WEB classification control and log audit method | |
| CN105447146A (en) | Massive data collecting and exchanging system and method | |
| CN113326165B (en) | Data processing method and device based on block chain and computer readable storage medium | |
| CN105357041A (en) | Edge node server, and log file uploading method and system | |
| CN105245392A (en) | Method and device for base line checking and repairing | |
| WO2013127147A1 (en) | File synchronization system and method | |
| CN105653209A (en) | Object storage data transmitting method and device | |
| CN110460668B (en) | File uploading method and device, computer equipment and storage medium | |
| CN106126419A (en) | The adjustment method of a kind of application program and device | |
| CN103927252A (en) | Cross-component log recording method, device and system | |
| DE112012000305T5 (en) | Joint restoration of data sources | |
| CN103109498B (en) | For the method concentrated the event of multi-level hierarchical computer management system | |
| CN105468502A (en) | Log collection method, device and system | |
| CN111367916B (en) | Data storage method and device | |
| JP2014035595A (en) | Testing device for communication system, testing program for communication system, and testing method for communication system | |
| CN103929763A (en) | Method for comparison and reconstruction of geographic redundancy database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160203 |
|
| RJ01 | Rejection of invention patent application after publication |