CN100518076C - Journal accounting method and system - Google Patents
Journal accounting method and system Download PDFInfo
- Publication number
- CN100518076C CN100518076C CNB2004100002060A CN200410000206A CN100518076C CN 100518076 C CN100518076 C CN 100518076C CN B2004100002060 A CNB2004100002060 A CN B2004100002060A CN 200410000206 A CN200410000206 A CN 200410000206A CN 100518076 C CN100518076 C CN 100518076C
- Authority
- CN
- China
- Prior art keywords
- log
- statistics
- daily record
- statistic
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
- 238000000034 method Methods 0.000 title claims description 27
- 238000009826 distribution Methods 0.000 claims description 20
- 238000012217 deletion Methods 0.000 claims description 4
- 230000037430 deletion Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 abstract description 5
- 238000007619 statistical method Methods 0.000 abstract 1
- 230000002354 daily effect Effects 0.000 description 105
- 230000009545 invasion Effects 0.000 description 68
- 241001269238 Data Species 0.000 description 6
- 238000004321 preservation Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 241000700605 Viruses Species 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 102100036255 Glucose-6-phosphatase 2 Human genes 0.000 description 2
- 101000930907 Homo sapiens Glucose-6-phosphatase 2 Proteins 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 238000012884 algebraic function Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- JLYXXMFPNIAWKQ-UHFFFAOYSA-N γ Benzene hexachloride Chemical compound ClC1C(Cl)C(Cl)C(Cl)C(Cl)C1Cl JLYXXMFPNIAWKQ-UHFFFAOYSA-N 0.000 description 1
Images
Landscapes
- Debugging And Monitoring (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a log statistical method, including log collecting and resolving step, log storing step, intermediate result regular generating step and statistical result generating step, and concretely speaking, collecting log from a device, resolving the received original log sent by the device into the one able to be identified by a log statistical system; collecting statistical dimension data of the log and storing them into a dictionary list, and adding the log as record to a log list; dividing the log into sets according to the time and statistical dimension data, making calculation on the sets and obtaining the intermediate result and then storing it to an intermediate list; determining statistical conditions and obtaining statistical result by the intermediate list. The invention can be applied to information safety field, able to raise log statistical speed and reduce the complexity in maintaining statistical dimension data of the log.
Description
Technical field
The present invention relates to field of computer information security, particularly relate to a kind of processing method and system daily record data.
Background technology
Along with the development of information technology, data volume increases rapidly, and the accumulation of data is also increasing.When the transmission of carrying out data, exchange and processing, fail safe is an important consideration, for this reason, many equipment relevant with information processing (as fire compartment wall, system for monitoring intrusion, router and server etc.) all can produce daily record, wherein write down on the equipment and the various things that take place every day in the network, can understand each equipment and whole network conditions by inquiry and statistics to daily record.
If daily record amount less relatively (hundreds of bar or still less), experienced keeper can be by reading one by one, finds to find wherein unusual the event log of being concerned about and count data.But, because at information security field, numerous aforementioned device every day even all producing daily record all the time, the quantity of daily record has reached several ten thousand of every days even up to a million records, such quantity has exceeded the treatable scope of keeper, and the keeper can't handle these daily records by reading one by one and handle in the time that limits.And the macro-data that from these daily records, counts, as: flow, unauthorized access number of times and invasion number of times of attack or the like, understand whole network conditions for the keeper, it is very important pinpointing the problems.Therefore the statistical function of realizing daily record data in auditing system is very important.
The statistics of daily record all is (Distributive) or (Algebraic) of algebraically that distributes usually.So-called statistics distributes, being meant that it can be added up by following distribution mode obtains: the daily record data that will need to add up is divided into a plurality of daily record data parts, on each part, can utilize operation function to obtain a statistics, and the statistics of all daily record datas can realize by two kinds of approach: use operation function all daily record datas to be added up or each partial log data is calculated with same operation function, the statistics of acquisition is identical.So-called statistics is an algebraically, be meant that it can calculate by an algebraic function with a plurality of parameters, and each parameter all is the statistics of a distribution.For example, the statistics of daily record datas such as flow, unauthorized access number of times and invaded number of times of attack just belongs to distribution, can add up respectively by daily record being divided into a plurality of parts, and then each statistics added up, the result of acquisition with to coming to the same thing that all daily records are added up simultaneously; The statistics that the flow of certain user capture accounts for the percentage of total flow then belongs to algebraically, can not be as the statistics that distributes the each several part daily record be added up respectively again and on this result's basis, add up, can only obtain by calculation of parameter to two distributions of whole daily record datas.
When daily record is added up, need count statistics under each different statistical condition according to the statistics dimension data, different daily records has different statistics dimensions.For example, when the alarm log of invasion attack is added up, the result of statistics can be the number of times that invasion is attacked, the statistics dimension comprises time range, invasion attack name etc., statistics dimension possible value is referred to as to add up dimension data, can count statistics under each different statistical condition (attacking name etc. as time range, invasion) according to these dimension datas.The statistics dimension data comprises two kinds: first kind is to import for the user, for example the time; Second kind is to select from tabulation for the user, and for example name is attacked in invasion.Second kind of statistics dimension data can be further divided into two classes: a class is stable, can not change at system's run duration, and procotol for example, the processing method of this class statistics dimension data is fairly simple; Another kind of is constantly to change, for example name or the like is attacked in user name, virus name and invasion, because there is the upgrading of user's additions and deletions, virus characteristic storehouse and invasion intrusion feature database when in use in each equipment, these statistics dimension datas are constantly to change, can be referred to as the dynamic statistics dimension data, prior art is owing to directly handle log sheet, directly carry out statistical computation from log sheet, make that comparatively the arithmetic speed of complexity and auditing system is lower to the processing of dynamic statistics dimension data.
Simultaneously, along with networks development, and a large amount of servers adopts the DNS repeating query to realize load balancing, makes and can't go and find out what's going on all sidedly from the daily record of single equipment simply usually.So-called DNS repeating query is meant uses a plurality of same roles' server to carry out the service on foreground, made things convenient for the distribution planning and the autgmentability of service, but the distribution of a plurality of servers makes the statistics of daily record become more complicated.Prior art uses log analysis tool such as webalizer that every machine is done log statistic respectively, and carrying out complicated data again gathers, and will influence the operation efficiency of auditing system significantly and to the log statistic result's that belongs to algebraically calculating.Therefore, the centralized collection processing is carried out in the daily record of plurality of devices become more and more important, and this can cause the daily record amount that needs to add up sharply to increase, make the keeper will spend a large amount of time and carry out statistical operation, and safeguard that in auditing system it is very difficult tabulating with the identical dynamic statistics dimension data of each equipment, will increase the complexity of auditing system and influence its arithmetic speed.Therefore solve the speed issue of how accelerating the massive logs statistics and become more and more urgent.
Summary of the invention
Because arithmetic speed is lower and comparatively difficult to the maintenance of dynamic statistics dimension data during prior art statistical log data, the technical problem that the present invention solves is to provide a kind of log statistic method and system, can accelerate the statistics speed of daily record, reduce the complexity that the log statistic dimension data is safeguarded simultaneously.
For this reason, the technical scheme of technical solution problem of the present invention is: a kind of log statistic method is provided, comprises:
Log collection and analyzing step, the slave unit collector journal, the original log that the equipment of receiving is sent resolves to the daily record that can be discerned by the log statistic system;
Step is preserved in daily record, and the statistics dimension data of collector journal also is saved in the dictionary table, and daily record is added in the log sheet as record;
Regularly generate the intermediate object program step, daily record divided set according to time and statistics dimension data, described set is calculated the statistics that distributes and algebraically statistics distribution parameter and the result is saved in middle table;
Generate the statistics step, determine that statistical condition and intermediate list calculate statistics.
The present invention further improves and is: described daily record is preserved step and is comprised the corresponding of the external key of setting up log sheet and dictionary table major key.
Wherein, described regular generation intermediate object program step comprises definite minimum time section and time started and concluding time.
Wherein, described regular generation intermediate object program step comprises deletion existing intermediate object program between time started and concluding time, is used to deposit in new intermediate object program.
Wherein, described intermediate object program comprises the parameter of each distribution of the statistics of the statistics of distribution and algebraically.
Wherein, adopt SQL statement to handle middle table in the described generation statistics step.
The present invention also provides a kind of log statistic system, comprise that original log that the log collection unit that is used for the slave unit collector journal, the equipment that is used for receiving send resolves to the daily record resolution unit of the daily record that can be discerned by the log statistic system, is used for daily record is preserved unit and log statistic unit as the daily record that record adds log sheet to, described daily record is preserved the unit and also is used for the statistics dimension data of collector journal and is saved in dictionary table; Described log statistic unit is used for daily record is divided set according to time and statistics dimension data, described set is calculated the statistics that distributes and algebraically statistics distribution parameter and the result is saved in middle table, determine that statistical condition and intermediate list calculate statistics.
Wherein, the corresponding of external key that the unit also is used to set up log sheet and dictionary table major key preserved in described daily record.Wherein, described statistic unit also is used for determining minimum time section and time started and concluding time.
Wherein, described statistic unit also is used to delete between time started and concluding time existing intermediate object program and deposits new intermediate object program in.
With respect to prior art, the invention has the beneficial effects as follows: because the present invention determines a minimum time section that statistical significance is arranged, utilize the statistics dimension data that set is further divided in the daily record of a minimum time section simultaneously, every just each daily record that forms in the last minimum time section set through a minimum time section added up, calculate the intermediate object program in this minimum time section, the i.e. parameter of the distribution of the statistics of statistics of Fen Buing and algebraically, and this intermediate object program is saved in the middle table, when definite statistical condition is added up, just can utilize these intermediate object program intermediate list directly to carry out statistical computation, thereby improve efficient and statistics speed significantly.
In addition, owing in every daily record, all can comprise the statistics dimension data relevant with this daily record, therefore when the log sheet of database is arrived in the preservation daily record, by the statistics dimension data that comprises in the dynamic collection daily record and it is saved in corresponding dictionary table just can obtains the statistics dimension data relevant with the daily record of all preservations, be convenient to maintenance to the dynamic statistics dimension data, reduce the complexity of log statistic system, improve its statistical efficiency.
Description of drawings
Fig. 1 is the theory diagram of log statistic of the present invention system;
Fig. 2 is the flow chart of log statistic method of the present invention;
Fig. 3 is a flow chart of preserving daily record in the log statistic method of the present invention;
Fig. 4 is a flow chart of preserving the statistics dimension data in the log statistic method of the present invention;
Fig. 5 is the flow chart that generates intermediate object program in the log statistic method of the present invention;
Fig. 6 is the flow chart that generates statistics in the log statistic method of the present invention.
Embodiment
See also Fig. 1, log statistic of the present invention system 100 can be to can certainly being that the content of the massive logs collected of a kind of equipment is added up from plurality of devices, utilize the characteristics that daily record is added up to accelerate massive logs statistics speed simultaneously, comprise log collection unit 110, daily record resolution unit 120, daily record preservation unit 130, log statistic unit 140.
Equipment of the present invention includes but not limited to that fire compartment wall, system for monitoring intrusion, router and server etc. generate the equipment of daily record.
The daily record that the 110 pairs of equipment 200 in described log collection unit send over is collected, and sends daily record resolution unit 120 to.
Real time parsing is carried out in the daily record of 120 pairs of equipment 200 of described daily record resolution unit.
Described daily record is preserved unit 130 daily record is saved in the log sheet 310 of database 300, so that handle the back.Log statistic of the present invention system 100 all sets up a log sheet 310 for every type daily record, and the alarm log of for example all invasion attacks all is kept in the invasion attack logs table.The daily record correspondence that each bar slave unit 200 is received a record in the log sheet 300, when daily record is saved in log sheet 310 in the database 300, may need to collect relevant statistics dimension data and be saved in the dictionary table 320, described dictionary table 320 is the attached tables in the database 300, be used for illustrating the attribute in a certain respect of daily record, for example for the alarm log of invading attack, just there are a plurality of dictionary tables 320, wherein the agreement dictionary table has been listed all protocol types, in log sheet 310, adopt a field to preserve any bar record that each bar daily record correspondence the agreement dictionary table, thereby represent the protocol attribute of each daily record.
Described log statistic unit 140 is every to be spent one period scheduled time the intermediate object program that generates statistics is handled in the daily record of collecting, and is saved in the middle table 330.In the statistics stage, log statistic unit 140 can middle table 300 obtain intermediate object program, handles and be shown in statistics interface 150, thereby accelerates the statistics speed of daily record.
See also Fig. 2, log statistic method of the present invention comprises that log collection and analyzing step D1, daily record preserve step D2, regularly generate middle table step D3 and generate statistics step D4.Figure 2 shows that the flow process that the daily record data in one period scheduled time is handled.Usually, parallel implementation step D1, D2 of the daily record data of 100 pairs of different time sections of log statistic system and D4.
Daily record can obtain from a plurality of or an equipment 200 by log collection unit 110, mainly is by Syslog or snmp protocol etc. daily record to be sent to log statistic system 100 from a plurality of equipment 200.
120 pairs of daily records of daily record resolution unit are then resolved, and the method for parsing can adopt existing daily record analytic technique, as the technology based on plug-in unit.Usually, daily record that equipment 200 sends or incident are character strings that certain format is arranged, log statistic system 100 receives after these character strings, daily record resolution unit 120 is according to these character strings of interpretation of rules of form, the original log that the equipment of receiving 200 is sent is output as the daily record that can be discerned by log statistic system 100, therefrom analyzes desired content.For example for daily record:
"＜1〉LX-NIDS[11174]: | web-attacks_6|WEB attacks: id command attempt[Classification:Web Application Attack] [Priority:0]: TCP}202.210.0.1:1090-＞10.50.10.8:80 " can parse following information:
Invasion is attacked name: WEB and is attacked: id command attempt
Host IP address: 202.210.0.1 is attacked in invasion
Invasion attacked port: 1090
Invaded attack host IP address: 10.50.10.8
Invaded attacked port: 80
Event level: 0
Protocol type: TCP.
Seeing also Fig. 3, is the flow chart that step is preserved in daily record.
After daily record preservation unit 130 received the daily record of an equipment of having resolved, implementation step S1 was according to the definite log sheet 310 that will preserve of the content of daily record.Log sheet 310 can exist a plurality of according to the difference of the kind of handling daily record, for example invades attack logs table, virus event log sheet and user access logs table or the like.
Simultaneously, implementation step S2 before concrete log content is saved in log sheet 310, collects the dynamic statistics dimension data.See also Fig. 4, when also having untreated dynamic statistics dimension data in the daily record after resolving, implementation step S21, obtain following first untreated dynamic statistics dimension data, and with its be kept at corresponding dictionary table 320 in the dynamic statistics dimension data collected compare, if the dynamic statistics dimension data that does not exist this daily record to comprise in the dictionary table 320, then implementation step S22 is saved in this dynamic statistics dimension data in the dictionary table 320.
Implementation step S3 subsequently, the daily record for after resolving deposits log sheet different in the database 300 310 respectively in.And the continuation processing finishes up to all daily records preservations.
The present invention will be described for clearer, be that example is described with the statistics of alarm log below to the invasion attack, but this method is not limited to the statistics to the alarm log of invasion attack, can also be applied to the statistics to other daily record (as virus event alarm log, user access logs).
The alarm log of all invasion attacks all will be kept in the invasion attack logs table, and the structure of this log sheet 310 is as shown in table 1.
Table 1 invasion attack logs list structure
| Field name | Explanation |
| LIndex | Automatically increasing since 0, is the major key of log sheet, is used for uniquely in log sheet determining a daily record. |
| GTime | Is described, if should the time in the daily record then insert as time proximity with the time of receiving daily record the time that present event takes place. |
| AlertTitleID | The ID of warning theme is used for illustrating that name is attacked in the invasion of this attack.It is an external key, illustrates this is attacked corresponding which invasion of invading in the attack famous-brand clock and attacks name, and is promptly corresponding with the major key AlertTitleID that invades the attack famous-brand clock, illustrates that name is attacked in the invasion of each bar daily record. |
| SrcIP | Source IP address just initiates to invade the IP address of the main frame of attacking |
| SrcPort | Source port points out that this attack is the attack of attacking which port initiation of main frame from invasion. |
| DestIP | Purpose IP address, the IP address of the main frame of being attacked is attacked in this invasion. |
| DestPort | Destination interface is pointed out port under attack. |
| IPProtoID | Protocol type, this attack by which procotol is undertaken.It is an external key, and explanation is that agreement in the IP protocol tables, and promptly the major key IPProtoID with the IP protocol tables is corresponding, and the protocol type of each bar daily record is described. |
| EventLevelID | The order of severity of this attack.It is an external key, and explanation is which rank in the event level table, and explanation is that agreement in the event level table, and promptly the major key EventLevelID with the event level table is corresponding, and the event level of each bar daily record is described. |
| RepeatNum | The same log occurrence number that this record is corresponding, default is 1 |
| OriginalMsg | The content of original log |
Relevant dictionary table 320 has three, comprises invasion attack famous-brand clock, IP protocol tables and event level table, and its structure is respectively shown in table 2, table 3 and table 4.
The name list structure is attacked in table 2 invasion
| Field name | Explanation |
| AlertTitleID | Major key is used for correspondingly with AlertTitleID in the log sheet, illustrates that name is attacked in the invasion of each bar daily record correspondence. |
| AlertTitle | Name is attacked in invasion. |
Table 3IP protocol tables structure
| Field name | Explanation |
| IPProtoID | Major key is used for correspondingly with IPProtoID in the log sheet, and each bar daily record corresponding protocols name is described. |
| IPProtoName | The agreement name. |
Table 4 event level list structure
| Field name | Explanation |
| EventLevelID | Major key is used for correspondingly with the EventLevelID of log sheet, and the order of severity of the corresponding incident of each bar daily record is described. |
| EventLevelName | The title of this order of severity. |
Wherein, invading and attacking the famous-brand clock initial condition is sky, replenishes by constantly collect AlertTtile in the daily record of receiving, maintenance invasion attack famous-brand clock is consistent with daily record.And because the kind of agreement and the value of event level determine that so the content of IP protocol tables and event level table do not need to collect, it is constant that the content of these two kinds of dictionary tables 320 remains,
Suppose that during two o'clock in the morning to three on January 1st, 2003 log statistic system received the alarm log of as shown in table 5 ten invasion attacks, the alarm log that any invasion is attacked is not received by the log statistic system in other running time.
The daily record that table 5 is received
| Receive the time of daily record | The log content of receiving |
| 2003-01-01?02:10:11 | <70>LX-NIDS[1876]：spp_bo：Back?Orifice?Traffic?detected(key：31337) {UDP}202.210.0.5:1125-＞10.50.10.8:31337 |
| 2003-01-01?02:11:51 | <69>LX-NIDS[1876]：spp_bo：Back?Orifice?Traffic?detected(key：31337) {UDP}202.210.0.5:1125-＞10.50.10.8:31337 |
| 2003-01-01?02:12:11 | <68>LX-NIDS[1876]：spp_bo：Back?Orifice?Traffic?detected(key：31337) {UDP}202.210.0.5:1125-＞10.50.10.8:31337 |
| 2003-01-01?02:12:32 | <67>LX-NIDS[1876]：spp_bo：Back?Orifice?Traffic?detected(key：31337) {UDP}202.210.0.2:1125-＞10.50.10.8:31337 |
| 2003-01-01?02:15:11 | <66>LX-NIDS[1876]：spp_bo：Back?Orifice?Traffic?detected(key：31337) {UDP}202.210.0.1:1125-＞10.50.10.8:31337 |
| 2003-01-01?02:20:17 | ＜1〉LX-NIDS[11174]: | web-attacks_6|WEB attacks: id command attempt [Classification:Web Application Attack] [Priority:0]: TCP}202.210.0.1:1090-＞10.50.10.8:80 |
| 2003-01-01?02:30:19 | ＜2〉LX-NIDS[3350]: | web-cgi_45|WEB attacks: files.pl access[Classification:Attempted Information Leak] [Priority:0]: { TCP}202.210.0.1:1090-＞10.50.10.8:80 |
| 2003-01-01?02:35:10 | ＜3〉LX-NIDS[3350]: | web-coldfusion_31|WEB attacks: onrequestend.cfm access [Classification:Attempted Information Leak] [Priority:0]: { TCP} 202.210.0.1:1090-＞10.50.10.8:80 |
| 2003-01-01?02:40:19 | ＜1〉LX-NIDS[11174]: | web-attacks_6|WEB attacks: id command attempt [Classification:Web Application Attack] [Priority:0]: TCP}202.210.0.1:1090-＞10.50.10.8:80 |
| 2003-01-01?02:50:22 | ＜1〉LX-NIDS[11174]: | web-attacks_6|WEB attacks: id command attempt [Classification:Web Application Attack] [Priority:0]: TCP}202.210.0.1:1090-＞10.50.10.8:80 |
After 2003-01-01 02:10:11 receives that alarm log is attacked in article one invasion, name Back Orifice is attacked in the invasion of its correspondence to be saved in the invasion attack famous-brand clock, and five day of the second to the of back invasion attack alarm log reports that it all is Back Orifice that name is attacked in invasion, therefore in log statistic system 100, do not increase new invasion and attack name, attack name WEB attack and in the 6th invasion attack alarm log, collect new invasion: id command attempt, adding it to invasion attacks in the famous-brand clock, the processing of back is similar, and this does not give unnecessary details.It is as shown in table 6 that the famous-brand clock content is attacked in the invasion that forms at last.
The content that famous-brand clock is attacked in table 6 invasion
| AlertTitleID | AlertTitle |
| 0 | Back?Orifice |
| 1 | WEB attacks: id command attempt |
| 2 | WEB attacks: files.pl access |
| 3 | WEB attacks: onrequestend.cfm access |
Wherein, AlertTitleID is the sign type in invasion attack famous-brand clock, and just system generates sequence number value automatically, the delegation in this sequence number value unique identification table.External key AlertTitleID in major key AlertTitleID and the log sheet has corresponding relation.
Because the kind of agreement and the value of event level determine, so the content of IP protocol tables and event level table do not need to collect, and the content of these two kinds of dictionary tables 320 remains constant, respectively shown in table 7 and table 8.
The content of table 7IP protocol tables
| IPProtoID (hexadecimal representation) | IPProtoName | Explanation |
| 0x00?00?00?01 0x00?00?00?02 0x00?00?00?04 0x00?00?00?08 0x00?00?00?10 0x00?00?00?20 0x00?00?00?40 0x00?00?00?80 | ICMP TCP UDP SIPP-ESP SIPP-AH IGRP OSPFIGP other | Other agreements of ICMP agreement Transmission Control Protocol udp protocol SIPP-ESP agreement SIPP-AH agreement IGRP agreement OSPFIGP agreement |
The content of table 8 event level table
| EventLevelID | EventLevelName |
| 0x01 | Do not have |
| 0x02 | Low |
| 0x04 | In |
| 0x08 | High |
Preserve after step finishes in daily record, aforementioned ten daily records are saved in the invasion attack logs table in the database, and the content of preservation is as shown in table 9.
Seeing also Fig. 5, is the flow chart that regularly generates the intermediate object program step in the log statistic method of the present invention.Regularly generating the intermediate object program stage, though the time is continuous, but because selected normally significant time period when carrying out log statistic, therefore determine a minimum time section that statistical significance is arranged as time statistics dimension, daily record is once calculated the daily record in the last minimum time section every a minimum time section, just daily record is divided set according to time and other statistics dimension data condition, each set is calculated the parameter of each distribution of the statistics of the statistics of each distribution and algebraically respectively, and the result is saved in middle table, when carrying out log statistic, just can utilize these intermediate object programs directly to carry out statistical computation, thereby greatly raise the efficiency.
At first judge between time started and concluding time intermediate object program whether as calculated? because allow manual this flow process of carrying out, therefore can recomputate, and replace original result of calculation with new result of calculation to the good as calculated time period.If therefore the intermediate object program between time started and concluding time as calculated a part then change step S4, otherwise change step S5.When regularly carrying out, time started and concluding time are respectively the time that finished a last time of carrying out and a last time period.
Step S4 is deletion existing intermediate object program between time started and concluding time, and purpose is in order to deposit new intermediate object program in.
What exist between permission time started and concluding time in this flow process is not a minimum time section, and comprises the situation of a plurality of minimum time sections.Step S5 is first minimum time section after the acquisition time started.
Implementation step S6 subsequently, the statistics that set that each statistics dimension in this minimum time section is formed utilizes log sheet 310 to calculate to distribute and the parameter of algebraically statistics, i.e. intermediate object program, and it is saved in the corresponding middle table 330.
Judge whether then to have reached the concluding time, if arrive, process ends then; If no, then enter step S7, obtain to calculate the next minimum time section of the time period of finishing, and jump to step S6 and calculate.
Alarm log with aforementioned invasion attack is an example, the result of as if statistics is included in the total degree that the invasion attack takes place in a certain fixed time and invades the percentage that the invasion attack of attacking name accounts for all invasion attacks with each, and then the structure of middle table can be as shown in table 10.
The structure of table 10 middle table
| Field name | Explanation |
| HourID | Since the hourage of fiducial time as 01 month 00:00:00 on the 01st in 2003 |
| AlertTitleID | The ID of warning theme is used for illustrating that name is attacked in the invasion of this attack.It is an external key, illustrates this is attacked corresponding which invasion of invading in the attack famous-brand clock and attacks name. |
| CountNum | Tolerance is just found the number of times that invasion is attacked under these conditions. |
The total degree of wherein invading the attack generation is the statistics that distributes, can be by each minimum time section and each invasion attack name be divided set to data, timing is added up the statistics of these set and they is saved in the middle table as intermediate object program again, just can calculate final statistics by these middle table after the time period of having determined final statistics.
The percentage that the invasion attack that name is attacked in each invasion accounts for all invasion attacks then is the statistics of an algebraically, number of times of attacking the invasion attack of name because of the invasion total degree of attack and each invasion all is the result who distributes, and this percentage can obtain divided by the total degree of invasion attack by number of times that the invasion attack of name is attacked in each invasion.
If the minimum time section of definition is one hour, be that 2003-01-0103:00:00 will be to handling in the daily record that 2003-01-01 02:00:00～2003-01-01 02:59:59 receives so after one hour, we are fiducial time with 2003-01-01 00:00:00, so above-mentioned 2003-01-0102:00:00～2003-01-01 02:59:59 should be corresponding be 01 minimum time section, 2003-01-0100:00:00～2003-01-01 01:59:59 correspondence be 00 minimum time section.
First minimum time section of obtaining among the step S5 is 01 minimum time section.
Among the step S6, can insert the record of each hour by SQL statement in middle table, the content of the middle table of generation is as shown in table 11.
The content of table 11 middle table
| HourID | AlertTitleID | CountNum |
| 1 | 0 | 5 |
| 1 | 1 | 3 |
| 1 | 2 | 1 |
| 1 | 3 | 1 |
Seeing also Fig. 6, is the flow chart that generates the statistics step in the log statistic method of the present invention.
Step S8 determines statistical condition; Among the step S9, utilize the intermediate object program generated, calculate the parameter of each distribution of the statistics of the statistics of finishing distribution and algebraically; Among the later step S10, utilize the statistics of the calculation of parameter algebraically of these distributions; By step S11 statistical result showed is come out at last.
For example calculate the total degree of the invasion attack of 2003-01-01 00:00:00～2003-01-01 04:59:59, can obtain by intermediate object program addition to the relevant invasion attack number of times of generation in each minimum time section 00,01,02,03,04.Specifically can adopt SQL statement that middle table is handled realization, can certainly adopt additive method to handle middle table and realize.
Equally, the number of times of the invasion attack of each invasion attack name also can obtain by SQL statement.
And percentages that each invasion attack of invading the attack name accounts for all invasion attacks pass through the total degree acquisition of number of times of each invasion attack of invading the attack name divided by the invasion attack.
The above only is a preferred implementation of the present invention; should be pointed out that for those skilled in the art, under the prerequisite that does not break away from the principle of the invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.
Claims (10)
1, a kind of log statistic method comprises:
Log collection and analyzing step, the slave unit collector journal, the original log that the equipment of receiving is sent resolves to the daily record that can be discerned by the log statistic system;
Step is preserved in daily record, and the statistics dimension data of collector journal also is saved in the dictionary table, and daily record is added in the log sheet as record;
Regularly generate the intermediate object program step, daily record divided set according to time and statistics dimension data, described set is calculated the statistics that distributes and algebraically statistics distribution parameter and the result is saved in middle table;
Generate the statistics step, determine that statistical condition and intermediate list calculate statistics.
2, log statistic method according to claim 1 is characterized in that: described daily record is preserved step and is comprised the corresponding of the external key of setting up log sheet and dictionary table major key.
3, log statistic method according to claim 1 is characterized in that: described regular generation intermediate object program step comprises determines minimum time section and time started and concluding time.
4, log statistic method according to claim 3 is characterized in that: described regular generation intermediate object program step comprises deletion existing intermediate object program between time started and concluding time, is used to deposit in new intermediate object program.
5, according to each described log statistic method of claim 1 to 4, it is characterized in that: described intermediate object program comprises the parameter of each distribution of the statistics of the statistics of distribution and algebraically.
6, log statistic method according to claim 5 is characterized in that: adopt SQL statement to handle middle table in the described generation statistics step.
7, a kind of log statistic system, comprise that original log that the log collection unit that is used for the slave unit collector journal, the equipment that is used for receiving send resolves to the daily record resolution unit of the daily record that can be discerned by the log statistic system, is used for daily record is preserved unit and log statistic unit as the daily record that record adds log sheet to, is characterized in that:
Described daily record is preserved the unit and also is used for the statistics dimension data of collector journal and is saved in dictionary table;
Described log statistic unit is used for daily record is divided set according to time and statistics dimension data, described set is calculated the statistics that distributes and algebraically statistics distribution parameter and the result is saved in middle table, determine that statistical condition and intermediate list calculate statistics.
8, log statistic according to claim 7 system is characterized in that: the corresponding of external key that the unit also is used to set up log sheet and dictionary table major key preserved in described daily record.
9, according to claim 7 or 8 described log statistic systems, it is characterized in that: described statistic unit also is used for determining minimum time section and time started and concluding time.
10, log statistic method according to claim 9 is characterized in that: described statistic unit also is used to delete between time started and concluding time existing intermediate object program and deposits new intermediate object program in.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2004100002060A CN100518076C (en) | 2004-01-02 | 2004-01-02 | Journal accounting method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2004100002060A CN100518076C (en) | 2004-01-02 | 2004-01-02 | Journal accounting method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN1642097A CN1642097A (en) | 2005-07-20 |
| CN100518076C true CN100518076C (en) | 2009-07-22 |
Family
ID=34866676
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2004100002060A Expired - Lifetime CN100518076C (en) | 2004-01-02 | 2004-01-02 | Journal accounting method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100518076C (en) |
Families Citing this family (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100394727C (en) * | 2005-12-26 | 2008-06-11 | 阿里巴巴公司 | Log analyzing method and system |
| CN101192227B (en) * | 2006-11-30 | 2011-05-25 | 阿里巴巴集团控股有限公司 | Log file analytical method and system based on distributed type computing network |
| US20090049547A1 (en) * | 2007-08-13 | 2009-02-19 | Yuan Fan | System for real-time intrusion detection of SQL injection web attacks |
| CN101237326B (en) * | 2008-02-29 | 2011-09-14 | 成都市华为赛门铁克科技有限公司 | Method, device and system for real time parsing of device log |
| CN101267338B (en) * | 2008-04-23 | 2010-10-13 | 杭州思福迪信息技术有限公司 | High-performance log and behavior auditing system |
| CN101286891B (en) * | 2008-05-30 | 2010-11-10 | 杭州华三通信技术有限公司 | Method and device for parsing system log |
| CN101325520B (en) * | 2008-06-17 | 2010-08-18 | 南京邮电大学 | Method for locating and analyzing fault of intelligent self-adapting network based on log |
| CN101483557B (en) * | 2009-03-03 | 2011-07-13 | 中兴通讯股份有限公司 | Log statistic, storing method and system used for deep packet detection apparatus |
| CN102271345A (en) * | 2010-06-01 | 2011-12-07 | 中兴通讯股份有限公司 | Statistical method and device for relevant information of network resident user |
| CN101951623B (en) * | 2010-09-13 | 2014-11-05 | 中兴通讯股份有限公司 | User behavior statistical method and device based on user events |
| CN102999506B (en) * | 2011-09-13 | 2016-03-30 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus obtaining user's independent access number |
| CN103036697B (en) * | 2011-10-08 | 2015-07-15 | 阿里巴巴集团控股有限公司 | Multi-dimensional data duplicate removal method and system |
| CN103209087B (en) * | 2012-01-17 | 2015-12-16 | 深圳市腾讯计算机系统有限公司 | Distributed information log statistical processing methods and system |
| CN102970363A (en) * | 2012-11-21 | 2013-03-13 | 用友软件股份有限公司 | Long-distance journal downloading system and long-distance journal downloading method |
| CN103647666A (en) * | 2013-12-13 | 2014-03-19 | 北京中创信测科技股份有限公司 | Method and apparatus for counting call detail record (CDR) messages and outputting results in real time |
| CN104317939A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Log statistics method and system on basis of digital film playing server |
| CN106503024A (en) * | 2015-09-08 | 2017-03-15 | 北京国双科技有限公司 | Log information processing method and device |
| CN105389352A (en) * | 2015-10-30 | 2016-03-09 | 北京奇艺世纪科技有限公司 | Log processing method and apparatus |
| CN106301896A (en) * | 2016-08-03 | 2017-01-04 | 合网络技术（北京）有限公司 | Log statistic method and device |
| CN106484780A (en) * | 2016-09-06 | 2017-03-08 | 努比亚技术有限公司 | Data statistical approach and device |
| CN106603749B (en) * | 2017-01-06 | 2017-11-21 | 浙江中都信息技术有限公司 | A kind of high efficiency method of dynamic IP to Host map |
| CN107562892A (en) * | 2017-09-06 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of method and device of raising SSR violation log statistic performances |
| CN109542902A (en) * | 2018-11-12 | 2019-03-29 | 珠海格力电器股份有限公司 | Data processing system and method |
| CN109450689B (en) * | 2018-11-19 | 2022-02-22 | 郑州云海信息技术有限公司 | Log printing method and device, storage medium and computer equipment |
| CN112738087A (en) * | 2020-12-29 | 2021-04-30 | 杭州迪普科技股份有限公司 | Attack log display method and device |
| CN112732759B (en) * | 2020-12-31 | 2023-02-03 | 青岛海尔科技有限公司 | Data processing method and device, storage medium and electronic device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2003140992A (en) * | 2001-11-06 | 2003-05-16 | Nippon Telegr & Teleph Corp <Ntt> | Stream distribution device, stream distribution method, stream distribution processing program, and recording medium having the processing program recorded therein |
-
2004
- 2004-01-02 CN CNB2004100002060A patent/CN100518076C/en not_active Expired - Lifetime
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2003140992A (en) * | 2001-11-06 | 2003-05-16 | Nippon Telegr & Teleph Corp <Ntt> | Stream distribution device, stream distribution method, stream distribution processing program, and recording medium having the processing program recorded therein |
Also Published As
| Publication number | Publication date |
|---|---|
| CN1642097A (en) | 2005-07-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100518076C (en) | Journal accounting method and system | |
| CN107454109B (en) | Network privacy stealing behavior detection method based on HTTP traffic analysis | |
| CN111277570A (en) | Data security monitoring method and device, electronic equipment and readable medium | |
| CN101325520B (en) | Method for locating and analyzing fault of intelligent self-adapting network based on log | |
| CN107370752B (en) | Efficient remote control Trojan detection method | |
| CN212259006U (en) | Network security management equipment | |
| CN102918534A (en) | Query pipeline | |
| DE112012002624T5 (en) | Regex compiler | |
| CN101741608B (en) | Traffic characteristic-based P2P application identification system and method | |
| CN109150869A (en) | A kind of exchanger information acquisition analysis system and method | |
| CN106375345A (en) | Malware domain name detection method and system based on periodic detection | |
| CN102611713A (en) | Entropy operation-based network intrusion detection method and device | |
| CN110958231A (en) | Industrial control safety event monitoring platform and method based on Internet | |
| CN110120957B (en) | Safe disposal digital twin method and system based on intelligent scoring mechanism | |
| CN115883236A (en) | Power grid intelligent terminal cooperative attack monitoring system | |
| CN112491860A (en) | Industrial control network-oriented collaborative intrusion detection method | |
| Labib et al. | Detecting and visualizing denialof-service and network probe attacks using principal component analysis | |
| Rosay et al. | From CIC-IDS2017 to LYCOS-IDS2017: A corrected dataset for better performance | |
| CN114598499A (en) | Network risk behavior analysis method combined with business application | |
| CN117614712A (en) | Security audit method and system based on user portrait and association analysis | |
| CN100383784C (en) | On-line analysing and treating system and method | |
| CN108566382A (en) | The fire wall adaptive ability method for improving of rule-based life cycle detection | |
| Campbell et al. | Intrusion detection at 100G | |
| CN112528325B (en) | Data information security processing method and system | |
| Li et al. | Effective DDoS attacks detection using generalized entropy metric |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CX01 | Expiry of patent term |
Granted publication date: 20090722 |
|
| CX01 | Expiry of patent term |