CN105162622B - A kind of storage method and system - Google Patents

A kind of storage method and system Download PDF

Info

Publication number
CN105162622B
CN105162622B CN201510481866.3A CN201510481866A CN105162622B CN 105162622 B CN105162622 B CN 105162622B CN 201510481866 A CN201510481866 A CN 201510481866A CN 105162622 B CN105162622 B CN 105162622B
Authority
CN
China
Prior art keywords
period
flow information
data
time
tables
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201510481866.3A
Other languages
Chinese (zh)
Other versions
CN105162622A (en
Inventor
田海燕
练书成
丁毅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Raisecom Technology Co Ltd
Original Assignee
Raisecom Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Raisecom Technology Co Ltd filed Critical Raisecom Technology Co Ltd
Priority to CN201510481866.3A priority Critical patent/CN105162622B/en
Publication of CN105162622A publication Critical patent/CN105162622A/en
Application granted granted Critical
Publication of CN105162622B publication Critical patent/CN105162622B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14—Network analysis or design
    • H04L41/142—Network analysis or design using statistical or mathematical methods

Landscapes

  • Physics & Mathematics (AREA)
  • Algebra (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Pure & Applied Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention discloses a kind of storage method and systems, comprising: the flow information that periodically acquisition gateway equipment generates from kernel.Establish multiple and different tables of data for storing the flow information in preset different time sections.The flow information of acquisition is stored in corresponding tables of data according to the preset period.Scheme through the invention is able to solve present in current techniques and only uses a big table to store integrated flow information, thus the problem of needing a large amount of memory capacity.

Description

A kind of storage method and system
Technical field
The present invention relates to network flow statistic technology more particularly to a kind of storage methods and system.
Background technique
Website traffic statistic is a kind of source that can accurately analyze Guest User, convenient for website webmaster according to visitor Increase in demand or modify website related content, convenient for preferably promoted website conversion ratio, improve website traffic.Website stream Amount statistics can accurately count the specific area of source and the address ip of visitor, and current web how many people online specifically have accessed Which page, visitor are by which page, the website of search key visitor, which page is visitor's browsing be, visitor's Browser is what version, ie6, ie7 or red fox, and the timesharing accounting of website divides day to count, point moon statistics, real-time statistics, Which page line accesses, what the operating system of visitor is, resolution ratio is how many.In these many events, relate to These events how are stored, user is then easily and efficiently shown to, allow users to analyze and use well.
It is only directly taken out from kernel for the integrated flow information that in the statistical module of early period, the page is shown, in this way After equipment is restarted, it is 0 that accumulative flow information, which will disappear,.Support 10 minutes simultaneously, 1 hour, 24 hours querying conditions, The bad function that second level can not be supported to inquire of scalability in this way.And when user needs to show integrated flow information, only from It is directly taken out in kernel or only uses a big table to store integrated flow information, the effect of integrated flow cannot be reached in this way Fruit either needs a large amount of memory capacity come the problem of storing integrated flow information.
Summary of the invention
To solve the above-mentioned problems, it the invention proposes a kind of storage method and system, is able to solve in current techniques and deposits Only use a big table and store integrated flow information, thus the problem of needing a large amount of memory capacity.
In order to achieve the above object, the invention proposes a kind of storage methods, this method comprises:
The timing flow information that acquisition gateway equipment generates from kernel.
Establish multiple and different tables of data for storing the flow information in preset different time sections.
The flow information of acquisition is stored in corresponding tables of data according to the preset period.
Preferably,
This method further include: classify to the flow information of acquisition；Sorted flow information is deposited according to the classification Enter database, and establishes database index corresponding with classification.
Carrying out classification to the flow information of acquisition includes: according to source network Protocol IP address and/or agreement to flow information Classify.
Wherein, which includes: network web protocol, peer-to-peer network p2p agreement and instant communication protocol.
Preferably, multiple and different tables of data packets for storing the flow information in preset different time sections are established It includes:
Period tables of data corresponding from the different periods is established, and/or establishes under section in different times and divides The corresponding period classification data table of class.
Preferably, the flow information of acquisition is stored in corresponding tables of data according to the preset period and includes:
At interval of preset period first time, will believe from the flow in preset period first time acquired in kernel In breath deposit first time period tables of data；And by the non-first time on the basis of current time in first time period tables of data The flow information in section is deleted.
At interval of preset second time period, the preset second time period that will be stored in first time period tables of data In interior flow information deposit second time period tables of data；And by second time period tables of data on the basis of current time Flow information in non-second time period is deleted.
At interval of the preset third time cycle, the preset third time cycle that will be stored in second time period tables of data In interior flow information deposit third period tables of data；And by third period tables of data on the basis of current time Flow information in the non-third period is deleted.
Wherein, period first time and second time period are less than first time period；When the third time cycle is less than second Between section；First time period is less than second time period；Second time period is less than the third period.
Preferably, the flow information of acquisition is stored in corresponding tables of data according to the preset period further include:
At interval of the preset xth time cycle, by from the preset xth time cycle acquired in kernel with source IP In the relevant flow information deposit a period m classification data table of location, and by a period m classification data table with Flow information relevant to source IP address in the non-a period on the basis of current time is deleted；And/or it will be adopted from kernel Being stored in the n-th classification data of a period table with protocol-dependent flow information in the preset xth time cycle of collection, and By in the non-a period on the basis of current time in a period the n-th classification data table with protocol-dependent flow Information deletion.
At interval of the preset y time cycle, by the preset y time cycle stored in database with source IP In the relevant flow information deposit b period m classification data table of location, and by b period m classification data table with Flow information relevant to source IP address in the non-b period on the basis of current time is deleted；And/or it will be deposited in database Being stored in the n-th classification data of b period table with protocol-dependent flow information in the preset y time cycle of storage, and By in the non-b period on the basis of current time in b period the n-th classification data table with protocol-dependent flow Information deletion.
At interval of the preset z time cycle, by the preset z time cycle stored in database with source IP In the relevant flow information deposit c period m classification data table of location, and by c period m classification data table with Flow information relevant to source IP address in the non-c period on the basis of current time is deleted；And/or it will be deposited in database Being stored in the n-th classification data of c period table with protocol-dependent flow information in the preset z time cycle of storage, and By in the non-c period on the basis of current time in c period the n-th classification data table with protocol-dependent flow Information deletion.
Wherein, xth time cycle and y time cycle are less than a period；The z time cycle is less than the b period； The a period is less than the b period；The b period is less than the c period.
Preferably, this method further include:
Divide when only establishing the m for storing the flow information relevant to source IP address in preset different time sections When class tables of data, convert the flow information relevant to source IP address stored in m classification data table to and source IP address phase Corresponding and protocol-dependent flow information.Alternatively,
When only establishing for storing n-th point with the protocol-dependent flow information in preset different time sections When class tables of data, it is converted into what is stored in the n-th classification data table and the association with the protocol-dependent flow information Discuss corresponding flow information relevant to the source IP address.
Preferably, this method further include: at interval of preset 4th time cycle, by the flow in the database in memory Information is compressed, and by the storage of compressed flow information into external memory；It, will be in external memory after equipment exception occurs and restarts The flow information of storage is put into memory, and is shown after flow information is decompressed and shown on the page in equipment.
In order to achieve the above object, the invention also provides a kind of storage system, which includes: acquisition module, data Table establishes module and the first memory module.
Acquisition module, for the timing flow information that acquisition gateway equipment generates from kernel.
Tables of data establishes module, for establishing for storing the multiple and different of the flow information in preset different time sections Tables of data.
First memory module, the flow information for that will acquire are stored in corresponding tables of data according to the preset period.
Preferably,
The system further include: categorization module, the second memory module and index establish module.
Categorization module, for classifying to the flow information of acquisition.
Second memory module, for sorted flow information to be stored in database according to classification.
Index establishes module, for establishing database index corresponding with classifying.
Categorization module carries out classification to the flow information of acquisition and refers to: according to source network Protocol IP address and/or agreement pair Flow information is classified.
Wherein, which includes: network web protocol, peer-to-peer network p2p agreement and instant communication protocol.
Preferably, tables of data establish module establish for store the flow information in preset different time sections it is multiple not Same tables of data refers to:
Period tables of data corresponding from the different periods is established, and/or establishes under section in different times and divides The corresponding period classification data table of class.
Preferably, the flow information of acquisition is stored in corresponding tables of data by the first memory module according to the preset period Refer to:
At interval of preset period first time, will believe from the flow in preset period first time acquired in kernel In the deposit first time period tables of data of breath；And when by non-first on the basis of current time in first time period tables of data Between flow information in section delete.
At interval of preset second time period, the preset second time period that will be stored in first time period tables of data In interior flow information deposit second time period tables of data；And by second time period tables of data on the basis of current time Flow information in non-second time period is deleted.
At interval of the preset third time cycle, the preset third time cycle that will be stored in second time period tables of data In interior flow information deposit third period tables of data；And by third period tables of data on the basis of current time Flow information in the non-third period is deleted.
Wherein, period first time and second time period are less than first time period；When the third time cycle is less than second Between section；First time period is less than second time period；Second time period is less than the third period.
Preferably, the flow information of acquisition is stored in corresponding by the first memory module according to the preset period Also refer in tables of data:
At interval of the preset xth time cycle, by from the preset xth time cycle acquired in kernel with source IP In the relevant flow information deposit a period m classification data table of location, and by a period m classification data table with Flow information relevant to source IP address in the non-a period on the basis of current time is deleted；And/or it will be adopted from kernel Being stored in the n-th classification data of a period table with protocol-dependent flow information in the preset xth time cycle of collection, and By in the non-a period on the basis of current time in a period the n-th classification data table with protocol-dependent flow Information deletion.
At interval of the preset y time cycle, by the preset y time cycle stored in database with source IP In the relevant flow information deposit b period m classification data table of location, and by b period m classification data table with Flow information relevant to source IP address in the non-b period on the basis of current time is deleted；And/or it will be deposited in database Being stored in the n-th classification data of b period table with protocol-dependent flow information in the preset y time cycle of storage, and By in the non-b period on the basis of current time in b period the n-th classification data table with protocol-dependent flow Information deletion.
At interval of the preset z time cycle, by the preset z time cycle stored in database with source IP In the relevant flow information deposit c period m classification data table of location, and by c period m classification data table with Flow information relevant to source IP address in the non-c period on the basis of current time is deleted；And/or it will be deposited in database Being stored in the n-th classification data of c period table with protocol-dependent flow information in the preset z time cycle of storage, and By in the non-c period on the basis of current time in c period the n-th classification data table with protocol-dependent flow Information deletion.
Wherein, xth time cycle and y time cycle are less than a period；The z time cycle is less than the b period； The a period is less than the b period；The b period is less than the c period.
Preferably, tables of data is established module and is also used to:
Divide when only establishing the m for storing the flow information relevant to source IP address in preset different time sections When class tables of data, convert the flow information relevant to source IP address stored in m classification data table to and source IP address phase Corresponding and protocol-dependent flow information.Alternatively,
When only establishing for storing classifying number in preset different time sections with the n-th of protocol-dependent flow information When according to table, and source IP corresponding with agreement is converted into protocol-dependent flow information by what is stored in the n-th classification data table The relevant flow information in address.
Preferably, system further include: third memory module and display module.
Third memory module, at interval of preset 4th time cycle, the flow in the database in memory to be believed Breath is compressed, and by the storage of compressed flow information into external memory；After equipment exception occurs and restarts, it will be deposited in external memory The flow information of storage is put into memory.
Display module shows the page in equipment for display after decompressing flow information.
Compared with prior art, the present invention includes: the timing flow information that acquisition gateway equipment generates from kernel.It establishes For storing multiple and different tables of data of the flow information in preset different time sections.By the flow information of acquisition according to pre- If period be stored in corresponding tables of data.Scheme through the invention is able to solve present in current techniques and only uses One big table stores integrated flow information, thus the problem of needing a large amount of memory capacity.
Detailed description of the invention
The attached drawing in the embodiment of the present invention is illustrated below, the attached drawing in embodiment be for of the invention into one Step understands, is used to explain the present invention, does not constitute a limitation on the scope of protection of the present invention together with specification.
Fig. 1 is storage method flow chart of the invention；
Fig. 2 is storage system composition block diagram of the invention；
Fig. 3 is the tables of data schematic diagram of the 10 minutes _ source ip.db of the embodiment of the present invention；
Fig. 4 is the tables of data schematic diagram of 10 minutes _ proto.db of the embodiment of the present invention；
Fig. 5 is 24 hours integrated flow information trend schematic diagrames of the embodiment of the present invention.
Specific embodiment
For the ease of the understanding of those skilled in the art, the invention will be further described with reference to the accompanying drawing, not It can be used to limit the scope of the invention.
When the present invention program mainly solves user and needs to show integrated flow information, only directly take out from kernel or It is to only use a big table to store integrated flow information, it is a large amount of cannot achievees the effect that integrated flow either needs in this way Memory capacity is come the problem of storing integrated flow information.
In order to solve to only use a big table present in above-mentioned current techniques to store integrated flow information, to need The problem of wanting a large amount of memory capacity, by the present invention in that storing the integrated flow of equipment generation with the method for multiple tables of data Information.
Specifically, in order to achieve the above object, the invention proposes a kind of storage methods, as shown in Figure 1, this method packet It includes:
S101, the flow information that periodically acquisition gateway equipment generates from kernel.
In embodiments of the present invention, a timer can be set, a secondary gateway is acquired every certain time interval and sets The standby flow information generated, this time interval can carry out customized according to specific application scenarios.
First classify to it before storing in the database to memory for collected flow information.
Preferably,
This method further include: classify to the flow information of acquisition；Sorted flow information is deposited according to the classification Enter database, and establishes database index corresponding with classification.
Carrying out classification to the flow information of acquisition includes: according to source network Protocol IP address and/or agreement to flow information Classify.
Wherein, which includes: network web protocol, peer-to-peer network p2p agreement and instant communication protocol.
Web protocol class includes http and https；P2p protocol class includes a sudden peal of thunder, the super whirlwind ... of bt, edonkey, QQ； Instant communication protocol includes that QQ, wechat, Fetion, Taobao Wang Wang ... etc. will not enumerate here.
Source IP address, for example, 10.10.25.251,10.10.25.80,10.10.25.6 ... etc..
When classifying to the flow information of acquisition, we can classify only in accordance with source IP address, can also be only Classify according to agreement, can also classify simultaneously according to source IP address and agreement.And in other examples, also It can classify according to different application scenarios for other classification foundations.
In embodiments of the present invention, classify in advance to the flow information of acquisition, so as in subsequent operation to needs Flow information quickly searched.
Be explained below how by sorted flow information according to classification be stored in database, and establish it is corresponding with classification Database index.
In embodiments of the present invention, after classifying through the above to flow information, we are by sorted stream Information storage is measured into database, and for convenience of searching, establishes the index of database according to above-mentioned classification situation in storage.
In storage, the field of each tables of data can be following form:
Time: 4 bytes, source IP: 4 bytes, agreement: 4 bytes upload data: 4 bytes, downloading data: 4 bytes.
Alternatively, by the field extension of each tables of data are as follows:
Time: 4 bytes, source IP: 4 bytes, agreement major class: 4 bytes upload data: 4 bytes, downloading data: 4 bytes.
In embodiments of the present invention, other extensions can also be done to field, be not limited to that extension.
By the field of tables of data, we can establish the index of the database about the flow information stored, specifically In embodiments of the present invention without limitation, can be established using the achievable method for building up of any one conventional should for method for building up Database index.
S102, multiple and different tables of data for storing the flow information in preset different time sections are established.
It is all to count whole integrated flows in one big table, this does in existing integrated flux memory technology Method needs a large amount of memory capacity.In embodiments of the present invention, in order to solve the problems in the existing technology, by using more The method of tables of data is opened to store the integrated flow information of equipment generation.And the secondary inquiry of user for convenience, Ke Yigen According to the query demand of user, multiple tables of data of diversified forms are established.
Preferably, multiple and different tables of data packets for storing the flow information in preset different time sections are established It includes:
Period tables of data corresponding from the different periods is established, and/or establishes under section in different times and divides The corresponding period classification data table of class.
It will be discussed in detail below by specific embodiment and how establish multiple various forms of tables of data.
Firstly, establishing period tables of data corresponding from the different periods
In practical application, user is usually required within inquiry 10 minutes, perhaps within an hour or within 24 hours Integrated flow information or the flow information in any other period, according to the different demands of user, we can build respectively Tables of data corresponding with the period is found, for example, 10 minute data tables, 1 hour data table, 24 hour data tables ..., it will be current Integrated flow information within 10 minutes moment, integrated flow information within an hour, the integrated flow information within 24 hours Deng being stored in the 10 minute data table, 1 hour data table and 24 hour data tables respectively.
Secondly, establishing period classification data table corresponding with classification under section in different times
In practical application, user also usually requires to look into for the flow information of a certain type in certain time period It askes, therefore, we can establish period classification data table corresponding with classification in section in different times respectively.For example, If having been set up 3 tables of data is respectively 10 minute data tables, 1 hour data table, 24 hour data tables.If user exists When needing to show the data traffic information only for the source address ip on the basis of above-mentioned period tables of data, then to above-mentioned 3 tables of data in collect information before, a database can be established respectively by the address each source ip of each period, such as Shown in Fig. 3, foundation is the database that the not homologous address ip is directed in 10 minutes sections, we can name the database For 10 minutes _ source ip.db, in 1 hour, 24 hour periods to the tables of data of the not homologous address ip and the foundation side of Fig. 3 Formula is identical, can be with the rest may be inferred.Similarly, if user needs to show on the basis of above-mentioned period tables of data only for association When the data traffic information of view, then before collecting information into 3 above-mentioned tables of data, it can be by each period not A database is established respectively with agreement, as shown in figure 4, what is established is the data for being directed to different agreement in 10 minutes sections Library, we can be named as the database 10 minutes _ proto.db, in 1 hour, 24 hour periods to different agreement Database and Fig. 4 to establish mode identical, can be with the rest may be inferred.It should be noted that when building table in the database of source ip Table can also be built according to ip network segment, as the tables of data in Fig. 3 can also build as 192.168.1_table, 192.168.2_ table…….Can also be built by each protocol classification in the database of agreement table be web_table (web include http and Https), instant messaging _ table (instant messaging includes qq, msn, wechat etc.) ....
Through the above scheme, if when user needs the flow information of query source IP so in database if according to each source Ip builds multiple tables of data to store.If when user needs the flow information of vlan query protocol VLAN so in database if assist according to every kind View builds multiple tables of data.When starting storage simultaneously storing data can also be all established simultaneously with source ip and agreement both tables of data. Data required for user can very easily inquire when according to source ip or agreement to inquire data in this way are believed Breath, greatly improves search efficiency.
S103, the flow information of acquisition is stored in corresponding tables of data according to the preset period.
How we in detail through the above establishes multiple tables of data if having understood, how be described in detail below by flow In multiple tables of data that information filling is established.It is different according to the form of foundation, two schemes can be divided into:
Scheme one
Preferably, the flow information of acquisition is stored in corresponding tables of data according to the preset period and includes:
At interval of preset period first time, will believe from the flow in preset period first time acquired in kernel In breath deposit first time period tables of data；And by the non-first time on the basis of current time in first time period tables of data The flow information in section is deleted.
At interval of preset second time period, the preset second time period that will be stored in first time period tables of data In interior flow information deposit second time period tables of data；And by second time period tables of data on the basis of current time Flow information in non-second time period is deleted.
At interval of the preset third time cycle, the preset third time cycle that will be stored in second time period tables of data In interior flow information deposit third period tables of data；And by third period tables of data on the basis of current time Flow information in the non-third period is deleted.
Wherein, period first time and second time period are less than first time period；When the third time cycle is less than second Between section；First time period is less than second time period；Second time period is less than the third period.
What needs to be explained here is that in the above scheme, although only listing the flow storage method of three tables of data, It is not limited to three tables of data in practical applications, user can also only establish 2, or establish 4,5, even more, tool Body quantity can according to user it is specific needs depending on, and according to user establish tables of data number how much, above scheme It can extend, e.g., when user establishes 4 tables of data, the present invention program can continue are as follows: at interval of preset week 4th time Flow information in preset 4th period stored in third period tables of data is stored in the 4th period tables of data by the phase In；And the flow information in non-4th period on the basis of current time in the 4th period tables of data is deleted.When When establishing more tables of data, scheme and so on.
In addition, period first time, second time period and third time cycle in above scheme can be according to users The different periods of setting carry out self-defining, that is, user customized can be spaced how long stream is stored into the tables of data of foundation Measure information.But it should be noted that period first time is necessarily less than first time period, because the period 1 is less than at the first time The flow information that can just collect in first time period when section is put into for example, acquiring primary information into kernel every 1 minute In 10 minute data tables, it just can guarantee that is stored in 10 minute data tables is the flow in 10 minutes in this way, if it is 15 minutes Acquisition is primary, it cannot be guaranteed that the information in the 10 minute data table in real-time storage nearly 10 minutes.Second time period is also necessary Less than first time period, because the flow information in second time period tables of data is obtained from first time period tables of data, If the duration of second time period is greater than the duration of first time period, can not be got from first time period tables of data Flow information, for example, acquiring primary information in order to establish 1 hour data table into 10 minute data tables every 5 minutes, being put into 1 In hour data table, 1 hour data table here is second time period tables of data, is within 5 minutes second time period, and 10 points Clock tables of data is first time period tables of data, and 10 minutes are first time period, and only second time period 5 minutes is less than the One 10 minutes periods, we just can collect data from first time period tables of data, if second time period is 15 Minute, we can not collect data in tables of data from ten minutes.Reason is same as above, when the third time cycle is necessarily less than second Between section, can with and so on, when there are the 4th time cycle, the 5th time cycle ..., when the 4th time cycle was less than third Between section, the 5th time cycle is less than the 4th period.
Below above scheme will be introduced by specific embodiment.In embodiments of the present invention, here still with 10 the number of minutes It is illustrated according to table, 1 hour data table, 24 hour data tables.Here 10 minutes, 1 hour and 24 hours are above-mentioned first Period, second time period and third period；10 minute data tables, 1 hour data table and 24 hour data tables, that is, above-mentioned First time period tables of data, second time period tables of data and third period tables of data.Firstly, we are into 10 minute data tables Data are inserted, period first time is set to 1 minute by we here, that is, every 1 minute by the flow in 1 minute in kernel Information is stored in 10 minute data tables；And by the flow in non-10 minutes on the basis of current time in 10 minute data tables Information deletion.In the following, we insert data into 1 hour data table, second time period is set to 5 minutes by we here, and And we are to acquire data from 10 minute data tables when acquiring data, that is, will be stored in 10 minute data tables every 5 minutes 5 minutes in flow information be stored in 1 hour data table in；And by non-1 on the basis of current time in 1 hour data table Flow information within hour is deleted.Finally, we insert data into 24 hour data tables, we are all by the second time here Phase is set to 20 minutes, also, when we acquire data is to acquire data from 1 hour data table, that is, small by 1 every 20 minutes When tables of data in flow information in store 20 minutes be stored in 24 hour data tables；And by 24 hour data tables to work as Flow information within non-24 hours on the basis of the preceding moment is deleted.
Through the above scheme, we establish multiple period tables of data, i.e. 10 minute data tables, 1 hour data table and 24 hour data tables, and corresponding flow information has been inserted in different period tables of data.If user needs longer The integrated flow information of time, then can suitably extend multiple tables of data according to the demand of user or reduce several data Table.As long as should be noted that for example, in 10 minute data tables collect information time in less than 10 minutes, 1 The acquisition time of hour data table be also less than 10 minutes can because be to 10 minutes tables in collect information；24 is small When acquisition time less than 1 hour can because be to 1 hour table in collect information.It is accumulative to be finally reached storage Flow information uses the purpose of least memory capacity.
User can inquire according to different time requirement into different tables of data.If necessary to show in nearly 10 minutes Data, then data query is come out into 10 minute data tables；If necessary to show the data in nearly 1 hour, then to 1 Data query is come out in hour data table；If necessary to show the data in nearly 24 hours, then to 24 hours tables of data It is middle to come out data query.
We are by the specific data of scheme through the invention and prior art comparatively bright the present invention program below Superiority.
In the prior art scheme, it is assumed that have recorded 24 hours flow informations by a tables of data, record within every 10 seconds Data, the field having time of tables of data: 4 bytes, source IP: 4 bytes, agreement: 4 bytes upload data: 4 bytes, downloading number According to: 4 bytes；So one record 4+4+4+4+4=20 byte.Now with 50 source IPs, average each IP has 50 applications.That 10 minutes memory capacity: 60*20*50*50=3M；One hour memory capacity: 60*20*50*50*6=18M；24 hours Memory capacity: 60*20*50*50*6*24=432M.
In the specific embodiment of the present invention program, if above-mentioned 24 hours tables of data is divided into three tables, One 10 minute data table, for storing the flow information in 10 minutes；One 1 hour data table, for storing in 1 hour Flow information；One 24 hour data table, for storing the flow information in 24 hours.In 10 minute data tables, every 10 Second a data are extracted into kernel, then the memory capacity in 10 minute data tables after ten minutes are as follows: 60*20*50*50= 3M.In 1 hour data table, the data in nearly 1 minute were taken out into 10 minute data tables every one minute and are calculated 1 minute Total flow is inserted into 1 hour data table, average each now with 50 source IPs due to a record 4+4+4+4+4=20 byte IP has 50 applications, then the flow information of 60 records can be stored after 1 hour, the memory capacity in 1 hour data table are as follows: 60*20*50*50=3M.Similarly, it in 24 hour data tables, was taken out in nearly 20 minutes every 20 minutes into 1 hour data table Data calculate 20 minutes total flows and be inserted into 24 hour data tables, due to a record 4+4+4+4+4=20 byte, show There are 50 source IPs, average each IP has 50 applications, then the flow information of 72 records can be stored after 24 hours, 24 hours Memory capacity in tables of data are as follows: 72*20*50*50=3.6M.
It is obtained according to above data, capacity of the flow information of storage 24 hours if only needing 432M if a table Size needs the amount of capacity of 3M+3M+3.6M=9.6M if dividing multilist to store.
Illustrate the superiority of the present invention program by query time time consuming analysis further below.In the embodiment of the present invention In, it is analyzed by taking sqlite database as an example.
Sqlite database performance measurement such as table one:
Table one
Sqlite
It is time-consuming to be inserted into 10000 records 0.42 second
It is time-consuming to be inserted into 100000 records 3.81 the second
It is time-consuming to be inserted into 7200000 records 249 seconds
It is time-consuming to be inserted into 57600000 records 2155.14 the second
It is time-consuming to be inserted into 172800000 records 6352.06 the second
10000 records look into 1 record time-consuming Less than 0.01 second
100000 records look into 1 record time-consuming Less than 0.01 second
7200000 records look into 1 record time-consuming Less than 0.01 second
57600000 records look into 1 record time-consuming 0.16 second
172800000 records look into 1 record time-consuming 0.09 second
In the above-described embodiments, if storing 24 hours flows using a table, there is the item number of record: 432M/ 20=21600000 item.It is immediate in performance test table be " insertion 57600000 records time-consumings 2155.14 seconds " and " it is 0.16 second time-consuming that 57600000 records look into 1 record ".If storing 24 hours flows using three tables, 10 points There is record 3M/20=150000 item in clock tables of data and 1 hour data table, has record 3.6M/20=in 24 hours tables of data 180000.Immediate in performance test table is " 100000 records of insertion are 3.81 seconds time-consuming " and " 100000 records 1 record time-consuming is looked into less than 0.01 second "
It is readily seen according to analysis above to store 24 hours flow informations using the method ratio of three tables and use one The method for opening table greatly reduces insertion data and inquires the time of data.
Fig. 5 is 24 hours integrated flow information trend figure, which is according to 72 points of data in 24 hours tables What information was drawn.It is that flow is calculated according to agreement when inquiring data.It should be noted that if being deposited using a table The flow information of storage 24 hours, then showing that Fig. 5 at least needs according to sqlite database performance measurement table analysis 0.16*72=8.64 seconds, and shown if using the accumulative storage of three tables, the time that Fig. 5 needs is 0.01*72=0.72 Second.Obviously 8.64 seconds > 0.72 second, it is seen that use than one table storage stream of method of the accumulative storage flow information of three tables The method of amount information shows that the time of Fig. 5 is many fastly.
Scheme two
Preferably, the flow information of acquisition is stored in corresponding tables of data according to the preset period further include:
At interval of the preset xth time cycle, by from the preset xth time cycle acquired in kernel with source IP In the relevant flow information deposit a period m classification data table of location, and by a period m classification data table with Flow information relevant to source IP address in the non-a period on the basis of current time is deleted；And/or it will be adopted from kernel Being stored in the n-th classification data of a period table with protocol-dependent flow information in the preset xth time cycle of collection, and By in the non-a period on the basis of current time in a period the n-th classification data table with protocol-dependent flow Information deletion.
At interval of the preset y time cycle, by the preset y time cycle stored in database with source IP In the relevant flow information deposit b period m classification data table of location, and by b period m classification data table with Flow information relevant to source IP address in the non-b period on the basis of current time is deleted；And/or it will be deposited in database Being stored in the n-th classification data of b period table with protocol-dependent flow information in the preset y time cycle of storage, and By in the non-b period on the basis of current time in b period the n-th classification data table with protocol-dependent flow Information deletion.
At interval of the preset z time cycle, by the preset z time cycle stored in database with source IP In the relevant flow information deposit c period m classification data table of location, and by c period m classification data table with Flow information relevant to source IP address in the non-c period on the basis of current time is deleted；And/or it will be deposited in database Being stored in the n-th classification data of c period table with protocol-dependent flow information in the preset z time cycle of storage, and By in the non-c period on the basis of current time in c period the n-th classification data table with protocol-dependent flow Information deletion.
Wherein, xth time cycle and y time cycle are less than a period；The z time cycle is less than the b period； The a period is less than the b period；The b period is less than the c period.
What needs to be explained here is that it is identical with scheme one, in the above scheme, although only listing three tables of data Flow storage method, but it is not limited to three tables of data in practical applications, user can also only establish 2, or establish 4,5 It is a, it is even more, depending on particular number can be according to the specific needs of user, and the number for the tables of data established according to user How much, above scheme can extend, and e.g., when user establishes 4 tables of data, the present invention program can continue are as follows: at interval of pre- If the h time cycle, by the flow information relevant to source IP address in the preset h time cycle stored in database Be stored in d period m classification data table, and by d period m classification data table on the basis of current time Flow information relevant to source IP address in the non-d period is deleted；And/or will be stored in database preset h when Between in the period with protocol-dependent flow information deposit the n-th classification data of d period table, and the d period n-th is divided Being deleted with protocol-dependent flow information in the non-d period on the basis of current time in class tables of data.Work as foundation When more tables of data, scheme and so on.
In addition, xth time cycle, y time cycle and z time cycle in above scheme and the h weeks subsequent Phase etc. can be according to the different periods of user setting come self-defining, that is, user can customized interval how long to foundation Tables of data in store flow information.But it should be noted that the xth time cycle is necessarily less than a period, because xth is all The flow information in a period can be just collected when phase is less than a period, for example, acquiring one into kernel every 1 minute Secondary information, is put into 10 minute data tables, just can guarantee that is stored in 10 minute data tables is the flow in 10 minutes in this way, such as Fruit is that acquisition in 15 minutes is primary, it cannot be guaranteed that the information in the 10 minute data table in real-time storage nearly 10 minutes.Week y time Phase is also necessarily less than a period, because the flow information in b period tables of data is obtained from a period tables of data It takes, if the duration of y time cycle is greater than the duration of a period, can not be obtained from a period tables of data To flow information, for example, acquiring primary information in order to establish 1 hour data table into 10 minute data tables every 5 minutes, putting Into 1 hour data table, 1 hour data table here is b period tables of data, is within 5 minutes the y time cycle, and 10 Minute data table is a period tables of data, and 10 minutes are a period, and only the y time cycle 5 minutes less than a Period 10 minutes, we just can collect data from a period tables of data, if the y time cycle is 15 minutes, We can not collect data in tables of data from ten minutes.Reason is same as above, and the z time cycle is necessarily less than the b period, can With and so on, when there are h time cycle, jth time cycle ..., the h time cycle is less than c period, jth time Period is less than the d period.
Also, source IP address and two kinds of agreement classification have only been carried out in embodiments of the present invention, in other embodiments, Classification can also be extended for different situations, however it is not limited to source IP address and two kinds of agreement.
Below above scheme will be introduced by specific embodiment.In embodiments of the present invention, here still with 10 the number of minutes It is illustrated according to table, 1 hour data table, 24 hour data tables, and classifies and be illustrated with source IP address and agreement.Here The a period m of tables of data inside 10 minutes _ source ip.db, 1 hour _ source ip.db and 24 hours _ source ip.db, that is, above-mentioned Classification data table, b period m classification data table and c period m classification data table.10 minutes _ proto.db, it is 1 small When _ proto.db and 24 hour _ proto.db inside tables of data, that is, above-mentioned a period the n-th classification data table, b when Between section the n-th classification data table and c period the n-th classification data table.
Firstly, we insert data into the tables of data of 10 minutes _ source ip.db, the xth time cycle is set to by we here 1 minute, that is, the flow information relevant to source IP address in be collected into 1 minute is stored in 10 minutes _ source every 1 minute In the tables of data of ip.db；And it will be in non-10 minutes on the basis of current time in the tables of data of 10 minutes _ source ip.db Flow information relevant to source IP address is deleted.In the following, we insert data into the tables of data of 1 hour _ source ip.db, here The y time cycle is set to 5 minutes by we, also, when we acquire data is adopted from the tables of data of 10 minutes _ source ip.db Collect data, that is, will be relevant to source IP address in 5 minutes stored in the tables of data of 10 minutes _ source ip.db every 5 minutes Flow information is stored in the tables of data of 1 hour _ source ip.db；And by the ip.db tables of data of 1 hour _ source using current time as base Quasi- non-flow information within an hour is deleted.Finally, we insert data into the tables of data of 24 hours _ source ip.db, this In we the y time cycle is set to 20 minutes, also, when we acquire data is from the tables of data of 1 hour _ source ip.db Acquire data, that is, will be related to source IP address in 20 minutes stored in the tables of data of 1 hour _ source ip.db every 20 minutes Flow information deposit 24 hours _ source ip.db tables of data in；And by the tables of data of 24 hours _ source ip.db with it is current when Flow information within non-24 hours on the basis of quarter is deleted.
The data of the tables of data of 10 minutes _ proto.db, the tables of data of 1 hour _ proto.db and 24 hours _ proto.db The tables of data of the fill method of table and 10 minutes _ source ip.db, the ip.db tables of data in 1 hour _ source and 24 hours _ source ip.db The fill method of tables of data is identical, and which is not described herein again.
Through the above scheme, we establish multiple period classification data tables, the i.e. data of 10 minutes _ proto.db It is table, the tables of data of 1 hour _ proto.db, the tables of data of 24 hours _ proto.db, the tables of data of 10 minutes _ source ip.db, 1 small When _ tables of data of source ip.db and the tables of data of 24 hours _ source ip.db, and when corresponding flow information has been inserted different Between in section classification data table.If user needs more polytypic integrated flow information, can close according to the demand of user The multiple classification data tables of suitable extension reduce several classification data tables.Furthermore, it is desirable to it is noted that the period information In acquisition as in scheme one, as long as example, collecting the time of information in the tables of data of 10 minutes source ip.db less than 10 Minute in can, the acquisition time of the tables of data of 1 hour source ip.db be also less than 10 minutes can because be to Information is collected in the tables of data of 10 minutes source ip.db；The acquisition time of the tables of data of 24 hours _ source ip.db is just less than 1 hour Can be with because be to the tables of data of 1 hour _ source ip.db in collect information.Storage integrated flow information is finally reached to use The purpose of least memory capacity.
User can inquire according to different time requirement and different classificating requirements into different tables of data.If Need to show the data of nearly 10 minutes endogenous ip, then coming out data query into the tables of data of 10 minutes _ source ip.db；Such as Fruit needs to show the data of agreement in nearly 1 hour, then coming out data query into the tables of data of 1 hour _ proto.db；Such as Fruit needs to show the data of nearly 24 hours endogenous ip, then coming out data query into the tables of data of 24 hours _ source ip.db.
What needs to be explained here is that in scheme two, in xth here, y, z time cycle and scheme one first and second, Three time cycles can be the same or different, and first, second and third in a, b, c time cycle here and scheme one Period can be the same or different, that is, scheme two can be established on the basis of scheme one, can also be independent of scheme One redefines time cycle and period and establishes.
We will illustrate the superiority of above scheme of the present invention by attached drawing below.
Firstly, as can be seen from Figure 5 coming in 24 hours in flow information, in this time range of 9:00-16:30 The flow-rate ratio of application protocol is more because within this time being mostly company's work hours, employees work while Web page browsing is done, p2p downloads resource, watches a little videos etc., may will affect the working efficiency of employees in this way.In order to fill What all does between dividing the working efficiency for improving employee either to supervise employees at work, we can be in this period It is interior that certain agreements are forbidden, make employees can preoccupied work, or open audit function the net of employees Network behavior is detailed to be recorded.In addition to this, we can also judge whether have in certain time by this tendency chart Flow attacking.If the flow of http agreement rapidly increases to very high within 9:00-16:30 this period.We look into again The flow information of source ip once, if the flow of the http agreement of one or several source ip be also rapidly increase to it is very high, We are it may determine that these sources ip is in the flow attacking for doing http agreement.
In addition, passing through the display in following tables two, it is obvious that in the integrated flow information table of the source ip in nearly ten minutes 10.10.25.251 total flow uses more flow, if feel flow information that the ip is used it is problematic or The ip should not be allowed using so more flow informations, we can carry out the source address ip according to the flow information of table two Speed limit current limliting is blocked or used completely, is controlled for example, the flow that control strategy passes through 10.10.25.251 can be added System, equipment cannot be passed through by making to be blocked by the flow information of the ip, that is, the flow of source ip is all thrown away.After a time, such as Fruit user wants to restore the flow of source ip, then the control strategy of addition is deleted can；Alternatively, right 10.10.25.251 speed limit current limliting is carried out, until the flow of the ip is restored to flow speed desired by user, after a time, if User feels the flow speed inadequate meet demand, then just deleting the strategy of speed limit current limliting can.
Table two
Source IP Uplink traffic (KB) Downlink traffic (KB) Total flow (KB)
10.10.25.251 20581 47493 68074
10.10.25.80 3968 53052 57020
10.10.25.6 1127 10215 11342
10.10.25.11 1235 7398 8633
Secondly, passing through the display of table three, it is obvious that a sudden peal of thunder in the integrated flow information table of the agreement in nearly ten minutes Total flow uses more flow, if feeling not allowing user using sudden peal of thunder agreement in the network segment, we can To be added control strategy to the agreement according to the flow information of table three, the flow that sudden peal of thunder agreement passes through is controlled, example Such as, make all to be blocked by the flow information of sudden peal of thunder agreement and cannot pass through equipment, that is, the flow of sudden peal of thunder agreement is all thrown away, mistake The section time, if user wants to restore the flow of sudden peal of thunder agreement, so delete the control strategy of addition can；Alternatively, Speed limit current limliting is carried out to sudden peal of thunder agreement, until the flow of sudden peal of thunder agreement is restored to flow speed desired by user, after a time, such as Fruit user feels the flow speed inadequate meet demand, then just deleting the strategy of speed limit current limliting can.
Table three
Agreement Uplink traffic (KB) Downlink traffic (KB) Total flow (KB)
A sudden peal of thunder 24042 41116 65158
QQ music 11 312 323
Taobao 33 59 92
Youku.com 12 23 35
Therefore, by the above content it is found that can intuitively judge flow attacking using the solution of the present invention, so as to Family takes timely measure, and user can more intuitive, neatly control the flow of some source ip or agreement, facilitate user Management to network traffic information.
In the present invention program, according to the statement of the embodiment above, we done again on the basis of above scheme into The extension of one step, both, can dynamic mapping according to demand carry out dynamic mapping storage method.For example, if user is initially to need Source ip is shown to inquire data information, demand has been converted again after a period of time, has needed display protocol to inquire data information, but We establish the tables of data according to source ip at the beginning, do not establish the tables of data according to agreement, then according to this demand I Need to store flow information using a kind of dynamic transformation classification information.Concrete scheme is as follows:
Preferably, this method further include:
Divide when only establishing the m for storing the flow information relevant to source IP address in preset different time sections When class tables of data, convert the flow information relevant to source IP address stored in m classification data table to and source IP address phase Corresponding and protocol-dependent flow information.Alternatively,
When only establishing for storing n-th point with the protocol-dependent flow information in preset different time sections When class tables of data, it is converted into what is stored in the n-th classification data table and the association with the protocol-dependent flow information Discuss corresponding flow information relevant to the source IP address.
In embodiments of the present invention, start that flow information can be stored according to the source address ip, that is, each of database Table be established according to the ip address information of each source ip, for example, only establish the tables of data of 10 minutes _ source ip.db, 1 hour _ The tables of data of the tables of data of source ip.db and 24 hours _ source ip.db needs the information according to agreement if user changes demand When query information, then being transformed to establish database table according to the type of agreement according to the information that source ip is collected into originally, and Information is dumped in the database table of agreement；That is, the form table for being Fig. 4 by the form table transform of Fig. 3；By 10 minutes _ source The tables of data of ip.db tables of data, the tables of data of 1 hour _ source ip.db and 24 hours _ source ip.db be transformed to respectively 10 minutes _ The tables of data of the tables of data of proto.db, the tables of data of 1 hour _ proto.db and 24 hours _ proto.db.User can in this way Easily, quickly to delete and inquire required information according to agreement.
For above-mentioned embodiment, in the prior art usually be all handled in the memory of equipment, once equipment There is exception, whole flow informations will be lost.Therefore, flow information, Wo Menke can be normally shown after restarting for equipment To dump to flow information in external memory, concrete scheme is as follows.
Preferably, this method further include: at interval of preset 4th time cycle, by the flow in the database in memory Information is compressed, and by the storage of compressed flow information into external memory；It, will be in external memory after equipment exception occurs and restarts The flow information of storage is put into memory, and is shown after flow information is decompressed and shown on the page in equipment.
It should be noted that the 4th time cycle here with the above-mentioned time cycle merely to distinguish, not There is the purpose of any sequence or expression size, the 4th time cycle can be according to concrete application scene self-defining.
In embodiments of the present invention, a time timer can be added, for example, will be in memory database every 5 minutes Flow information compression after dump in external memory, when equipment occurs abnormal, when failure is restarted, the flow information in external memory is fetched After memory, decompression is shown to the page.At this moment a period of time will be consumed when equipment is restarted, and the flow information in this period Be it is no, we can fill out 0, by flow information completion in 10 minutes, 1 hour or 24 hours.Expand to just turn-on flow rate When information monitoring, if also to fill out 0 supplement complete for the flow information that is less than in the period that 10 minutes are so less than of information collected, Similarly 1 hour, 24 hours is also the same.
In order to achieve the above object, the invention also provides a kind of storage systems 01, as shown in Fig. 2, the system includes: to adopt Collection module 02, tables of data establish module 03 and the first memory module 04.
Acquisition module 02, for the timing flow information that acquisition gateway equipment generates from kernel.
Tables of data establishes module 03, for establish for store the flow information in preset different time sections it is multiple not Same tables of data.
First memory module 04, it is corresponding for the flow information stored in database to be stored according to the preset period In tables of data.
Preferably,
The system further include: categorization module 05, the second memory module 06 and index establish module 07.
Categorization module 05, for classifying to the flow information of acquisition.
Second memory module 06, for sorted flow information to be stored in database according to classification.
Index establishes module 07, for establishing database index corresponding with classifying.
The flow information of 05 pair of categorization module acquisition carries out classification and refers to: according to source network Protocol IP address and/or agreement Classify to flow information.
Wherein, which includes: network web protocol, peer-to-peer network p2p agreement and instant communication protocol.
Preferably, tables of data is established module 03 and is established for storing the multiple of the flow information in preset different time sections Different tables of data refer to:
Period tables of data corresponding from the different periods is established, and/or establishes under section in different times and divides The corresponding period classification data table of class.
Preferably, the flow information of acquisition is stored in corresponding tables of data according to the preset period by the first memory module 04 In refer to:
At interval of preset period first time, from the deposit of the flow information in period first time acquired in kernel In first time period tables of data；And it will be in the non-first time period on the basis of current time in first time period tables of data Flow information is deleted.
At interval of preset second time period, the preset second time period that will be stored in first time period tables of data In interior flow information deposit second time period tables of data；And by second time period tables of data on the basis of current time Flow information in non-second time period is deleted.
At interval of the preset third time cycle, the preset third time cycle that will be stored in second time period tables of data In interior flow information deposit third period tables of data；And by third period tables of data on the basis of current time Flow information in the non-third period is deleted.
Wherein, period first time and second time period are less than first time period；When the third time cycle is less than second Between section；First time period is less than second time period；Second time period is less than the third period.
Preferably, the flow information of acquisition is stored in corresponding tables of data according to the preset period by the first memory module 04 In also refer to:
At interval of the preset xth time cycle, by from the preset xth time cycle acquired in kernel with source IP In the relevant flow information deposit a period m classification data table of location, and by a period m classification data table with Flow information relevant to source IP address in the non-a period on the basis of current time is deleted；And/or it will be adopted from kernel Being stored in the n-th classification data of a period table with protocol-dependent flow information in the preset xth time cycle of collection, and By in the non-a period on the basis of current time in a period the n-th classification data table with protocol-dependent flow Information deletion.
At interval of the preset y time cycle, by the preset y time cycle stored in database with source IP In the relevant flow information deposit b period m classification data table of location, and by b period m classification data table with Flow information relevant to source IP address in the non-b period on the basis of current time is deleted；And/or it will be deposited in database Being stored in the n-th classification data of b period table with protocol-dependent flow information in the preset y time cycle of storage, and By in the non-b period on the basis of current time in b period the n-th classification data table with protocol-dependent flow Information deletion.
At interval of the preset z time cycle, by the preset z time cycle stored in database with source IP In the relevant flow information deposit c period m classification data table of location, and by c period m classification data table with Flow information relevant to source IP address in the non-c period on the basis of current time is deleted；And/or it will be deposited in database Being stored in the n-th classification data of c period table with protocol-dependent flow information in the preset z time cycle of storage, and By in the non-c period on the basis of current time in c period the n-th classification data table with protocol-dependent flow Information deletion.
Wherein, xth time cycle and y time cycle are less than a period；The z time cycle is less than the b period； The a period is less than the b period；The b period is less than the c period.
Preferably, tables of data is established module 03 and is also used to:
Divide when only establishing the m for storing the flow information relevant to source IP address in preset different time sections When class tables of data, convert the flow information relevant to source IP address stored in m classification data table to and source IP address phase Corresponding and protocol-dependent flow information.Alternatively,
When only establishing for storing classifying number in preset different time sections with the n-th of protocol-dependent flow information When according to table, and source IP corresponding with agreement is converted into protocol-dependent flow information by what is stored in the n-th classification data table The relevant flow information in address.
Preferably, system further include: third memory module 08 and display module 09.
Third memory module 08 is used at interval of preset 4th time cycle, by the flow in the database in memory Information is compressed, and by the storage of compressed flow information into external memory；It, will be in external memory after equipment exception occurs and restarts The flow information of storage is put into memory.
Display module 09 shows the page in equipment for display after decompressing flow information.
The invention proposes by using multiple tables of data, and can the extension of appropriateness multiple tables of data the accumulative stream of storage The method for measuring information.Usual way is the one big table of design to store all integrated flow information, has stored user in this way A large amount of memory capacity is needed after required information, also wastes many time when being inserted into and inquiring integrated flow information. Either only the information preservation of integrated flow in memory, after such equipment restarting integrated flow information just without, Accumulative effect is not achieved at all.The methods of different multiple tables of data is set according to the demand of user, not only can solve The problem of storing integrated flow information can also reduce the memory capacity of storage integrated flow information and reduce insertion and delete tired Count the time of flow information data.To just reduce the cost of a large amount of memory, greatly reduce equipment itself at This, also improves the speed to data manipulation.
Meanwhile the present invention program extends again and establishes tables of data by the demand of user in every kind of database, so not only Can clean and tidy storage flow information can also greatly improve the speed of database table insertion, deletion and query information because Information data amount inside each tables of data has become smaller much than mixing the information content deposited originally, such operation data information when Between just shorten, greatly improve operating efficiency.
Further, the present invention program extends dynamic mapping according to demand, carrys out the implementation of dynamic mapping storage method Scheme.For example, the storage mode of source ip type to be converted to the storage mode of protocol type according to demand, this scheme also can Conveniently delete and inquire required information.Greatly improve operating efficiency.
Finally, the present invention program increases equipment occur that flow information is dumped to external memory after exception, after equipment starting again The embodiment that memory is returned in the information unloading of external memory thus can be avoided when restarting after exception occurs in equipment, add up The mistake that flow information is lost.And the flow information zero padding of this period will be restarted, can be obtained by this way one it is complete Whole integrated flow information trend figure.
In conclusion present invention has the advantage that
1, the method for dividing multilist to store integrated flow information, the size that can not only reduce memory capacity can also reduce The speed of inquiry and insertion flow information, to substantially increase the performance of equipment.
2, multiple storing data libraries table suitably can be extended by type according to the demand of user, it is big to be finally reached reduction It measures the purpose of memory capacity and greatly improves the purpose of operating efficiency.
3, it further expands to and converts storage mode according to the demand of user, accomplish more suitably to extend by type multiple Data table memory, the purpose for being finally reached the purpose for reducing a large amount of memory capacity and greatly improving operating efficiency.
4, it increases flow information after abnormal restarting occurs in equipment and dumps to external memory, again by the information of external memory after equipment starting Memory, and the flow information zero padding that will restart this period are returned in unloading.A complete accumulative stream can finally be obtained Measure information database table and tendency chart.
It should be noted that embodiment described above be merely for convenience of it will be understood by those skilled in the art that, and It is not used in and limits the scope of the invention, under the premise of not departing from inventive concept of the invention, those skilled in the art couple Any obvious replacement and improvement that the present invention is made etc. are within the scope of the present invention.

Claims (14)

1. a kind of storage method, which is characterized in that the described method includes:
The timing flow information that acquisition gateway equipment generates from kernel；
Establish multiple and different tables of data for storing the flow information in preset different time sections；The different time sections For different durations incremented by successively；
The flow information of acquisition is stored in corresponding tables of data according to the preset period, wherein most grow in short-term Period corresponding tables of data in data obtained from the flow information of acquisition, the non-most long in short-term period counts accordingly It is obtained from the period corresponding tables of data of a upper duration according to the data in table.
2. storage method as described in claim 1, which is characterized in that
The method also includes: classify to the flow information of acquisition；By the sorted flow information according to institute Classification deposit database is stated, and establishes database index corresponding with the classification；
It includes: according to source network Protocol IP address and/or agreement to described that the flow information of described pair of acquisition, which carries out classification, Flow information is classified；
Wherein, the agreement includes: network web protocol, peer-to-peer network p2p agreement and instant communication protocol.
3. storage method as claimed in claim 2, which is characterized in that the foundation is for storing in preset different time sections Multiple and different tables of data of flow information include:
Period tables of data corresponding from the different period is established, and/or is established under the different period Period classification data table corresponding with the classification.
4. storage method as claimed in claim 3, which is characterized in that the flow information by acquisition is according to preset The period is stored in corresponding tables of data
It, will be from the stream in preset period first time acquired in the kernel at interval of preset period first time It measures in information deposit first time period tables of data；And it will be non-on the basis of current time in the first time period tables of data The flow information in first time period is deleted；
At interval of preset second time period, the preset second time period that will be stored in the first time period tables of data In interior flow information deposit second time period tables of data；And by the second time period tables of data with current time On the basis of non-second time period in the flow information delete；
At interval of the preset third time cycle, the preset third time cycle that will be stored in the second time period tables of data In interior flow information deposit third period tables of data；And by the third period tables of data with current time On the basis of the non-third period in the flow information delete；
Wherein, period first time and the second time period are less than the first time period；Week third time Phase is less than the second time period；The first time period is less than the second time period；The second time period is less than described The third period.
5. storage method as claimed in claim 3, which is characterized in that the flow information by acquisition is according to preset The period is stored in corresponding tables of data further include:
At interval of the preset xth time cycle, by from the preset xth time cycle acquired in the kernel with institute It states in the relevant flow information deposit a period m classification data table of source IP address, and by a period m The flow letter relevant to the source IP address in the non-a period on the basis of current time in classification data table Breath is deleted；And/or by from the preset xth time cycle acquired in the kernel with the protocol-dependent flow Information be stored in the n-th classification data of a period table in, and by the n-th classification data of a period table with current time On the basis of the non-a period in deleted with the protocol-dependent flow information；
At interval of the preset y time cycle, the preset y that will be stored in a period m classification data table In the flow information deposit b period m classification data table relevant to the source IP address in time cycle, and will In the non-b period on the basis of current time in the b period m classification data table with the source IP address The relevant flow information is deleted；And/or the preset y time that will be stored in the n-th classification data of a period table In period in protocol-dependent flow information deposit the n-th classification data of b period table, and by the b In the non-b period on the basis of current time in period the n-th classification data table with it is described protocol-dependent described Flow information is deleted；
At interval of the preset z time cycle, the preset z that will be stored in the b period m classification data table In the flow information deposit c period m classification data table relevant to the source IP address in time cycle, and will In the non-c period on the basis of current time in the c period m classification data table with the source IP address The relevant flow information is deleted；And/or the preset z time that will be stored in the n-th classification data of b period table In period in protocol-dependent flow information deposit the n-th classification data of c period table, and by the c In the non-c period on the basis of current time in period the n-th classification data table with it is described protocol-dependent described Flow information is deleted；
Wherein, the xth time cycle and the y time cycle are less than a period；The z time cycle is small In the b period；The a period is less than the b period；The b period is less than the c time Section.
6. storage method as claimed in claim 5, which is characterized in that the method also includes:
Divide when only establishing the m for storing the flow information relevant to the source IP address in preset different time sections When class tables of data, convert the flow information relevant to the source IP address stored in the m classification data table to and institute It is corresponding with the protocol-dependent flow information to state source IP address；Alternatively,
When only establishing for storing classifying number in preset different time sections with the n-th of the protocol-dependent flow information When according to table, it is converted into what is stored in the n-th classification data table and the agreement phase with the protocol-dependent flow information Corresponding flow information relevant to the source IP address.
7. storage method as described in claim 1, which is characterized in that the method also includes: when at interval of the preset 4th Between the period, the flow information in the database in memory is compressed, and the compressed flow information is stored Into external memory；After equipment exception occurs and restarts, the flow information stored in the external memory is put into the memory, And display after flow information decompression is shown on the page in equipment.
8. a kind of storage system, which is characterized in that the system comprises: acquisition module, tables of data establish module and the first storage Module；
The acquisition module, for the timing flow information that acquisition gateway equipment generates from kernel；
The tables of data establishes module, for establishing for storing the multiple and different of the flow information in preset different time sections Tables of data；The different time sections are different durations incremented by successively；
First memory module, the flow information for that will acquire is according to the preset corresponding number of period deposit According in table, wherein the data in most period corresponding tables of data long in short-term are obtained from the flow information of acquisition, non-most short Data in the period of duration corresponding tables of data are obtained from the period corresponding tables of data of a upper duration.
9. storage system as claimed in claim 8, which is characterized in that the system also includes: categorization module, the second storage mould Block and index establish module；
The categorization module, for classifying to the flow information of acquisition；
Second memory module, for the sorted flow information to be stored in database according to the classification；
The index establishes module, for establishing database index corresponding with the classification；
The categorization module carries out classification to the flow information of acquisition and refers to: according to source network Protocol IP address and/or association View classifies to the flow information；
Wherein, the agreement includes: network web protocol, peer-to-peer network p2p agreement and instant communication protocol.
10. storage system as claimed in claim 9, which is characterized in that it is pre- for storing that the tables of data establishes module foundation If different time sections in multiple and different tables of data of flow information refer to:
Period tables of data corresponding from the different period is established, and/or is established under the different period Period classification data table corresponding with the classification.
11. storage system as claimed in claim 10, which is characterized in that first memory module is by the flow of acquisition Information is stored in corresponding tables of data according to the preset period to be referred to:
It, will be from the stream in preset period first time acquired in the kernel at interval of preset period first time It measures in information deposit first time period tables of data；And it will be non-on the basis of current time in the first time period tables of data The flow information in first time period is deleted；
At interval of preset second time period, the preset second time period that will be stored in the first time period tables of data In interior flow information deposit second time period tables of data；And by the second time period tables of data with current time On the basis of non-second time period in the flow information delete；
At interval of the preset third time cycle, the preset third time cycle that will be stored in the second time period tables of data In interior flow information deposit third period tables of data；And by the third period tables of data with current time On the basis of the non-third period in the flow information delete；
Wherein, period first time and the second time period are less than the first time period；Week third time Phase is less than the second time period；The first time period is less than the second time period；The second time period is less than described The third period.
12. storage system as claimed in claim 10, which is characterized in that first memory module is by the flow of acquisition Information is stored in corresponding tables of data according to the preset period further include:
At interval of the preset xth time cycle, by from the preset xth time cycle acquired in the kernel with institute It states in the relevant flow information deposit a period m classification data table of source IP address, and by a period m The flow letter relevant to the source IP address in the non-a period on the basis of current time in classification data table Breath is deleted；And/or by from the preset xth time cycle acquired in the kernel with the protocol-dependent flow Information be stored in the n-th classification data of a period table in, and by the n-th classification data of a period table with current time On the basis of the non-a period in deleted with the protocol-dependent flow information；
At interval of the preset y time cycle, the preset y that will be stored in a period m classification data table In the flow information deposit b period m classification data table relevant to the source IP address in time cycle, and will In the non-b period on the basis of current time in the b period m classification data table with the source IP address The relevant flow information is deleted；And/or the preset y time that will be stored in the n-th classification data of a period table In period in protocol-dependent flow information deposit the n-th classification data of b period table, and by the b In the non-b period on the basis of current time in period the n-th classification data table with it is described protocol-dependent described Flow information is deleted；
At interval of the preset z time cycle, the preset z that will be stored in the b period m classification data table In the flow information deposit c period m classification data table relevant to the source IP address in time cycle, and will In the non-c period on the basis of current time in the c period m classification data table with the source IP address The relevant flow information is deleted；And/or the preset z time that will be stored in the n-th classification data of b period table In period in protocol-dependent flow information deposit the n-th classification data of c period table, and by the c In the non-c period on the basis of current time in period the n-th classification data table with it is described protocol-dependent described Flow information is deleted；
Wherein, the xth time cycle and the y time cycle are less than a period；The z time cycle is small In the b period；The a period is less than the b period；The b period is less than the c time Section.
13. storage system as claimed in claim 12, which is characterized in that the tables of data is established module and is also used to:
Divide when only establishing the m for storing the flow information relevant to the source IP address in preset different time sections When class tables of data, convert the flow information relevant to the source IP address stored in the m classification data table to and institute It is corresponding with the protocol-dependent flow information to state source IP address；Alternatively,
When only establishing for storing classifying number in preset different time sections with the n-th of the protocol-dependent flow information When according to table, it is converted into what is stored in the n-th classification data table and the agreement phase with the protocol-dependent flow information Corresponding flow information relevant to the source IP address.
14. storage system as claimed in claim 8, which is characterized in that the system also includes: third memory module and display Module；
The third memory module is used at interval of preset 4th time cycle, by the stream in the database in memory Amount information is compressed, and by the compressed flow information storage into external memory；It, will after equipment exception occurs and restarts The flow information stored in the external memory is put into the memory；
The display module, for display after flow information decompression to be shown the page in equipment.
CN201510481866.3A 2015-08-03 2015-08-03 A kind of storage method and system Active CN105162622B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510481866.3A CN105162622B (en) 2015-08-03 2015-08-03 A kind of storage method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510481866.3A CN105162622B (en) 2015-08-03 2015-08-03 A kind of storage method and system

Publications (2)

Publication Number Publication Date
CN105162622A CN105162622A (en) 2015-12-16
CN105162622B true CN105162622B (en) 2019-02-12

Family

ID=54803385

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510481866.3A Active CN105162622B (en) 2015-08-03 2015-08-03 A kind of storage method and system

Country Status (1)

Country Link
CN (1) CN105162622B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108108402B (en) * 2017-12-07 2020-07-17 泰康保险集团股份有限公司 Data item paraphrasing updating method and device
CN110347661A (en) * 2019-07-05 2019-10-18 北京红山信息科技研究院有限公司 Method, apparatus, server and the storage medium that data source is quasi real time put in storage
CN110795427B (en) * 2019-09-27 2022-10-11 苏宁云计算有限公司 Data separation storage method and device, computer equipment and storage medium
CN111625541A (en) * 2020-05-11 2020-09-04 紫光云技术有限公司 Method for measuring IP (Internet protocol) of elastic public network according to flow
CN111897816B (en) * 2020-07-16 2024-04-02 中国科学院上海微系统与信息技术研究所 Interaction method of calculation information between satellites and generation method of information table applied by same

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101247271A (en) * 2008-03-24 2008-08-20 中兴通讯股份有限公司 Performance data storage method and device
CN101739455A (en) * 2009-12-24 2010-06-16 北京世纪互联宽带数据中心有限公司 Analysis method of streaming media information on demand and method thereof
CN101790257A (en) * 2009-01-22 2010-07-28 大唐移动通信设备有限公司 Method for memorizing data and network management system
CN103458456A (en) * 2013-08-27 2013-12-18 中国科学院信息工程研究所 Method and device for user behavior detection based on mobile terminal Wi-Fi data

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8850137B2 (en) * 2010-10-11 2014-09-30 Cisco Technology, Inc. Memory subsystem for counter-based and other applications

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101247271A (en) * 2008-03-24 2008-08-20 中兴通讯股份有限公司 Performance data storage method and device
CN101790257A (en) * 2009-01-22 2010-07-28 大唐移动通信设备有限公司 Method for memorizing data and network management system
CN101739455A (en) * 2009-12-24 2010-06-16 北京世纪互联宽带数据中心有限公司 Analysis method of streaming media information on demand and method thereof
CN103458456A (en) * 2013-08-27 2013-12-18 中国科学院信息工程研究所 Method and device for user behavior detection based on mobile terminal Wi-Fi data

Also Published As

Publication number Publication date
CN105162622A (en) 2015-12-16

Similar Documents

Publication Publication Date Title
CN105162622B (en) A kind of storage method and system
CN108683720B (en) Container cluster service configuration method and device
CN105138592B (en) A kind of daily record data storage and search method based on distributed structure/architecture
CN111176976B (en) Test data generation method and device, storage medium and electronic device
CN109672591B (en) Method for real-time programmable sampling in-band network telemetry
CN103559217A (en) Heterogeneous database oriented massive multicast data storage implementation method
CN103152352A (en) Perfect information security and forensics monitoring method and system based on cloud computing environment
US11625412B2 (en) Storing data items and identifying stored data items
CN103645904A (en) Cache realization method of interface calling
CN102026228B (en) Statistical method and equipment for communication network performance data
CN101409732B (en) System and method for managing network address conversion information
CN104967632B (en) Webpage abnormal data processing method, data server and system
CN103067479A (en) Network disk synchronized method and system based on file coldness and hotness
CN103530335B (en) In-stockroom operation method and device of electric power measurement acquisition system
CN111224894A (en) Traffic collection marking method and system for iOS device
WO2017000592A1 (en) Data processing method, apparatus and system
CN110069457A (en) One kind is based on scratch file delet method and system in distributed file storage system
CN113852490A (en) Method, device and equipment for realizing quality of service (QoS) of samba user in distributed storage system
CN105872731A (en) Data processing method and device
KR20160103110A (en) Network element data access method and apparatus, and network management system
CN110399534B (en) Terminal performance report generation method, device, equipment and storage medium
KR101736382B1 (en) Ems server and log data management method thereof
CN108430067A (en) A kind of Internet service mass analysis method and system based on XDR
CN115525603A (en) Storage statistics method and device, computer readable storage medium and AI device
CN105591833A (en) Flow-acquiring method based on rule engine

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant