CN102495851B - Method, system and device for storing and querying timing sequence data - Google Patents
Method, system and device for storing and querying timing sequence data Download PDFInfo
- Publication number
- CN102495851B CN102495851B CN201110367018.1A CN201110367018A CN102495851B CN 102495851 B CN102495851 B CN 102495851B CN 201110367018 A CN201110367018 A CN 201110367018A CN 102495851 B CN102495851 B CN 102495851B
- Authority
- CN
- China
- Prior art keywords
- time
- information
- granularity
- quaternary group
- duration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention discloses a method for storing timing sequence data. The method comprises the following steps of: acquiring original stream information; performing first convergence on the original stream information by using first time granularity and storing the firstly-converged stream information as first stream information; and performing second convergence on the original stream information by using second time granularity and storing the secondly-converged stream information as second stream information, wherein the second time granularity is larger than the first time granularity. The invention further discloses a device for storing timing sequence data and a method and a system for querying timing sequence data. Due to the adoption of the method, the system and the device, accurate statistical analysis and query of network flow can be realized, a real-time, reliable and accurate data basis is provided for network planning, network monitoring and the like, and the speeds of operation and maintenance responses are increased.
Description
Technical field
The present invention relates to communication network technology field, particularly a kind of storage means of time series data and memory storage, and the querying method of time series data and inquiry system.
Background technology
In express network, carry out Real-time Collection, statistics, analyze and provide fast query function to network traffic information, be the main bugbear of facing in Network Traffic Monitoring.Mostly traditional traffic statistics and inquiry are based on rill quantitative statistics, and the data volume of storage is also less, and therefore inquiry mode is also fairly simple.But traditional simple inquiry mode cannot meet the quick storage of the large-scale data of high-speed network flow statistics generation, the demand of fast query.
For some concrete data query application, patent (application number is 201010115869.2, and the applying date is on March 1st, 2010) discloses a kind of data processing and querying method and device, while inquiry for large-scale data, set up secondary index, utilize index to accelerate inquiry.The method of recording in this application safeguards that index information bothers relatively, and can not reach the object of fast query for structurized time series data.
Traditional traffic statistics and inquiry comprise following two kinds of modes:
(1) build search index.The shortcoming of this mode is to build search index needs the storage space outside occupying volume, and index maintenance also bothers relatively, and maintenance cost is high;
(2) build query caching.The shortcoming of this mode is to constantly update cache information, and time series data is constantly updated, thereby can be reduced in the query hit rate in buffer memory, cannot meet the requirement of fast query.
Summary of the invention
Object of the present invention is intended at least solve one of above-mentioned technological deficiency.
For this reason, first object of the present invention is to provide a kind of storage means of time series data, and the method can be added up accurately to network traffics, thereby for the network planning, network monitor provide in real time, reliable data foundation.Second object of the present invention is to provide a kind of memory storage of time series data.The 3rd object of the present invention is to provide a kind of querying method of time series data.The method can realize the fast query to statistics.The 4th object of the present invention is to provide a kind of inquiry system of time series data.
For achieving the above object, the embodiment of first aspect present invention provides a kind of storage means of time series data, comprises the steps: to collect primary flow information; With very first time granularity, described primary flow information is converged for the first time and the stream information after converging is for the first time stored as to first-class information; And with the second time granularity, described first-class information is converged for the second time and the stream information after converging is for the second time stored as to second information, wherein, described the second time granularity is greater than described very first time granularity.
According to the storage means of the time series data of the embodiment of the present invention, can realize the accurate statistical study to network traffics and the storage to statistics is provided, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, thereby improve the speed of O&M response.
The embodiment of second aspect present invention provides a kind of memory storage of time series data, comprising: collection module, for collecting primary flow information; The first convergence module, for converging for the first time described primary flow information with very first time granularity and the stream information after converging being for the first time stored as to first-class information; The second convergence module, for converging for the second time described first-class information with the second time granularity and the stream information after converging being for the second time stored as to second information.
According to the memory storage of the time series data of the embodiment of the present invention, can realize the accurate statistical study to network traffics and the storage to statistics is provided, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, thereby improve the speed of O&M response.
The embodiment of third aspect present invention provides a kind of querying method of time series data, comprises the steps: to collect primary flow information; With very first time granularity, described primary flow information is converged for the first time and the stream information after converging is for the first time stored as to first-class information; With the second time granularity, described first-class information is converged for the second time and the stream information after converging is for the second time stored as to second information, wherein, described the second time granularity is greater than described very first time granularity; Receive user's inquiry request, and resolve the time interval in described inquiry request; With described very first time granularity and/or the second time granularity, described time interval is divided; And from described first-class information and/or second information, inquire about time series data respectively according to division result.
Can realize the fast query of the statistics to network traffics according to the querying method of the time series data of the embodiment of the present invention, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, and then improve the speed of O&M response.
The embodiment of fourth aspect present invention provides a kind of inquiry system of time series data, comprising: gathering-device, for collecting primary flow information; Converging device, for with very first time granularity, described primary flow information being converged for the first time and the stream information after converging being for the first time stored as to first-class information, and with the second time granularity, described first-class information is converged for the second time and the stream information after converging is for the second time stored as to second information; Database, for storing described first-class information and described second information; Inquiry unit, for receiving user's inquiry request, and resolve the time interval in described inquiry request, and with described very first time granularity and/or the second time granularity, described time interval is divided, and inquire about time series data in the first-class information of storing respectively according to division result and/or second information from described database.
Can realize the fast query of the statistics to network traffics according to the inquiry system of the time series data of the embodiment of the present invention, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, and then improve the speed of O&M response.
The aspect that the present invention is additional and advantage in the following description part provide, and part will become obviously from the following description, or recognize by practice of the present invention.
Brief description of the drawings
The present invention above-mentioned and/or additional aspect and advantage will become from the following description of the accompanying drawings of embodiments obviously and easily and understand, wherein:
Fig. 1 is according to the FB(flow block) of the storage means of the time series data of the embodiment of the present invention;
Fig. 2 is according to the schematic diagram of the storage means of the time series data of the embodiment of the present invention;
Fig. 3 is according to the structural drawing of the memory storage of the time series data of the embodiment of the present invention;
Fig. 4 is according to the structural drawing of the first convergence module of the embodiment of the present invention;
Fig. 5 is according to the structural drawing of the second convergence module of the embodiment of the present invention;
Fig. 6 is according to the FB(flow block) of the querying method of the time series data of the embodiment of the present invention;
Fig. 7 is according to the schematic diagram of the querying method of the time series data of the embodiment of the present invention; With
Fig. 8 is according to the structure of the inquiry system of the embodiment of the present invention.
Embodiment
Describe embodiments of the invention below in detail, the example of described embodiment is shown in the drawings, and wherein same or similar label represents same or similar element or has the element of identical or similar functions from start to finish.Be exemplary below by the embodiment being described with reference to the drawings, only for explaining the present invention, and can not be interpreted as limitation of the present invention.
With reference to description and accompanying drawing below, these and other aspects of embodiments of the invention will be known.In these descriptions and accompanying drawing, specifically disclose some specific implementations in embodiments of the invention, represent some modes of the principle of implementing embodiments of the invention, but should be appreciated that the scope of embodiments of the invention is not limited.On the contrary, embodiments of the invention comprise all changes, amendment and the equivalent within the scope of spirit and the intension that falls into additional claims.
Describe according to the storage means of the time series data of the embodiment of the present invention below with reference to Fig. 1 and Fig. 2.
As shown in Figure 1, the storage means of the time series data that the embodiment of the present invention provides, comprises the steps:
Step S101, collects primary flow information.
Step S102, converges for the first time primary flow information with very first time granularity and the stream information after converging is for the first time stored as to first-class information.
Step S103, converges for the second time first-class information with the second time granularity and the stream information after converging is for the second time stored as to second information, and wherein, the second time granularity is greater than very first time granularity.
According to the storage means of the time series data of the embodiment of the present invention, can realize the accurate statistical study to network traffics and the storage to statistics is provided, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, thereby improve the speed of O&M response.
The storage means of the time series data below in conjunction with Fig. 2 to the embodiment of the present invention is described in detail.
As shown in Figure 2, the storage means of the time series data that the embodiment of the present invention provides, comprises the steps:
Step S201, receives primary flow information.
The network traffics netflow data that gather switch output, wherein, netflow data are to be produced by the traffic statistics module of switch, can truly reflect the flow information in network.Then, extract primary flow information according to netflow message format.Wherein, primary flow information comprises source IP (InternetProtocol, the agreement interconnecting between network) address, object IP address, flows into switch ports themselves index, flows out switch ports themselves index, duration and start time.Wherein, source IP address, object IP address, inflow switch ports themselves index and outflow switch ports themselves index form quaternary group information jointly.
Step S202, whether the primary flow information that judgement receives arrives very first time granularity, if so, performs step S203, otherwise returns to step S201, and repeated execution of steps S201 and step S202 are until receive primary flow information accumulation to very first time granularity.
In one embodiment of the invention, very first time granularity is a minute level, for example, can be 5 minutes.Receiving after network traffics netflow data, checking that accumulative total receives data time and whether arrives 5 minutes, if arrive 5 minutes, proceeding to step S203, otherwise proceed to step S201.
Step S203, carries out traffic aggregation for the first time according to the four-tuple of stream unit.
The four-tuple of all streams unit to primary flow information in very first time granularity is added up.Wherein quaternary group information comprises source IP address, object IP address, flows into switch ports themselves index and flows out switch ports themselves index.Then the quaternary group information of statistics is converged to generate for the first time first-class information, and export this first-class information to step S204.Wherein first-class information comprises quaternary group information and corresponding the first start time and the first duration of quaternary group information.In one embodiment of the invention, the first duration was less than very first time granularity.
For example: from 1:00 to 1:05, the quaternary group information of all streams unit is added up, very first time granularity is 5 minutes.Wherein, the time that data flow into switch is 1:02, and the time that data flow out switch is 1:04, and the first start time was 1:02, and the first duration was 2 minutes.First-class information comprises quaternary group information and corresponding the first start time and first duration of the data of the switch of flowing through.
Step S204, is stored as first-class information by the stream information of traffic aggregation for the first time.
The first-class information that step S203 is obtained after converging is for the first time stored, and designated store is in the storage unit of corresponding and very first time granularity.
Step S205, whether the first-class information that judgement receives arrives the second time granularity, if so, performs step S206, otherwise returns to step S201, and repeated execution of steps S201 is to step S205 until receive primary flow information accumulation to the second time granularity.
In one embodiment of the invention, the second time granularity is a hour level, for example, can be 1 hour.Whether the first-class information data time that inspection accumulative total receives arrives 1 hour, if arrive 1 hour, proceeds to step S206, otherwise proceeds to step S201.
Step S206, carries out traffic aggregation for the second time according to the four-tuple of stream unit.
The four-tuple of all first-class information to first-class information in very first time granularity is added up.Then the quaternary group information of statistics is converged to generate for the second time second information, and export this second information to step S207.Wherein, second information comprises quaternary group information and corresponding the second start time and second duration of quaternary group information of first-class information.The second start time and the second duration are by converging and obtain the first start time and the first duration according to quaternary group information.In one embodiment of the invention, the second duration was less than the second time granularity.
For example: from 1:00 to 2:00, the quaternary group information of first-class information is added up, the second time granularity is 1 hour.Wherein, the time that data flow into switch is 1:02, and the time that data flow out switch is 1:55, and the second start time was 1:02, and the second duration was 53 minutes.Second information comprises quaternary group information and corresponding the second start time and second duration of first-class information.Particularly, the quaternary group information of 1:00～1:05, the quaternary group information of 1:05～1:10 in first-class information ... 12 quaternary group informations such as the quaternary group information of .1:55～2:00.Converge for the second time and do not need the quaternary group information of the primary flow information to the switch of flowing through from 1:00 to 2:00 to converge, only need to from converge for the first time the first-class information obtaining, find corresponding to the quaternary group information of the second time granularity and converge.
Step S207, is stored as second information by the stream information of traffic aggregation for the second time.
The second information that step S206 is obtained after converging is for the second time stored, and designated store is in the storage unit of corresponding and the second time granularity.
It should be noted that, need to separately store for converging the stream information obtaining under different time granularity, store into respectively in different storage unit.
Be understandable that, the number of times converging is not limited to twice, also can be for repeatedly.For example, then after having carried out converging for the second time, with the 3rd time granularity, second information is converged to generate the 3rd stream information, and store in the storage space of appointment.Wherein, the 3rd time granularity can be 8 hours.The number of times converging and the time granularity at every turn converging can be arranged according to the load of machinery systems of switch by keeper.
According to the storage means of the time series data of the embodiment of the present invention, can adopt different time granularities to converge to obtain corresponding stream information to the flow of the switch of flowing through, thereby realize the accurate statistical study to network traffics and the storage to statistics is provided, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, thereby improve the speed of O&M response.
Describe according to the memory storage 300 of the time series data of the embodiment of the present invention below with reference to Fig. 3.
The memory storage 300 of the time series data that as shown in Figure 3, the embodiment of the present invention provides comprises collection module 310, the first convergence module 320 and the second convergence module 330.Wherein, collection module 310 is for collecting primary flow information, the first convergence module 320 is for very first time granularity, primary flow information being converged for the first time and the stream information after converging being for the first time stored as to first-class information, and the second convergence module 330 is for converging for the second time first-class information with the second time granularity and the stream information after converging being for the second time stored as to second information.
Collection module 310 gathers the network traffics netflow data of switch output, extracts primary flow information, and this primary flow information is sent to the first convergence module 320 according to netflow message format.Wherein, primary flow information comprises source IP (Internet Protocol, the agreement interconnecting between network) address, object IP address, flows into switch ports themselves index, flows out switch ports themselves index, duration and start time.Wherein, source IP address, object IP address, inflow switch ports themselves index and outflow switch ports themselves index form quaternary group information jointly.
As shown in Figure 4, the first convergence module 320 comprises that the first statistics submodule 321 and first converges submodule 322.Wherein, the first statistics submodule 321 is added up for the quaternary group information of all streams unit in very first time granularity to primary flow information, and first converges submodule 322 converges to generate first-class information for the first time for the quaternary group information that the first statistics submodule 321 is added up.Wherein, first-class information comprises quaternary group information and corresponding the first start time and the first duration of quaternary group information.In one embodiment of the invention, the first duration was less than very first time granularity.
Particularly, the first statistics submodule 321 judges whether the primary flow information receiving arrives very first time granularity, and if so, to primary flow information, the four-tuple of all streams unit in very first time granularity is added up.In one embodiment of the invention, very first time granularity is a minute level, for example, can be 5 minutes.Receiving after network traffics netflow data, the first statistics submodule 321 checks that accumulative total receives data time and whether arrives 5 minutes, and if it is to primary flow information, the four-tuple of all streams unit in 5 minutes is added up.Then converge submodule 322 by first the quaternary group information of statistics is converged to generate first-class information for the first time, and export this first-class information to second convergence module 330.
Second converges submodule 322 stores the first-class information obtaining after converging for the first time, and designated store is in the storage unit of corresponding and very first time granularity.
As shown in Figure 5, the second convergence module 330 comprises that the second statistics submodule 331 and second converges submodule 332, wherein, the second statistics submodule 331 is added up for the quaternary group information of all first-class information in the second time granularity to first-class information, and second converges submodule 332 for the quaternary group information of the second statistics submodule statistics is converged to generate second information for the second time.Wherein, second information comprises quaternary group information and corresponding the second start time and the second duration of quaternary group information.Wherein the second start time and the second duration are for converging and obtain the first start time and the first duration according to quaternary group information.In one embodiment of the invention, the second duration was less than the second time granularity.
The second statistics submodule 331 is to adding up from the first four-tuple that converges first-class information that submodule 322 the exports all first-class information in very first time granularity.In one embodiment of the invention, the second time granularity is a hour level, for example, can be 1 hour.The second statistics submodule 331 is receiving after network traffics netflow data, check that accumulative total receives data time and whether arrives 1 hour, if so, to adding up from the first four-tuple that converges first-class information that submodule 322 the exports all first-class information in very first time granularity.Then converge submodule 332 by second the quaternary group information of statistics is converged to generate second information for the second time.
Second converges submodule 332 stores the second information obtaining after converging for the second time, and designated store is in the storage unit of corresponding and the second time granularity.
It should be noted that, need to separately store for converging the stream information obtaining under different time granularity, store into respectively in different storage unit.
Be understandable that, the number of times converging is not limited to twice, also can be for repeatedly.For example, then after having carried out converging for the second time, with the 3rd time granularity, second information is converged to generate the 3rd stream information, and store in the storage space of appointment.Wherein, the 3rd time granularity can be 8 hours.The number of times converging and the time granularity at every turn converging can be arranged according to the load of machinery systems of switch by keeper.
According to the memory storage of the time series data of the embodiment of the present invention, can adopt different time granularities to converge to obtain corresponding stream information to the flow of the switch of flowing through, thereby realize the accurate statistical study to network traffics and the storage to statistics is provided, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, thereby improve the speed of O&M response.
Fig. 6 and Fig. 7 describe according to the querying method of the time series data of the embodiment of the present invention below.
As shown in Figure 6, the querying method of the time series data of the embodiment of the present invention, comprises the steps:
Step S601, collects primary flow information.
Step S602, converges for the first time primary flow information with very first time granularity and the stream information after converging is for the first time stored as to first-class information.In one embodiment of the invention, very first time granularity is a minute level, for example, can be 5 minutes.
Step S603, converges for the second time first-class information with the second time granularity and the stream information after converging is for the second time stored as to second information.Wherein, the second time granularity is greater than very first time granularity.In one embodiment of the invention, the second time granularity is a hour level, for example, can be 1 hour.
Step S604, receives user's inquiry request, and resolves the time interval in inquiry request.
Step S605, divides time interval with very first time granularity and/or the second time granularity.
Step S606 inquires about time series data respectively according to division result from first-class information and/or second information.
Can realize the fast query of the statistics to network traffics according to the querying method of the time series data of the embodiment of the present invention, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, and then improve the speed of O&M response.
In one embodiment of the invention, with very first time granularity, primary flow information converged for the first time and the stream information after converging is for the first time stored as to first-class information, comprising the steps: to add up with the four-tuple of very first time grain all streams unit in very first time granularity to primary flow information.Wherein quaternary group information comprises source IP address, object IP address, flows into switch ports themselves index and flows out switch ports themselves index.Then the quaternary group information of statistics is converged to generate for the first time first-class information.Wherein first-class information comprises quaternary group information and corresponding the first start time and the first duration of quaternary group information.In one embodiment of the invention, the first duration was less than very first time granularity.The first-class information obtaining after converging is for the first time stored, and designated store is in the storage unit of corresponding and very first time granularity.
In one embodiment of the invention, with the second time granularity, first-class information converged for the second time and the stream information after converging is for the second time stored as to second information, comprising the steps: to add up with the four-tuple of the second time granularity all first-class information in very first time granularity to first-class information.Then the quaternary group information of statistics is converged to generate for the second time second information.Wherein, second information comprises quaternary group information and corresponding the second start time and second duration of quaternary group information of first-class information.The second start time and the second duration are by converging and obtain the first start time and the first duration according to quaternary group information.In one embodiment of the invention, the second duration was less than the second time granularity.The second information obtaining after converging is for the second time stored, and designated store is in the storage unit of corresponding and the second time granularity.
It should be noted that, need to separately store for converging the stream information obtaining under different time granularity, store into respectively in different storage unit.
Be understandable that, the number of times converging is not limited to twice, also can be for repeatedly.For example, then after having carried out converging for the second time, with the 3rd time granularity, second information is converged to generate the 3rd stream information, and store in the storage space of appointment.Wherein, the 3rd time granularity can be 8 hours.The number of times converging and the time granularity at every turn converging can be arranged according to the load of machinery systems of switch by keeper.
As shown in Figure 7, after having converged for the second time, also comprise the steps:
S701, receives the inquiry request from user.
S702, resolves the time interval in inquiry request.
The inquiry request that user is sent is resolved, the time interval that need to inquire about to obtain user.Then with very first time granularity and/or the second time granularity, time interval is divided.Particularly, time interval can be divided into following several situation:
(1) time interval can be divided into one or more very first time granularities.For example: time interval is 1:00～3:00, very first time granularity is 5 minutes, and time interval can be divided into 24 very first time granularities.
(2) time interval can be divided into one or more the second time granularities.For example: time interval is 1:00～3:00, the second time granularity is 1 hour, and time interval can be divided into 2 the second time granularities.
(3) time interval can be divided into one or more very first time granularities and one or more the second time granularity.For example: time interval is 1:00～3:00, very first time granularity is 5 minutes, and the second time granularity is 1 hour, and time interval can be divided into 12 very first time granularities and 1 the second time granularity.
S703, selects active data storage unit according to time interval.
In database according to time interval in the first-class information of storage and second information, select active data storage unit.Particularly, if time interval is divided into one or more very first time granularities, from database, select to store the data storage cell of first-class information.If time interval is divided into one or more the second time granularities, from database, select to store the data storage cell of second information.If time interval is divided into one or more very first time granularities and one or more the second time granularity, from database, select to store the data storage cell of first-class information and the data storage cell of second information.
S704, merges into into a new data storage cell by the data storage cell of selection.
The active data storage unit of selecting in step S703 is merged into new storage unit.Wherein, new storage unit has identical structure with effective storage unit.
S705, builds new query SQL statement, Query Information from new data storage cell.
According to new storage unit reconstruct query SQL statement, thus in new data storage cell Query Information.
S706, formats processing to the information checking out.
S707, the Query Information of output format.
According to the querying method of the time series data of the embodiment of the present invention, query time interval corresponding user's inquiry request is divided according to very first time granularity and/or the second time granularity, in first-class information and/or second information, inquire about time series data according to division result, thereby realize the fast query of the statistics to network traffics, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, and then improve the speed of O&M response.
Describe according to the inquiry system 800 of the time series data of the embodiment of the present invention below with reference to Fig. 8.
As shown in Figure 8, the inquiry system 800 of the time series data that the embodiment of the present invention provides, comprising: gathering-device 810, converging device 820, database 830 and inquiry unit 840.Wherein, gathering-device 810 is for collecting primary flow information, converging device 820 is for converging for the first time primary flow information with very first time granularity and the stream information after converging being for the first time stored as to first-class information, and with the second time granularity, first-class information is converged for the second time and the stream information after converging is for the second time stored as to second information, database 830 is for storing first-class information and described second information, inquiry unit 840 is for receiving user's inquiry request, and resolve the time interval in inquiry request, and with very first time granularity and/or the second time granularity, time interval is divided, with in the first-class information of storing respectively according to division result and/or second information, inquire about time series data from database.
In one embodiment of the invention, converging device 840 further comprises the first convergence module and the second convergence module.
The first convergence module is for converging for the first time primary flow information with very first time granularity and the stream information after converging being for the first time stored as to first-class information.Particularly, the first convergence module further comprises that the first statistics submodule and first converges submodule.The first statistics submodule is added up for the quaternary group information of all streams unit in very first time granularity to primary flow information, and first converges submodule for the quaternary group information of the first statistics submodule statistics is converged to generate first-class information for the first time.Wherein, first-class information comprises quaternary group information and corresponding the first start time and the first duration of quaternary group information.In one embodiment of the invention, very first time granularity is a minute level, for example, can be 5 minutes.
Particularly, the first statistics submodule is added up with the four-tuple of very first time grain all streams unit in very first time granularity to primary flow information.Wherein quaternary group information comprises source IP address, object IP address, flows into switch ports themselves index and flows out switch ports themselves index.First converges submodule converges to generate first-class information for the first time by the quaternary group information of statistics.Wherein first-class information comprises quaternary group information and corresponding the first start time and the first duration of quaternary group information.In one embodiment of the invention, the first duration was less than very first time granularity.The first-class information obtaining after converging is for the first time stored, and designated store is in the storage unit of corresponding and very first time granularity.
The second convergence module is for converging for the second time first-class information with the second time granularity and the stream information after converging being for the second time stored as to second information.Particularly, the second convergence module further comprises that the second statistics submodule and second converges submodule.Wherein, the second statistics submodule is added up for the quaternary group information of all first-class information in the second time granularity to first-class information, and second converges submodule for the quaternary group information of the second statistics submodule statistics is converged to generate second information for the second time.In one embodiment of the invention, the second time granularity is a hour level, for example, can be 1 hour.
In one embodiment of the invention, the second statistics submodule is added up with the four-tuple of the second time granularity all first-class information in very first time granularity to first-class information.Second converges submodule converges to generate second information for the second time by the quaternary group information of statistics.Wherein, second information comprises quaternary group information and corresponding the second start time and second duration of quaternary group information of first-class information.The second start time and the second duration are by converging and obtain the first start time and the first duration according to quaternary group information.In one embodiment of the invention, the second duration was less than the second time granularity.The second information obtaining after converging is for the second time stored, and designated store is in the storage unit of corresponding and the second time granularity.
It should be noted that, need to separately store for converging the stream information obtaining under different time granularity, store into respectively in different storage unit.Be understandable that, the number of times converging is not limited to twice, also can be for repeatedly.For example, then after having carried out converging for the second time, with the 3rd time granularity, second information is converged to generate the 3rd stream information, and store in the storage space of appointment.Wherein, the 3rd time granularity can be 8 hours.The number of times converging and the time granularity at every turn converging can be arranged according to the load of machinery systems of switch by keeper.
Inquiry unit 840 receives the inquiry request from user, and the inquiry request that user is sent resolves to obtain the time interval that user need to inquire about.Then with very first time granularity and/or the second time granularity, time interval is divided.Particularly, time interval can be divided into following several situation:
(1) time interval can be divided into one or more very first time granularities.For example: time interval is 1:00～3:00, very first time granularity is 5 minutes, and time interval can be divided into 24 very first time granularities.
(2) time interval can be divided into one or more the second time granularities.For example: time interval is 1:00～3:00, the second time granularity is 1 hour, and time interval can be divided into 2 the second time granularities.
(3) time interval can be divided into one or more very first time granularities and one or more the second time granularity.For example: time interval is 1:00～3:00, very first time granularity is 5 minutes, and the second time granularity is 1 hour, and time interval can be divided into 12 very first time granularities and 1 the second time granularity.
Inquiry unit 840 is selected active data storage unit according to time interval in the database in the first-class information of storage and second information.Particularly, if time interval is divided into one or more very first time granularities, inquiry unit 840 selects to store the data storage cell of first-class information from database.If time interval is divided into one or more the second time granularities, inquiry unit 840 selects to store the data storage cell of second information from database.If time interval is divided into one or more very first time granularities and one or more the second time granularity, inquiry unit 840 selects to store the data storage cell of first-class information and the data storage cell of second information from database.
The active data storage unit of selection is merged into new storage unit by inquiry unit 840.Wherein, new storage unit has identical structure with effective storage unit.Then, according to new storage unit reconstruct query SQL statement, thereby in new data storage cell Query Information.Then, inquiry unit 840 formats processing to the information checking out, and the Query Information of output format.
According to the inquiry system of the time series data of the embodiment of the present invention, query time interval corresponding user's inquiry request is divided according to very first time granularity and/or the second time granularity, in first-class information and/or second information, inquire about time series data according to division result, thereby realize the fast query of the statistics to network traffics, thereby for the application such as the network planning, network monitor provide in real time, reliably, data foundation accurately, and then improve the speed of O&M response.
Any process of otherwise describing in process flow diagram or at this or method are described and can be understood to, represent to comprise that one or more is for realizing module, fragment or the part of code of executable instruction of step of specific logical function or process, and the scope of the preferred embodiment of the present invention comprises other realization, wherein can be not according to order shown or that discuss, comprise according to related function by the mode of basic while or by contrary order, carry out function, this should be understood by embodiments of the invention person of ordinary skill in the field.
The logic and/or the step that in process flow diagram, represent or otherwise describe at this, for example, can be considered to the sequencing list of the executable instruction for realizing logic function, may be embodied in any computer-readable medium, use for instruction execution system, device or equipment (as computer based system, comprise that the system of processor or other can and carry out the system of instruction from instruction execution system, device or equipment instruction fetch), or use in conjunction with these instruction execution systems, device or equipment.With regard to this instructions, " computer-readable medium " can be anyly can comprise, device that storage, communication, propagation or transmission procedure use for instruction execution system, device or equipment or in conjunction with these instruction execution systems, device or equipment.The example more specifically (non-exhaustive list) of computer-readable medium comprises following: the electrical connection section (electronic installation) with one or more wirings, portable computer diskette box (magnetic device), random access memory (RAM), ROM (read-only memory) (ROM), the erasable ROM (read-only memory) (EPROM or flash memory) of editing, fiber device, and portable optic disk ROM (read-only memory) (CDROM).In addition, computer-readable medium can be even paper or other the suitable medium that can print described program thereon, because can be for example by paper or other media be carried out to optical scanning, then edit, decipher or process in electronics mode and obtain described program with other suitable methods if desired, be then stored in computer memory.
Should be appreciated that each several part of the present invention can realize with hardware, software, firmware or their combination.In the above-described embodiment, multiple steps or method can realize with being stored in software or the firmware carried out in storer and by suitable instruction execution system.For example, if realized with hardware, the same in another embodiment, can realize by any one in following technology well known in the art or their combination: there is the discrete logic for data-signal being realized to the logic gates of logic function, there is the special IC of suitable combinational logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) etc.
Those skilled in the art are appreciated that realizing all or part of step that above-described embodiment method carries is can carry out the hardware that instruction is relevant by program to complete, described program can be stored in a kind of computer-readable recording medium, this program, in the time carrying out, comprises step of embodiment of the method one or a combination set of.
In addition, the each functional unit in each embodiment of the present invention can be integrated in a processing module, can be also that the independent physics of unit exists, and also can be integrated in a module two or more unit.Above-mentioned integrated module both can adopt the form of hardware to realize, and also can adopt the form of software function module to realize.If described integrated module realizes and during as production marketing independently or use, also can be stored in a computer read/write memory medium using the form of software function module.
The above-mentioned storage medium of mentioning can be ROM (read-only memory), disk or CD etc.
In the description of this instructions, the description of reference term " embodiment ", " some embodiment ", " example ", " concrete example " or " some examples " etc. means to be contained at least one embodiment of the present invention or example in conjunction with specific features, structure, material or the feature of this embodiment or example description.In this manual, the schematic statement of above-mentioned term is not necessarily referred to identical embodiment or example.And specific features, structure, material or the feature of description can be with suitable mode combination in any one or more embodiment or example.
Although illustrated and described embodiments of the invention, for the ordinary skill in the art, be appreciated that without departing from the principles and spirit of the present invention and can carry out multiple variation, amendment, replacement and modification to these embodiment, scope of the present invention is by claims and be equal to and limit.
Claims (14)
1. a storage means for time series data, is characterized in that, comprises the following steps:
Collect primary flow information, wherein, described primary flow information comprises source address, destination address, inflow switch ports themselves index, flows out switch ports themselves index, duration and start time;
With very first time granularity, described primary flow information is converged for the first time and the stream information after converging is for the first time stored as to first-class information;
With the second time granularity, described first-class information is converged for the second time and the stream information after converging is for the second time stored as to second information, wherein, described the second time granularity is greater than described very first time granularity;
Wherein, described with very first time granularity, described primary flow information is converged for the first time and the stream information after converging is for the first time stored as to first-class information further comprise:
Quaternary group information to all streams unit in very first time granularity in described primary flow information is added up, and wherein, described quaternary group information comprises described source address, destination address, inflow switch ports themselves index and flows out switch ports themselves index; And
The quaternary group information of statistics is converged to generate first-class information for the first time, and wherein, described first-class information comprises described quaternary group information and corresponding the first start time and the first duration of described quaternary group information;
Describedly with the second time granularity, described first-class information is converged further and is comprised for the second time:
Quaternary group information to all first-class information in the second time granularity in described first-class information is added up; And
The quaternary group information of statistics is converged to generate second information for the second time, wherein, described second information comprises described quaternary group information and corresponding the second start time and the second duration of described quaternary group information, described the second start time and described the second duration are for according to described quaternary group information, described the first start time and the first duration being converged and obtained, and described the second duration is less than described the second time granularity.
2. the storage means of time series data as claimed in claim 1, is characterized in that, described the first duration is less than described very first time granularity.
3. the storage means of time series data as claimed in claim 1, is characterized in that, described the second time granularity is a hour level, and described very first time granularity is a minute level.
4. a memory storage for time series data, is characterized in that, comprising:
Collection module, for collecting primary flow information, wherein, described primary flow information comprises source address, destination address, inflow switch ports themselves index, flows out switch ports themselves index, duration and start time;
The first convergence module, for converging for the first time described primary flow information with very first time granularity and the stream information after converging being for the first time stored as to first-class information; The second convergence module, for converging for the second time described first-class information with the second time granularity and the stream information after converging being for the second time stored as to second information;
Wherein, described the first convergence module further comprises:
The first statistics submodule, quaternary group information for all streams unit in very first time granularity to described primary flow information is added up, wherein, described quaternary group information comprises described source address, destination address, inflow switch ports themselves index and flows out switch ports themselves index; And
First converges submodule, for the quaternary group information of described the first statistics submodule statistics is converged to generate first-class information for the first time, wherein, described first-class information comprises described quaternary group information and corresponding the first start time and the first duration of described quaternary group information;
Described the second convergence module further comprises:
The second statistics submodule, adds up for the quaternary group information of all first-class information in the second time granularity to described first-class information; And
Second converges submodule, for the quaternary group information of described the second statistics submodule statistics is converged to generate second information for the second time, wherein, described second information comprises described quaternary group information and corresponding the second start time and the second duration of described quaternary group information, described the second start time and described the second duration are for according to described quaternary group information, described the first start time and the first duration being converged and obtained, and described the second duration is less than described the second time granularity.
5. the memory storage of time series data as claimed in claim 4, is characterized in that, described the first duration is less than described very first time granularity.
6. the memory storage of the time series data as described in claim 4-5 any one, is characterized in that, described the second time granularity is a hour level, and described very first time granularity is a minute level.
7. a querying method for time series data, is characterized in that, comprises the following steps:
Collect primary flow information, wherein, described primary flow information comprises source address, destination address, inflow switch ports themselves index, flows out switch ports themselves index, duration and start time;
With very first time granularity, described primary flow information is converged for the first time and the stream information after converging is for the first time stored as to first-class information;
With the second time granularity, described first-class information is converged for the second time and the stream information after converging is for the second time stored as to second information, wherein, described the second time granularity is greater than described very first time granularity;
Receive user's inquiry request, and resolve the time interval in described inquiry request;
With described very first time granularity and/or the second time granularity, described time interval is divided;
From described first-class information and/or second information, inquire about time series data respectively according to division result;
Wherein, described with very first time granularity, described primary flow information is converged for the first time and the stream information after converging is for the first time stored as to first-class information further comprise:
Quaternary group information to all streams unit in very first time granularity in described primary flow information is added up, and wherein, described quaternary group information comprises described source address, destination address, inflow switch ports themselves index and flows out switch ports themselves index; And
The quaternary group information of statistics is converged to generate first-class information for the first time, and wherein, described first-class information comprises described quaternary group information and corresponding the first start time and the first duration of described quaternary group information;
Describedly with the second time granularity, described first-class information is converged further and is comprised for the second time:
Quaternary group information to all first-class information in the second time granularity in described first-class information is added up; And
The quaternary group information of statistics is converged to generate second information for the second time, wherein, described second information comprises described quaternary group information and corresponding the second start time and the second duration of described quaternary group information, described the second start time and described the second duration are for according to described quaternary group information, described the first start time and the first duration being converged and obtained, and described the second duration is less than described the second time granularity.
8. the querying method of time series data as claimed in claim 7, is characterized in that, describedly from described first-class information and/or second information, inquires about time series data respectively according to division result and further comprises:
If described time interval is divided into one or more very first time granularities, from described first-class information, inquire about time series data;
If described time interval is divided into one or more the second time granularities, from described second information, inquire about time series data; And
Described time interval is divided into one or more very first time granularities and one or more the second time granularity, from described first-class information and second information, inquires about time series data.
9. the querying method of time series data as claimed in claim 7, is characterized in that, also comprises:
In database according to described time interval in the described first-class information of storage and second information, select active data storage unit;
Selected active data storage unit is merged into new storage unit, and described new storage unit has identical structure with described effective storage unit; And
According to described new storage unit reconstruct query statement.
10. the querying method of time series data as claimed in claim 7, is characterized in that, described the first duration is less than described very first time granularity.
The querying method of 11. time series datas as described in claim 7-10 any one, is characterized in that, described the second time granularity is a hour level, and described very first time granularity is a minute level.
The inquiry system of 12. 1 kinds of time series datas, is characterized in that, comprising:
Gathering-device, for collecting primary flow information, wherein, described primary flow information comprises source address, destination address, inflow switch ports themselves index, flows out switch ports themselves index, duration and start time;
Converging device, for with very first time granularity, described primary flow information being converged for the first time and the stream information after converging being for the first time stored as to first-class information, and with the second time granularity, described first-class information is converged for the second time and the stream information after converging is for the second time stored as to second information;
Database, for storing described first-class information and described second information;
Inquiry unit, for receiving user's inquiry request, and resolve the time interval in described inquiry request, and with described very first time granularity and/or the second time granularity, described time interval is divided, and inquire about time series data in the first-class information of storing respectively according to division result and/or second information from described database;
Wherein, described converging device comprises further and comprising:
The first convergence module, for converging for the first time described primary flow information with very first time granularity and the stream information after converging being for the first time stored as to first-class information;
The second convergence module, for converging for the second time described first-class information with the second time granularity and the stream information after converging being for the second time stored as to second information;
Wherein, the first statistics submodule, quaternary group information for all streams unit in very first time granularity to described primary flow information is added up, and wherein, described quaternary group information comprises described source address, destination address, inflow switch ports themselves index and flows out switch ports themselves index; And
First converges submodule, for the quaternary group information of described the first statistics submodule statistics is converged to generate first-class information for the first time, wherein, described first-class information comprises described quaternary group information and corresponding the first start time and the first duration of described quaternary group information;
Described the second convergence module further comprises:
The second statistics submodule, adds up for the quaternary group information of all first-class information in the second time granularity to described first-class information; And
Second converges submodule, for the quaternary group information of described the second statistics submodule statistics is converged to generate second information for the second time, wherein, described second information comprises described quaternary group information and corresponding the second start time and the second duration of described quaternary group information, described the second start time and described the second duration are for according to described quaternary group information, described the first start time and the first duration being converged and obtained, and described the second duration is less than described the second time granularity.
The inquiry system of 13. time series datas as claimed in claim 12, it is characterized in that, described inquiry unit also for selecting active data storage unit according to described time interval in the database of the described first-class information of storage and second information, and selected active data storage unit is merged into new storage unit, and according to described new storage unit reconstruct query statement, wherein, described new storage unit has identical structure with described effective storage unit.
The inquiry system of 14. time series datas as claimed in claim 12, is characterized in that, described the first duration is less than described very first time granularity.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110367018.1A CN102495851B (en) | 2011-11-17 | 2011-11-17 | Method, system and device for storing and querying timing sequence data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110367018.1A CN102495851B (en) | 2011-11-17 | 2011-11-17 | Method, system and device for storing and querying timing sequence data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102495851A CN102495851A (en) | 2012-06-13 |
| CN102495851B true CN102495851B (en) | 2014-11-05 |
Family
ID=46187676
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110367018.1A Active CN102495851B (en) | 2011-11-17 | 2011-11-17 | Method, system and device for storing and querying timing sequence data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102495851B (en) |
Families Citing this family (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102833775B (en) * | 2012-08-31 | 2015-09-02 | 电信科学技术研究院 | A kind of method and apparatus carrying out system-level off-line test |
| JP5380608B1 (en) | 2012-09-19 | 2014-01-08 | 株式会社東芝 | Time series data storage device |
| CN104486116A (en) * | 2014-12-12 | 2015-04-01 | 北京百度网讯科技有限公司 | Multidimensional query method and multidimensional query system of flow data |
| WO2016141590A1 (en) * | 2015-03-12 | 2016-09-15 | 华为技术有限公司 | Time sequence data processing method and apparatus |
| CN106648446B (en) * | 2015-10-30 | 2020-07-07 | 阿里巴巴集团控股有限公司 | Storage method and device for time sequence data and electronic equipment |
| CN106937241B (en) | 2015-12-31 | 2021-05-18 | 华为技术有限公司 | Time sequence data detection method and device |
| CN106383864B (en) * | 2016-09-02 | 2019-08-27 | 北京百度网讯科技有限公司 | A kind of inquiry request treating method and apparatus of time series database |
| CN106777111B (en) * | 2016-12-15 | 2020-08-11 | 华南师范大学 | Time sequence retrieval index system and method for super-large scale data |
| CN108052551B (en) * | 2017-11-28 | 2021-04-30 | 北京航天云路有限公司 | Method for storing large amount of time sequence data realized on REDIS |
| CN108460094A (en) * | 2018-01-30 | 2018-08-28 | 上海天旦网络科技发展有限公司 | The method and system of storage statistical data |
| CN110046183A (en) * | 2019-04-16 | 2019-07-23 | 北京易沃特科技有限公司 | A kind of time series data polymerization search method, equipment and medium |
| CN110943883B (en) * | 2019-11-13 | 2023-01-31 | 深圳市东进技术股份有限公司 | Network flow statistical method, system, gateway and computer readable storage medium |
| CN111552722A (en) * | 2020-05-06 | 2020-08-18 | 山东汇贸电子口岸有限公司 | Method and device for processing granularity data |
| CN111966695B (en) * | 2020-10-23 | 2021-01-08 | 山东鼎滏软件科技有限公司 | Time sequence database system for oil and gas field production |
| CN115529232A (en) * | 2021-06-08 | 2022-12-27 | 中国移动通信有限公司研究院 | Control method and device for convergence and distribution equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101267349A (en) * | 2008-04-29 | 2008-09-17 | 杭州华三通信技术有限公司 | Network traffic analysis method and device |
| CN101668006A (en) * | 2009-10-12 | 2010-03-10 | 哈尔滨工程大学 | Self adaptive network traffic sampling method for anomaly detection |
| CN101729301A (en) * | 2008-11-03 | 2010-06-09 | 中国移动通信集团湖北有限公司 | Monitor method and monitor system of network anomaly traffic |
| CN101848160A (en) * | 2010-05-26 | 2010-09-29 | 钱叶魁 | Method for detecting and classifying all-network flow abnormity on line |
| CN102073735A (en) * | 2011-01-20 | 2011-05-25 | 百度在线网络技术（北京）有限公司 | Searching method and searching system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101223059B1 (en) * | 2009-02-02 | 2013-01-18 | 레벨 3 커뮤니케이션즈 엘엘씨 | Analysis of network traffic |
| CN101997701B (en) * | 2009-08-10 | 2014-12-17 | 中兴通讯股份有限公司 | Data processing method and a device thereof |
-
2011
- 2011-11-17 CN CN201110367018.1A patent/CN102495851B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101267349A (en) * | 2008-04-29 | 2008-09-17 | 杭州华三通信技术有限公司 | Network traffic analysis method and device |
| CN101729301A (en) * | 2008-11-03 | 2010-06-09 | 中国移动通信集团湖北有限公司 | Monitor method and monitor system of network anomaly traffic |
| CN101668006A (en) * | 2009-10-12 | 2010-03-10 | 哈尔滨工程大学 | Self adaptive network traffic sampling method for anomaly detection |
| CN101848160A (en) * | 2010-05-26 | 2010-09-29 | 钱叶魁 | Method for detecting and classifying all-network flow abnormity on line |
| CN102073735A (en) * | 2011-01-20 | 2011-05-25 | 百度在线网络技术（北京）有限公司 | Searching method and searching system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102495851A (en) | 2012-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102495851B (en) | Method, system and device for storing and querying timing sequence data | |
| CN104123369B (en) | A kind of implementation method of the configuration management Database Systems based on graphic data base | |
| Gao et al. | Continuous pattern detection over billion-edge graph using distributed framework | |
| CN109951463A (en) | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column | |
| CN103532739B (en) | A kind of monitoring analysis system based on network service with application | |
| CN105408881A (en) | Bandwidth metering in large-scale networks | |
| CN104683164A (en) | Method of generating monitoring topological graph by using simplified configuration item correlation model | |
| WO2023108967A1 (en) | Joint credit scoring method and apparatus based on privacy protection calculation and cross-organization | |
| CN103823660A (en) | Method and system for IT (information technology) operation and maintenance configuration management | |
| CN104486116A (en) | Multidimensional query method and multidimensional query system of flow data | |
| CN114666224A (en) | Dynamic allocation method, device, equipment and storage medium for business resource capacity | |
| CN109088747A (en) | The management method and device of resource in cloud computing system | |
| CN104778528A (en) | Method and system for obtaining smart city evaluation indexes by utilizing big data | |
| KR20200010624A (en) | Big Data Integrated Diagnosis Prediction System Using Machine Learning | |
| CN107220360A (en) | A kind of Unified Modeling storage cut-in method based on magnanimity electric power monitoring data | |
| CN109325200B (en) | Method and device for acquiring data and computer readable storage medium | |
| CN105139328B (en) | Hourage real-time predicting method and device towards license plate identification data | |
| CN110837496A (en) | Data quality management method and system based on dynamic sql | |
| CN111125450A (en) | Management method of multilayer topology network resource object | |
| CN102868601B (en) | Routing system related to network topology based on graphic configuration database businesses | |
| Zu et al. | Data-centric publish-subscribe approach for Distributed Complex Event Processing deployment in smart grid Internet of Things | |
| Validi et al. | Hybrid on/off blockchain approach for vehicle data management, processing and visualization exemplified by the ADAPT platform | |
| Ding et al. | DS-harmonizer: a harmonization service on spatiotemporal data stream in edge computing environment | |
| CN108833304A (en) | The management method and device of message in cloud data system | |
| CN110069565A (en) | A kind of method and device of distributed data base batch data processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |