CN102867071B - Management method for massive network management historical data - Google Patents
Management method for massive network management historical data Download PDFInfo
- Publication number
- CN102867071B CN102867071B CN201210401421.6A CN201210401421A CN102867071B CN 102867071 B CN102867071 B CN 102867071B CN 201210401421 A CN201210401421 A CN 201210401421A CN 102867071 B CN102867071 B CN 102867071B
- Authority
- CN
- China
- Prior art keywords
- submeter
- data
- sequence number
- current
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/024—Standardisation; Integration using relational databases for representation of network management data, e.g. managing via structured query language [SQL]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/085—Retrieval of network configuration; Tracking network configuration history
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention relates to management of historical data in a network management system, specifically to a management method for massive network management historical data. A database can be divided into tables on the basis of a two-level table division rule, and the data saved in a single table in which the massive network management historical data are stored are stored in a plurality of divided tables in the same structure after being divided, so that the capacity of the single table can be reduced, the processing efficiency of the single table can be increased, and each of the divided tables can be managed as follows: A, management of historical data in divided tables; B, insertion management of historical data; C, query management of historical data; and D, transfer management of historical data. The management method for massive network management historical data disclosed by the invention can save massive historical data and helps to efficiently insert, inquire and transfer the data without affecting the normal use of network management software, so that the working efficiency of the network management system can be improved.
Description
Technical field
The present invention relates to the management of historical data in webmaster management system, is a kind of management method for massive network management historical data specifically.
Background technology
Current network technology is maked rapid progress, broadband user's scale constantly expands, the number of devices of telecom network management system management is also in continuous growth, the data such as the daily record of telecom network management system daily management, the alarm data of equipment all need to preserve in a database, and telecom network management system is record data, query analysis data to the occupation mode of these data.Through accumulating over a long period, these data grows are huge, if effectively do not managed these data, no matter be the insertion of data or the inquiry of data, performance all can sharply decline, thus affects the normal use of webmastering software.Database use hard drive space preserves data wherein, if historical data increases without limitation, exhaust all disk spaces the most at last, so need regularly to clear up historical data, the data dump part in database seldom used carries out to other position preserving, filing.And in the process of data batchmove, webmaster then cannot use these data, if the inefficiency of dump, affect the availability of network management system, a kind of data batchmove scheme efficiently must be provided.
Summary of the invention
For the defect existed in prior art, the object of the present invention is to provide a kind of management method for massive network management historical data, can mass historical data be preserved, and the insertion of data, inquiry, transfer and do not affect the normal use of webmastering software can be carried out efficiently.
For reaching above object, the technical scheme that the present invention takes is:
A kind of management method for massive network management historical data, it is characterized in that: by secondary submeter rule fulfillment database submeter, the data that the single table preserving webmaster mass historical data is preserved, be saved in the identical submeter of multiple structure after cutting, reduce the capacity of single table, thus improve the treatment effeciency of single table, and following management is carried out to each submeter:
A historical data submeter manages,
B historical data inserts management,
C the inquiry of historical data manages,
D historical data transfer management.
On the basis of technique scheme, the file record of xml form of secondary submeter rule, wherein:
Historical data type data_type, represents that tables of data preserves the type of data,
Total number packets group_count, represents the sum divided into groups to this tables of data,
In grouping, submeter sum split_count, represents submeter number maximum in each grouping,
Submeter expert sum row_count, represents submeter expert sum, when the data volume of a submeter exceedes this value time, just needs the submeter that establishment one is new.
On the basis of technique scheme, the handling process of described historical data submeter management comprises the following steps:
A1, read and record the file of the secondary submeter rule of tables of data, obtain submeter sum split_count and submeter expert sum row_count in historical data type data_type, total number packets group_count, grouping;
Reading database submeter information table, obtain the packet number group_no of all submeters, submeter sequence number table_no, current just at usage flag current_flag and usage flag in_use_flag, these information being saved in a packet number is key, submeter information list is in the map object of value, described submeter information list comprises submeter sequence number table_no, current just at usage flag current_flag and usage flag in_use_flag, wherein the span of the key of map is from 1 to total number packets group_count, if the submeter information not having certain key corresponding in database submeter information table, the submeter information list that so this key is corresponding in map is just empty,
The map object that A2, traversal A1 obtain;
A3, according to traversal result, if in map object a current group number corresponding submeter information list be sky, then go to step A4, otherwise, go to step A5;
A4, in the grouping of current group number, create an available submeter, distributing submeter sequence number is 1, and arranging submeter current is being just 1 in usage flag, and usage flag is 1, goes to step A5 by after the information data inserting storehouse submeter information table of this submeter;
A5, the submeter information will recorded in database submeter information table, pressing packet number is key, is inserted in grouping submeter information cache one by one, this buffer memory for recording the information of the submeter created in all groupings,
By current in usage flag be just 1 submeter information, pressing packet number is key, by submeter sequence number insert grouping current submeter sequence number buffer memory, in each grouping of this caching record, current_flag is the submeter sequence number of the submeter of 1, goes to step A6;
The submeter divided into groups in current submeter sequence number buffer memory, needs to check whether it exceedes submeter expert sum;
A6, use timer waiting for, if the non-time-out of timer, wait for always, when the timer times out, if receive the signal that application program exits, then exit the handling process of historical data submeter management; Otherwise, proceed to steps A 7;
A7, the current submeter sequence number buffer memory of traversal grouping, if do not traveled through, gone to step A8, if traveled through, gone to step A6;
Submeter sequence number in A8, reading current group, according to the group number of current group and the title of submeter sequence number structure submeter, obtains submeter number of data lines;
If A9 submeter number of data lines is greater than submeter expert sum row_count in submeter rule, then goes to step A10 otherwise go to step A7;
A10, create new submeter, the submeter sequence number of new submeter is that in the grouping of submeter place, original current_flag is that the submeter sequence number value of the submeter of 1 adds 1, then in database submeter information table, insert the information of this new submeter, and the information of this new submeter is inserted in grouping submeter information cache, then submeter sequence number corresponding for the current group in the current submeter sequence number buffer memory of grouping is changed into the submeter sequence number of new submeter;
A11, to judge in current group whether the total number of submeter is greater than submeter sum split_count in grouping, if be greater than, goes to step A12, if be less than, goes to step A7;
In A12, more new database submeter information table, in this grouping, the usage flag in_use_flag of the minimum submeter of submeter sequence number is 0, identify this submeter can not insert in data and use in inquiry, and will by historical data transfer management flow processing, and from grouping submeter information cache, delete the record of this submeter, go to step A11.
On the basis of technique scheme, the handling process that described historical data inserts management comprises the following steps:
B1, group number according to the content determination data traffic packets of data inserting, submeter sequence number is obtained according to the current submeter sequence number buffer memory of grouping, according to submeter and mathematical logic corresponding relation, rearrange data inserting content, stored in packet number and submeter sequence number for key, save as in data inserting in the map of value;
Map in B2, traversal B1, has traveled through, has exited, otherwise gone to step B3;
B3, according to current group number and submeter sequence number, structure submeter title, data inserting, in current submeter, goes to step B2.
On the basis of technique scheme, the handling process of described the inquiry of historical data management comprises the following steps:
In C1, reading database submeter information table, usage flag is the submeter information of 1;
C2, analysis and consult condition, obtain the table name list of all associated packets submeters;
C3, inquire about the submeter in all C2 according to querying condition.
On the basis of technique scheme, the handling process of described historical data transfer management comprises the following steps:
D1, be the submeter information record of 0 by all usage flags in packet number reading database submeter information table, being saved in packet number is key, and the list of submeter sequence number is in the map of value;
The map obtained in D2, traversal D1, has traveled through backed off after random;
D3, according to current group number and submeter sequence number, structure submeter title, uses batch processing instrument directly to derive all line items in the submeter of specifying submeter title;
D4, from database submeter information table, delete submeter information according to packet number and submeter sequence number;
D5, from database, delete submeter according to submeter title, go to step D2.
Management method for massive network management historical data of the present invention, by secondary submeter rule fulfillment database submeter (database submeter namely: the data that single table is preserved, according to being saved in the identical table of multiple structure (being called submeter) after certain regular cutting, reduce the capacity of single table, thus improve the treatment effeciency of single table, different submeters can be accessed under different business demand, reduce lock competition, improve operational efficiency), historical data is distributed in multiple submeter, and ensure that recording number in each submeter is roughly more or less the same with the submeter capacity limited, not only ensure that the response efficiency of the inquiry of historical data, also assures that the insertion efficiency of data, improve customer experience impression, improve the operating efficiency of webmaster management system.
Management method for massive network management historical data of the present invention, by usage flag by using with need the tables of data of transfer to be separated, network management system and data batchmove function would not influence each other, and the efficiency far of the whole list processing of tables of data is higher than the efficiency of being undertaken processing by line item, decrease the impact on Database Systems, improve the efficiency of network management system entirety.
Accompanying drawing explanation
The present invention has following accompanying drawing:
Fig. 1 is according to after submeter regular partition, and submeter is shown;
Fig. 2 is the content of tables of data submeter rule file;
Fig. 3 is the data content that database submeter information table comprises;
Fig. 4 is submeter management processing flow chart;
Fig. 5 is that data insert process chart;
Fig. 6 is data query process chart;
Fig. 7 is data batchmove process chart.
Detailed description of the invention
Below in conjunction with accompanying drawing, the present invention is described in further detail.
Management method for massive network management historical data of the present invention, by secondary submeter rule fulfillment database submeter, the data that the single table preserving webmaster mass historical data is preserved, be saved in the identical submeter of multiple structure after cutting, reduce the capacity of single table, thus improve the treatment effeciency of single table, and following management is carried out to each submeter:
A historical data submeter manages,
B historical data inserts management,
C the inquiry of historical data manages,
D historical data transfer management.
On the basis of technique scheme, Fig. 2 shows the content of tables of data submeter rule file, this namely described secondary submeter rule, the file record of xml form of secondary submeter rule, wherein:
Historical data type data_type, represents that tables of data preserves the type of data,
Total number packets group_count, represents the sum divided into groups to this tables of data, namely represents that a tables of data can be divided into several grouping at most, the packet number that each grouping is corresponding unique;
In grouping, submeter sum split_count, represents submeter number maximum in each grouping, namely represents in a grouping and can comprise how many submeters, be i.e. submeter number maximum in grouping, the submeter sequence number that each submeter is corresponding unique;
Submeter expert sum row_count, represents submeter expert sum, and namely submeter expert sum row_count represents in a submeter and can comprise how many row data, i.e. data line maximum in submeter; When the data volume of a submeter exceedes this value time, just need the submeter that establishment one is new.
That is: a tables of data has at most group_count grouping, and have split_count submeter in a grouping, the capable data of row_count noted down at most by a submeter.
On the basis of technique scheme, as shown in Figure 4, the handling process of described historical data submeter management comprises the following steps:
A1, read and record the file of the secondary submeter rule of tables of data, obtain submeter sum split_count and submeter expert sum row_count in historical data type data_type, total number packets group_count, grouping;
Reading database submeter information table, obtain the packet number group_no of all submeters, submeter sequence number table_no, current just at usage flag current_flag and usage flag in_use_flag, these information being saved in a packet number is key, submeter information list is in the map object of value, described submeter information list comprises submeter sequence number table_no, current just at usage flag current_flag and usage flag in_use_flag, wherein the span of the key of map is from 1 to total number packets group_count, if the submeter information not having certain key corresponding in database submeter information table, the submeter information list that so this key is corresponding in map is just empty,
Database submeter information tableau format and content can embodiments shown in Figure 3; Fig. 3 shows the content of database submeter information table, and demonstrates a state that may occur, the information of each line item submeter; Suppose the group group_count=3 that submeter rule divides according to business function, split_count=3 submeter is allowed at most in each group, so after creating the 4th submeter, this submeter sequence number table_no be 4 submeter just become the table that current using state current_flag is 1, and the usage flag in_use_flag that submeter numbering table_no is the submeter of 1 is then set to 0, the sum of all available submeters so just can be kept to be no more than the value of split_count;
The map object that A2, traversal A1 obtain;
A3, according to traversal result, if in map object a current group number corresponding submeter information list be sky, then go to step A4, otherwise, go to step A5;
A4, in the grouping of current group number, create an available submeter, distributing submeter sequence number is 1, and arranging submeter current is being just 1 in usage flag, and usage flag is 1, goes to step A5 by after the information data inserting storehouse submeter information table of this submeter;
A5, the submeter information will recorded in database submeter information table, pressing packet number is key, is inserted in grouping submeter information cache one by one, this buffer memory for recording the information of the submeter created in all groupings,
By current in usage flag be just 1 submeter information, pressing packet number is key, by submeter sequence number insert grouping current submeter sequence number buffer memory, in each grouping of this caching record, current_flag is the submeter sequence number of the submeter of 1, goes to step A6;
The submeter divided into groups in current submeter sequence number buffer memory, needs to check whether it exceedes submeter expert sum;
A6, use timer waiting for, if the non-time-out of timer, wait for always, when the timer times out, if receive the signal that application program exits, then exit the handling process of historical data submeter management; Otherwise, proceed to steps A 7;
A7, the current submeter sequence number buffer memory of traversal grouping, if do not traveled through, gone to step A8, if traveled through, gone to step A6;
Submeter sequence number in A8, reading current group, according to the group number of current group and the title of submeter sequence number structure submeter, obtains submeter number of data lines;
If A9 submeter number of data lines is greater than submeter expert sum row_count in submeter rule, then goes to step A10 otherwise go to step A7;
A10, create new submeter, the submeter sequence number of new submeter is that in the grouping of submeter place, original current_flag is that the submeter sequence number value of the submeter of 1 adds 1, then in database submeter information table, insert the information of this new submeter, and the information of this new submeter is inserted in grouping submeter information cache, then submeter sequence number corresponding for the current group in the current submeter sequence number buffer memory of grouping is changed into the submeter sequence number of new submeter;
A11, to judge in current group whether the total number of submeter is greater than submeter sum split_count in grouping, if be greater than, goes to step A12, if be less than, goes to step A7;
In A12, more new database submeter information table, in this grouping, the usage flag in_use_flag of the minimum submeter of submeter sequence number is 0, identify this submeter can not insert in data and use in inquiry, and will by historical data transfer management flow processing, and from grouping submeter information cache, delete the record of this submeter, go to step A11.
On the basis of technique scheme, as shown in Figure 5, the handling process of described historical data insertion management comprises the following steps:
B1, group number according to the content determination data traffic packets of data inserting, submeter sequence number is obtained according to the current submeter sequence number buffer memory of grouping, according to submeter and mathematical logic corresponding relation, rearrange data inserting content, stored in packet number and submeter sequence number for key, save as in data inserting in the map of value;
Map in B2, traversal B1, has traveled through, has exited, otherwise gone to step B3;
B3, according to current group number and submeter sequence number, structure submeter title, data inserting, in current submeter, goes to step B2.
On the basis of technique scheme, as shown in Figure 6, the handling process of described the inquiry of historical data management comprises the following steps:
In C1, reading database submeter information table, usage flag is the submeter information of 1;
C2, analysis and consult condition, obtain the table name list of all associated packets submeters;
C3, inquire about the submeter in all C2 according to querying condition.
On the basis of technique scheme, as shown in Figure 7, the handling process of described historical data transfer management comprises the following steps:
D1, be the submeter information record of 0 by all usage flags in packet number reading database submeter information table, being saved in packet number is key, and the list of submeter sequence number is in the map of value;
The map obtained in D2, traversal D1, has traveled through backed off after random;
D3, according to current group number and submeter sequence number, structure submeter title, uses batch processing instrument directly to derive all line items in the submeter of specifying submeter title;
D4, from database submeter information table, delete submeter information according to packet number and submeter sequence number;
D5, from database, delete submeter according to submeter title, go to step D2.
Fig. 1 shows point table status according to occurring after submeter regular partition, described submeter rule is stored in (such as: be stored in tables of data submeter rule file) in static file usually, longitudinally represent the grouping divided according to business function, M the grouping as diagram root.May have multiple submeter in each grouping, submeter sequence number, from 1, arranges backward.Submeter quantity in each group is not necessarily identical, but determines according to the quantity of real data, uses packet number and the right mode of submeter sequence number can unique identification submeter, and such as, in (2,1) identification packet 2, sequence number is the submeter of 1.
It is below a specific embodiment.
Fig. 4 shows webmaster mass historical data and divides table management method:
A1, read tables of data submeter rule from static file, read from database submeter information table the submeter information that usage flag in_use_flag is 1.
Only to manage usage flag be the submeter of 1 in submeter management, usage flag be 0 submeter used by data batchmove function.
A2, the submeter information read from database submeter information table by packet number traversal, traveled through and gone to step A6, otherwise gone to step A3.
Traversal carries out divide into groups submeter information cache and the initialization of the current submeter sequence number buffer memory of grouping, carries out the submeter management process of timing after completing.
If in A3 A2 packet number be 1 grouping in a submeter all do not have, then go to step A4, if the grouping that packet number is 1 has at least one submeter, then go to step A5;
During first time use webmaster, historical data does not create any submeter, so can enter steps A 4.
A4, to create a new sequence number table_no be 1 submeter, more new database submeter information table, put submeter sequence number be 1 current be just 1 at usage flag current_flag, only have a submeter in each grouping current be just 1 at usage flag current_flag, be identified at the submeter of access when data are inserted.
A5, renewal grouping submeter information cache, increase the submeter information of the submeter created in the submeter or A4 traveled through in A2; Upgrade the current submeter sequence number buffer memory of grouping, this buffer memory is used for recording the submeter sequence number that each grouping is using simultaneously.
The grouping current submeter sequence number buffer memory obtained in A6, regular check A5, if timer expiry, do not receive submeter management to exit signal and then go to step A7, if receive submeter management to exit signal, exit submeter management process, rule of thumb data arrange check_interval between timer expiry is 1 hour.
A7, the current submeter sequence number buffer memory of traversal grouping, if do not traveled through, gone to step A8, if traveled through, gone to step A6;
Check all current be just after the submeter of 1 in usage flag, enter the timer latent period next time checked, the value in this cycle is estimated value, when checking submeter capacity, data are inserted unavailable, and in order to prevent frequent inspection, it is longer that this time cycle preferably sets, and needing between guarantee twice checkpoint, the data line of submeter exceedes the regular submeter expert sum row_count allowed of submeter can not be too many.
A8, according to the group number of current group and submeter sequence number, obtain current at usage flag current_flag be just 1 submeter number of data lines table_row_count, suppose that packet number is 2, submeter sequence number is 3;
If the table_row_count in A9 A8, be greater than submeter expert sum row_count in submeter rule, then go to step A10 otherwise go to step A7;
A10, establishment packet number are 2, submeter sequence number is the new submeter of 4, arrange submeter sequence number be 4 current_flag be 1, in_use_flag is 1, be the submeter information data inserting storehouse submeter information table of 4 by submeter sequence number, revise in submeter information table simultaneously submeter sequence number be 3 current_flag be 0, upgrade grouping submeter information cache, insert the submeter information of submeter sequence number 4, and the current submeter sequence number upgrading in the current submeter sequence number buffer memory of grouping 2 correspondences of dividing into groups is 4;
After creating new submeter, need more new database submeter information table, the submeter of up-to-date increase is exactly necessarily the submeter of current use, also will upgrade grouping submeter information cache and the current submeter sequence number buffer memory of grouping simultaneously, obtain the foundation of submeter when inserting as data.
A11, to judge in current group according to grouping submeter information cache whether submeter sum is greater than in submeter rule the total split_count of submeter number in grouping, if be greater than, goes to step A12, if be less than, goes to step A7;
The submeter information that the submeter sequence number that in A12, more new database submeter information table, this packet number is corresponding is minimum, the usage flag in_use_flag of this submeter is set to 0, represent that this submeter can not use when inquiring about, and need data to shift, then from grouping submeter information cache, delete the record of this submeter, go to step A11.Such as in grouping 2, if split_count is 3, be then that the in_use_flag of the submeter of 1 is set to 0 by submeter sequence number, then in grouping submeter information cache, deleting submeter sequence number in grouping 2 is the information of 1 submeter.
Fig. 5 shows the handling process that historical data is inserted, and comprises the following steps:
B1, group number according to the content determination data traffic packets of data inserting, according to the current submeter information cache of grouping obtain current in usage flag be just 1 submeter sequence number, the submeter corresponding according to data rearranges data inserting order, insert with packet number and submeter sequence number for key, data inserting content is the map of value;
B2, traversal map, traveled through, exited, do not completed and go to step B3.
B3, be that submeter sequence number obtains a point table name according to packet number, data are inserted in submeter.
Due to the existence of the submeter that divides into groups, one batch data may be written in the different submeters of different grouping, the submeter used when writing to prevent continuous switch data, before insertion according to the submeter mode corresponding with data by data permutation, reduce table and switch the time used, so both can ensure the order of data, also can improve the efficiency that data are inserted.
Fig. 6 shows the handling process of the inquiry of historical data, comprises the following steps:
In C1, reading database, in submeter information table, usage flag in_use_flag is the submeter information of 1;
C2, parsed query terms, analyze the grouping needing to comprise, then submeters all in associated packets be set to the list of query aim submeter, such as use (X, Y) to identify unique submeter, wherein an X identification packet number, submeter sequence number in Y identification packet, comprises 1,2 when needing the packet number of inquiry, when 3, then the submeter list of final needs inquiry is: (1,1), (2,2), (2,3), (2,4), (3,1), (3,2), (3,3);
The all submeters of C3, traversal queries, if Query Result has the requirement of priority, then the list in C2 needs to sort by priority, such as: (1,1), (2,4), (2,3), (2,2), (3,3), (3,2), (3,1), then Series poll successively; If Query Result is without the requirement of priority, then the list in C2 is order not, and inquiry can executed in parallel, merges all Query Results, returns.
Fig. 7 shows the handling process of historical data transfer, comprises the following steps:
In D1, reading database submeter information table, all usage flag in_use_flag are the submeter information of 0, being saved in packet number is key, and the list of submeter sequence number is in the map of value, and such as packet number is in the grouping of 2, submeter sequence number is the usage flag of the submeter of 1,2 is 0;
The map obtained in D2, successively traversal D1, the packet number of acquisition and the list of submeter sequence number, then travel through the list of submeter sequence number, if do not complete traversal, then go to step D3, complete traversal backed off after random;
D3, as traversal current submeter be that in grouping 2, submeter sequence number is the submeter of 1, the table name of structure is T_HISDATA_2_1, directly records all in submeter is exported to external file by data base tool or SQL script;
D4, from database submeter information table, delete a point table number be 2, and submeter sequence number is the information of 1;
D5, from database, delete submeter T_HISDATA_2_1, go to step D2.
The content be not described in detail in this description belongs to the known prior art of professional and technical personnel in the field.
Claims (4)
1. a management method for massive network management historical data, it is characterized in that: by secondary submeter rule fulfillment database submeter, the data that the single table preserving webmaster mass historical data is preserved, be saved in the identical submeter of multiple structure after cutting, reduce the capacity of single table, thus improve the treatment effeciency of single table, and following management is carried out to each submeter:
A historical data submeter manages, and B historical data inserts management, and C the inquiry of historical data manages, D historical data transfer management;
The file record of xml form of secondary submeter rule, wherein:
Historical data type data_type, represents that tables of data preserves the type of data,
Total number packets group_count, represents the sum divided into groups to this tables of data,
In grouping, submeter sum split_count, represents submeter number maximum in each grouping,
Submeter expert sum row_count, represents submeter expert sum, when the data volume of a submeter exceedes this value time, just needs the submeter that establishment one is new;
The handling process of described historical data submeter management comprises the following steps:
A1, read and record the file of the secondary submeter rule of tables of data, obtain submeter sum split_count and submeter expert sum row_count in historical data type data_type, total number packets group_count, grouping;
Reading database submeter information table, obtain the packet number group_no of all submeters, submeter sequence number table_no, current just at usage flag current_flag and usage flag in_use_flag, these information being saved in a packet number is key, submeter information list is in the map object of value, described submeter information list comprises submeter sequence number table_no, current just at usage flag current_flag and usage flag in_use_flag, wherein the span of the key of map is from 1 to total number packets group_count, if the submeter information not having certain key corresponding in database submeter information table, the submeter information list that so this key is corresponding in map is just empty,
The map object that A2, traversal A1 obtain;
A3, according to traversal result, if in map object a current group number corresponding submeter information list be sky, then go to step A4, otherwise, go to step A5;
A4, in the grouping of current group number, create an available submeter, distributing submeter sequence number is 1, and arranging submeter current is being just 1 in usage flag, and usage flag is 1, goes to step A5 by after the information data inserting storehouse submeter information table of this submeter;
A5, the submeter information will recorded in database submeter information table, pressing packet number is key, is inserted in grouping submeter information cache one by one, this buffer memory for recording the information of the submeter created in all groupings,
By current in usage flag be just 1 submeter information, pressing packet number is key, by submeter sequence number insert grouping current submeter sequence number buffer memory, in each grouping of this caching record, current_flag is the submeter sequence number of the submeter of 1, goes to step A6;
The submeter divided into groups in current submeter sequence number buffer memory, needs to check whether it exceedes submeter expert sum;
A6, use timer waiting for, if the non-time-out of timer, wait for always, when the timer times out, if receive the signal that application program exits, then exit the handling process of historical data submeter management; Otherwise, proceed to steps A 7;
A7, the current submeter sequence number buffer memory of traversal grouping, if do not traveled through, gone to step A8, if traveled through, gone to step A6;
Submeter sequence number in A8, reading current group, according to the group number of current group and the title of submeter sequence number structure submeter, obtains submeter number of data lines;
If A9 submeter number of data lines is greater than submeter expert sum row_count in submeter rule, then goes to step A10 otherwise go to step A7;
A10, create new submeter, the submeter sequence number of new submeter is that in the grouping of submeter place, original current_flag is that the submeter sequence number value of the submeter of 1 adds 1, then in database submeter information table, insert the information of this new submeter, and the information of this new submeter is inserted in grouping submeter information cache, then submeter sequence number corresponding for the current group in the current submeter sequence number buffer memory of grouping is changed into the submeter sequence number of new submeter;
A11, to judge in current group whether the total number of submeter is greater than submeter sum split_count in grouping, if be greater than, goes to step A12, if be less than, goes to step A7;
In A12, more new database submeter information table, in this grouping, the usage flag in_use_flag of the minimum submeter of submeter sequence number is 0, identify this submeter can not insert in data and use in inquiry, and will by historical data transfer management flow processing, and from grouping submeter information cache, delete the record of this submeter, go to step A11.
2. management method for massive network management historical data as claimed in claim 1, is characterized in that, the handling process that described historical data inserts management comprises the following steps:
B1, group number according to the content determination data traffic packets of data inserting, submeter sequence number is obtained according to the current submeter sequence number buffer memory of grouping, according to submeter and mathematical logic corresponding relation, rearrange data inserting content, stored in packet number and submeter sequence number for key, save as in data inserting in the map of value;
Map in B2, traversal B1, has traveled through, has exited, otherwise gone to step B3;
B3, according to current group number and submeter sequence number, structure submeter title, data inserting, in current submeter, goes to step B2.
3. management method for massive network management historical data as claimed in claim 1, is characterized in that, the handling process of described the inquiry of historical data management comprises the following steps:
In C1, reading database submeter information table, usage flag is the submeter information of 1;
C2, analysis and consult condition, obtain the table name list of all associated packets submeters;
C3, inquire about the submeter in all C2 according to querying condition.
4. management method for massive network management historical data as claimed in claim 1, is characterized in that, the handling process of described historical data transfer management comprises the following steps:
D1, be the submeter information record of 0 by all usage flags in packet number reading database submeter information table, being saved in packet number is key, and the list of submeter sequence number is in the map of value;
The map obtained in D2, traversal D1, has traveled through backed off after random;
D3, according to current group number and submeter sequence number, structure submeter title, uses batch processing instrument directly to derive all line items in the submeter of specifying submeter title;
D4, from database submeter information table, delete submeter information according to packet number and submeter sequence number;
D5, from database, delete submeter according to submeter title, go to step D2.
Priority Applications (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210401421.6A CN102867071B (en) | 2012-10-19 | 2012-10-19 | Management method for massive network management historical data |
| PCT/CN2013/079993 WO2014059808A1 (en) | 2012-10-19 | 2013-07-24 | Method for managing mass historical data of network management |
| MYPI2014703210A MY187641A (en) | 2012-10-19 | 2013-07-24 | Method for managing mass historical data of network management |
| PH12014502368A PH12014502368B1 (en) | 2012-10-19 | 2014-10-22 | Method for managing mass historical data of network management |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210401421.6A CN102867071B (en) | 2012-10-19 | 2012-10-19 | Management method for massive network management historical data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102867071A CN102867071A (en) | 2013-01-09 |
| CN102867071B true CN102867071B (en) | 2015-04-29 |
Family
ID=47445940
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210401421.6A Active CN102867071B (en) | 2012-10-19 | 2012-10-19 | Management method for massive network management historical data |
Country Status (4)
| Country | Link |
|---|---|
| CN (1) | CN102867071B (en) |
| MY (1) | MY187641A (en) |
| PH (1) | PH12014502368B1 (en) |
| WO (1) | WO2014059808A1 (en) |
Families Citing this family (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102867071B (en) * | 2012-10-19 | 2015-04-29 | 烽火通信科技股份有限公司 | Management method for massive network management historical data |
| CN103810238A (en) * | 2013-11-20 | 2014-05-21 | 国家电网公司 | Large-data-volume Excel file exporting method based on asynchronous processing |
| CN103812691A (en) * | 2014-01-14 | 2014-05-21 | 浪潮电子信息产业股份有限公司 | Log statistical system and method for cluster load balancing system |
| CN104360997B (en) * | 2014-04-01 | 2017-11-14 | 芜湖齐创自动化系统有限公司 | A kind of big data bleach-out process based on structured database |
| CN104035974B (en) * | 2014-05-22 | 2017-02-15 | 烽火通信科技股份有限公司 | General Web network management object selector and Web network management selecting method |
| CN104182546B (en) * | 2014-09-09 | 2017-10-27 | 北京国双科技有限公司 | The data query method and device of database |
| CN104715076B (en) * | 2015-04-13 | 2019-03-12 | 东信和平科技股份有限公司 | A kind of data processing of multithread and device |
| CN106294423A (en) * | 2015-05-25 | 2017-01-04 | 阿里巴巴集团控股有限公司 | Data base divides wiring method and the device of table |
| CN104965873A (en) * | 2015-06-10 | 2015-10-07 | 努比亚技术有限公司 | Paging inquiring method and apparatus |
| CN105095393B (en) * | 2015-06-30 | 2018-11-16 | 努比亚技术有限公司 | A kind of date storage method and device |
| CN105930504B (en) * | 2016-05-10 | 2019-04-16 | 烽火通信科技股份有限公司 | A kind of network management timing file cocurrent processing system and concurrent processing method |
| CN106897365B (en) * | 2017-01-11 | 2020-02-14 | 北京市天元网络技术股份有限公司 | Data processing method and device |
| CN106886568B8 (en) * | 2017-01-12 | 2018-05-01 | 掌阅科技股份有限公司 | One kind divides table method, apparatus and electronic equipment |
| CN109144991B (en) * | 2017-06-15 | 2021-09-14 | 北京京东尚科信息技术有限公司 | Method and device for dynamic sub-metering, electronic equipment and computer-storable medium |
| CN107633097B (en) * | 2017-10-16 | 2018-09-25 | 马上消费金融股份有限公司 | A kind of dynamic capacity-expanding method and system of database table |
| CN108108434A (en) * | 2017-12-19 | 2018-06-01 | 福建中金在线信息科技有限公司 | A kind of method and device for managing database |
| CN108304671B (en) * | 2018-02-12 | 2021-07-27 | 厦门海迈科技股份有限公司 | Data management method and related device for building information model |
| CN110222115A (en) * | 2019-04-30 | 2019-09-10 | 武汉达梦数据库有限公司 | The database synchronization method and equipment loaded based on table initialisation packet |
| CN110740198B (en) * | 2019-10-21 | 2022-12-23 | 杭州迪普科技股份有限公司 | Neighbor table item management method and device, electronic equipment and machine-readable storage medium |
| CN110825739B (en) * | 2019-10-30 | 2021-07-16 | 京东数字科技控股有限公司 | Table building statement generation method, device, equipment and storage medium |
| CN115208899B (en) * | 2022-06-29 | 2023-06-06 | 安阳师范学院 | Improved P2P mass scientific data synchronization method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216821A (en) * | 2007-01-05 | 2008-07-09 | 中兴通讯股份有限公司 | Data acquisition system storage management method |
| CN101566986A (en) * | 2008-04-21 | 2009-10-28 | 阿里巴巴集团控股有限公司 | Method and device for processing data in online business processing |
| CN101697152A (en) * | 2009-10-23 | 2010-04-21 | 金蝶软件（中国）有限公司 | Database storage system and method and device for splitting data thereof |
| CN102033882A (en) * | 2009-09-25 | 2011-04-27 | 中兴通讯股份有限公司 | Method and system for storing performance data |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7509342B2 (en) * | 2002-02-08 | 2009-03-24 | Sanyo Electric Co., Ltd. | File data storage management method, file data storage device, program executing processing for storing file data, and storage medium |
| CN100543749C (en) * | 2007-10-18 | 2009-09-23 | 中兴通讯股份有限公司 | The method that the data of a plurality of data sources are unified to sort |
| CN102542054B (en) * | 2011-12-29 | 2017-02-08 | 厦门雅迅网络股份有限公司 | Method utilizing cache tables to improve insertion performance of data in database |
| CN102867071B (en) * | 2012-10-19 | 2015-04-29 | 烽火通信科技股份有限公司 | Management method for massive network management historical data |
-
2012
- 2012-10-19 CN CN201210401421.6A patent/CN102867071B/en active Active
-
2013
- 2013-07-24 WO PCT/CN2013/079993 patent/WO2014059808A1/en active Application Filing
- 2013-07-24 MY MYPI2014703210A patent/MY187641A/en unknown
-
2014
- 2014-10-22 PH PH12014502368A patent/PH12014502368B1/en unknown
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216821A (en) * | 2007-01-05 | 2008-07-09 | 中兴通讯股份有限公司 | Data acquisition system storage management method |
| CN101566986A (en) * | 2008-04-21 | 2009-10-28 | 阿里巴巴集团控股有限公司 | Method and device for processing data in online business processing |
| CN102033882A (en) * | 2009-09-25 | 2011-04-27 | 中兴通讯股份有限公司 | Method and system for storing performance data |
| CN101697152A (en) * | 2009-10-23 | 2010-04-21 | 金蝶软件（中国）有限公司 | Database storage system and method and device for splitting data thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| MY187641A (en) | 2021-10-06 |
| CN102867071A (en) | 2013-01-09 |
| WO2014059808A1 (en) | 2014-04-24 |
| PH12014502368A1 (en) | 2015-01-12 |
| PH12014502368B1 (en) | 2015-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102867071B (en) | Management method for massive network management historical data | |
| CN101553813B (en) | Managing storage of individually accessible data units | |
| US5237681A (en) | Relational data base memory utilization analyzer | |
| CN102542071B (en) | Distributed data processing system and method | |
| CN102129458B (en) | Method and device for storing relational database | |
| WO1992021090A9 (en) | Relational data base memory utilization analyzer | |
| CN107992492B (en) | Data block storage method, data block reading method, data block storage device, data block reading device and block chain | |
| CN104462141A (en) | Data storage and query method and system and storage engine device | |
| CN103037344B (en) | A kind of ticket De-weight method and device | |
| CN107741947B (en) | Method for storing and acquiring random number key based on HDFS file system | |
| CN101763415B (en) | Method and device for generating B tree index of database | |
| CN106503008B (en) | File storage method and device and file query method and device | |
| CN106682077A (en) | Method for storing massive time series data on basis of Hadoop technologies | |
| US20110238677A1 (en) | Dynamic Sort-Based Parallelism | |
| CN106649602B (en) | Business object data processing method, device and server | |
| CN102893265A (en) | Managing storage of individually accessible data units | |
| CN102779138A (en) | Hard disk access method of real time data | |
| CN101093482A (en) | Method for storing and retrieving mass information | |
| CN110096509A (en) | Realize that historical data draws the system and method for storage of linked list modeling processing under big data environment | |
| CN102411632B (en) | Chain table-based memory database page type storage method | |
| CN113177090A (en) | Data processing method and device | |
| CN103246549A (en) | Method and system for data transfer | |
| JPH11154155A (en) | File managing method | |
| CN100399338C (en) | A sorting method of data record | |
| CN107818155A (en) | A kind of storage method of distribution main website and distribution main website data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20161209 Address after: 430000 Hubei Province, Wuhan East Lake Development Zone No. 4 road of Guanshan two Patentee after: WUHAN FIBERHOME TECHNICAL SERVICES CO., LTD. Address before: China Science and Technology Park Dongxin road East Lake Development Zone 430074 Hubei Province, Wuhan City, No. 5 Patentee before: Fenghuo Communication Science &. Technology Co., Ltd. |