WO2014059808A1 - Method for managing mass historical data of network management - Google Patents

Method for managing mass historical data of network management Download PDF

Info

Publication number
WO2014059808A1
WO2014059808A1 PCT/CN2013/079993 CN2013079993W WO2014059808A1 WO 2014059808 A1 WO2014059808 A1 WO 2014059808A1 CN 2013079993 W CN2013079993 W CN 2013079993W WO 2014059808 A1 WO2014059808 A1 WO 2014059808A1
Authority
WO
WIPO (PCT)
Prior art keywords
table
sub
number
group
data
Prior art date
Application number
PCT/CN2013/079993
Other languages
French (fr)
Chinese (zh)
Inventor
邹昊
Original Assignee
烽火通信科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to CN201210401421.6 priority Critical
Priority to CN201210401421.6A priority patent/CN102867071B/en
Application filed by 烽火通信科技股份有限公司 filed Critical 烽火通信科技股份有限公司
Publication of WO2014059808A1 publication Critical patent/WO2014059808A1/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/02Arrangements for maintenance or administration or management of packet switching networks involving integration or standardization
    • H04L41/024Arrangements for maintenance or administration or management of packet switching networks involving integration or standardization using relational databases for representation of network management data, e.g. managing via structured query language [SQL]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/08Configuration management of network or network elements
    • H04L41/085Keeping track of network configuration

Abstract

The present invention relates to management of historical data in a management system of network management, and particularly to a method for managing mass historical data of network management. A database is divided into tables via a second-level table division rule, the data stored in a single table in which mass historical data of network management is stored is segmented and then stored in a plurality of divided tables with the same structure so that the capacity of the single table is reduced, thereby improving the processing efficiency of the single table, and each divided table is managed as follows: A management of historical data in divided tables; B insertion management of historical data; C query management of historical data; and D transfer management of historical data. The method for managing mass historical data of network management described in the present invention can save mass historical data, and can efficiently perform data insertion, query and transfer without affecting the normal use of network management software, thereby improving the working efficiency of a management system of network management.

Description

 Mass management historical data management method for network management

Technical field

 The invention relates to the management of historical data in the network management system, in particular to a method for managing massive historical data of the network management. Background technique

 At present, the network technology is changing with each passing day. The scale of broadband users is constantly expanding. The number of devices managed by the telecom network management system is also increasing. The logs of daily management of the telecom network management system, alarm data of devices and other data need to be stored in the database. Shake is used to record data and query analysis data. After the R-month, these data are getting bigger and bigger. If the data is not effectively managed, whether it is data insertion or data query, the performance will drop sharply, which will affect the normal use of the network management software. The database uses hard disk space to store the W data. If the historical data grows indefinitely, it will eventually exhaust all the disk space. Therefore, the historical data needs to be cleaned regularly, and some of the infrequently used data in the database are dumped to other locations. Save, archive. In the process of data transfer, the tube cannot use the data. If the efficiency of the dump is low, it will affect the availability of the network management system. An efficient data transfer scheme must be provided. Hairline

 In view of the deficiencies in the prior art, the object of the present invention is to provide a method for managing massive historical data of a network management system, which can save a large amount of historical data, and can efficiently insert, query, and transfer data without affecting the normal use of the network management software. .

 In order to achieve the above object, the technical solution adopted by the present invention is:

A method for managing a large amount of historical data of a W-tube, characterized in that: the database is divided into two tables by the second slow-sorting rule, and the data stored in a single table storing the massive historical data of the network management is segmented and saved to a plurality of identical structures. In the table, reduce the capacity of a single table, thereby The processing efficiency of high single table, and the following management of each sub-table:

 A historical data table management,

 B historical data insertion management,

 C historical data query management,

 D Historical data transfer management.

 Based on the above technical solutions, the secondary table rules are recorded in the x l format file, where:

 The historical data type data_type indicates the type of data stored in the data table. The total number of groups is group_count, which indicates the total number of groups of such data tables. The total number of sub-tables in the group sp lit-count indicates that the number of sub-tables per packet is the largest. Value, the total number of rows in the table, row_count, indicates the total number of table rows. When the data volume of a table exceeds this value, you need to create a new table.

 Based on the above technical solution, the processing flow of the historical data table management includes the following steps:

 Al, read the file that records the secondary table rules of the data table, obtain the historical data type data-ty e, the total number of groups group-count, the total number of sub-tables in the group sp l it-count, the total number of rows in the table — count ;

 Read the data table of the sub-table table, obtain all the sub-table group number group-110, the sub-table number table jo, the currently using the mark current_flag and the used tag 3⁄4 in-usej!ag, save the information to a The group number is a key, and the part table information list is a value of the ap object, the part table information list includes the part table number tab le no, the current label is currently used and the flag is used in the in_use_flag, where the map key is The value ranges from 1 to the total number of groups - count. If there is no sub-table information corresponding to a key in the database table information table, the list of sub-table information corresponding to the key in ap is empty;

 A2, traversing the map object obtained by A1;

 A3, according to the result of the traversal, if the sub-table information list corresponding to the current group number in the ap object is empty, then go to step A4, otherwise, go to step A5;

A4. Create a usable sub-table in the grouping of the current group number, and assign the sub-sequence number. Set the sub-table to be marked as 1 . The used flag is 1, insert the information of the sub-table into the database sub-table information table, and then go to step A5 ;

 A5. The sub-table information recorded in the database sub-table information table is inserted into the packet table information cache by the group number as a key, and the cache is used to record the information of the sub-tables created in all the groups.

The sub-table information marked with 1 is currently being used, the group number is the key, and the sub-sequence will be used.

Figure imgf000005_0001

For the sub-table number of the sub-table of 1, turn to step A6 ;

 To group the sub-tables in the current sub-table number buffer, you need to check whether it exceeds the total number of rows in the sub-table;

A6, using the timer and waiting, if the timer has not timed out, it waits for a while, when the timing expires, if the signal of the application exit is received, the processing flow of the historical data sub-table management is exited; otherwise, the process proceeds to step A7 ;

 A7, traversing the current sub-table serial number cache of the packet, if not traversed, then go to step A8, if the traversal is completed, then step A6;

 A8, reading the sub-table number in the current group, constructing the name of the sub-table according to the group number and the sub-table number of the current group, and obtaining the number of sub-table data lines;

 A9, if the number of sub-tables «the number of rows is greater than the total number of rows in the sub-table rules row_count, then go to step A10, otherwise go to step A7;

 A10. Create a new sub-table. The sub-table number of the new sub-table is the sub-table number value of the sub-table with the original current-flag of 1 in the group where the sub-table is located, and then add 1 to the database sub-table information table Φ. The information of the table, and the information of the new sub-table is inserted into the packet information cache, and then the sub-table number corresponding to the current group in the current sub-table number buffer of the packet is changed to the sub-table number of the new sub-table;

 A1 determines whether the total number of sub-tables in the current group is greater than the total number of sub-tables in the group. split_count» If it is greater than then, transfer to A12, if it is less, then go to step A7;

A12. In the update database table information table, the used tag injsejla of the sub-table serial number minimum sub-table in the group is 0, and the identifier is not used in data insertion and query, and will be processed by the historical data transfer management process. And delete from the packet table information cache In addition to the record of the sub-table, go to step A ll.

 Based on the above technical solution, the processing flow of the historical data insertion management includes the following steps:

 The BU determines the group number of the data service group according to the content of the inserted data, and obtains the sub-table number according to the current sub-table number of the packet, and the root "correlation table and the number of logical correspondences, rearrange the inserted data content, and store the group number and points. The table number is a key, and the data memory is inserted into the map of values;

 B2. Appropriate map in B1, exit when it is completed, otherwise go to step K3;

 B3. According to the current grouping number and the sub-table number, construct the sub-table name, and the insertion number is held in the current sub-table, and the process proceeds to step B2.

 Based on the above technical solution, the processing flow of the historical number query management includes the following steps:

 The CU reads the sub-table information marked with 1 in the database sub-table information table; C2, analyzes the query conditions, and obtains a list of table names of all relevant sub-component tables; 03, the club queries all the sub-tables in C2 according to the query conditions.

 Based on the above technical solution, the processing flow of the historical data transfer management includes the following steps:

 D Read all the table information records in the database table information table that have been marked with 0 by the group number, and save them to the map with the group number as the key and the table number list as the value;

 D2, traversing the map obtained in D1, traversing is completed, and then exiting;

 D3. Construct a sub-table name according to the current group number and the sub-table number, and directly use the batch tool to directly export all the row records in the sub-table of the specified sub-table name;

 D4. The part table information is deleted from the database part table information table according to the group number and the part table number;

D5, delete the sub-table from the database according to the sub-table name, and go to step D2. The method for managing massive data historical data of the network management system according to the present invention implements database sub-tables through two-level sub-table rules (database sub-tables: data stored in a single table, and divided according to certain rules and saved to a plurality of tables with the same structure ( In the case of sub-tables, reduce the capacity of a single table, Therefore, the processing efficiency of the single table can be improved, different sub-tables can be accessed under different service requirements, the lock competition can be reduced, and the operation efficiency can be improved. The historical data is dispersed into multiple sub-tables, and the number of records in each sub-table is guaranteed. Compared with the limited sub-table capacity, it not only ensures the response efficiency of historical data query, but also ensures the data insertion efficiency, improves the customer experience, and improves the efficiency of the network management system.

 The method for managing the massive historical data of the network management according to the present invention separates the data table that is being used and needs to be transferred by using the mark, and the network management system and the data transfer function do not affect each other, and the efficiency of the whole table processing of the data table It is much higher than the efficiency of processing according to row records, which reduces the impact on the database system and improves the overall efficiency of the network management system.

BRIEF DESCRIPTION OF THE DRAWINGS

 The invention has the following figures:

 Circle 1 is divided according to the rules of the sub-table, and the sub-table is displayed;

 Figure 2 is the content of the data table sub-rule rule file;

 Figure 3 is the contents of the database containing the information contained in the table information table;

 Figure 4 is a sub-table management processing flow circle;

 圏 5 is a flow chart of data insertion processing;

 Figure 6 is the data query processing flow ffl;

 Circle 7 is the data transfer processing flow circle.

钵 钵 钵

 The invention will be further described in detail below with reference to the accompanying drawings.

 The method for managing the massive historical data of the network management system according to the present invention realizes the database sub-table by the second-level sub-rule rule, and divides the data saved in the single table storing the massive historical data of the network management, and then saves the data to a plurality of sub-tables with the same structure. In the process, the capacity of the single table is reduced, thereby improving the processing efficiency of the single table, and the following management is performed for each of the sub-tables:

 A historical data table management,

B history number is inserted into management, c historical data query management,

 D Historical data transfer management. On the basis of the above technical solution, FIG. 2 shows the content of the data table sub-rule rule file, that is, the second-level sub-table rule, the second-encryption sub-table rule is recorded in the xml format file, where - the historical data type Kta^type, which indicates the type of data stored in the data table, the total number of groups, group_count, indicates the total number of groups of such data tables, that is, a data table can be divided into several groups, and each group corresponds to a unique group number;

 The total number of sub-tables in the group, split_count, indicates the maximum number of sub-tables in each group, that is, how many sub-tables can be included in a group, that is, the maximum number of sub-tables in the group, and each sub-table corresponds to a unique one. Table number

 The total number of rows in the sub-table is row_count, which indicates the total number of rows in the sub-table, that is, the total number of rows in the sub-rows row_count indicates how many rows can be included in a sub-table, that is, the maximum number of rows in the sub-table; when the data volume of a sub-list exceeds When this value is used, you need to create a new sub-table.

 That is: a data table can have at most group-count groupings, one group has split-count sub-tables, and one sub-table records up to row c oun t rows. On the basis of the foregoing technical solutions, as shown in FIG. 4, the processing flow of the historical data table management includes the following steps:

 A Read the file that records the secondary table rules of the dig-in table, obtain the history number data type data_type, the total number of groups group—couir The total number of sub-tables in the group sp 1 it— count and the total number of rows in the sub-row row~ count

Read the database sub-table information table, obtain the group number group_ no of all sub-tables, the sub-table number table- no, currently using the mark current_flag and 巳 use the mark in^use^fiag, save the information to a group number For the key, the table information list is a value of the map object, the table information list includes the table number table - n0 , current The flag current_flag and 巳 is used to make ffl mark in_ use_flag, the value of the Φ map key ranges from 1 to the total number of groups group_count, if there is no corresponding table in the database table information table Information, then in the map Ψ the list of sub-table information corresponding to the key is empty;

 The format and content of the database table information table can be seen in the embodiment shown in FIG. 3; FIG. 3 shows the contents of the number table information table, and demonstrates a possible state, each line records a sub-table information. Assume that the sub-table rules refer to the group group count = 3, and each group allows up to split_count = 3 sub-tables, then when the fourth sub-table is created, the sub-table number table — The sub-table of no 4 becomes the table in the current state of use current—fla is 1, and the used flag injisej ag of the sub-table number table jo is set to 0, so that all the available values can be kept. The total number of sub-tables does not exceed the value of split-co wake up t;

 A2, traversing the map object obtained by A1;

 A3, according to the result of the traversal, if the list of sub-table information corresponding to the current group number in the map object is empty, then go to step M, otherwise, go to step A5;

 A4. In the group Φ of the current group number, create an available sub-table, assign the sub-sequence number to 1, set the sub-table currently using the tag as 1, and use the tag as 1, insert the information of the sub-table into the database sub-table information. After the table, go to 歩骠A5;

 Λ 5. The sub-table information recorded in the database sub-table information table is inserted into the sub-component table information cache by the group number as a key, and the cache is used to record the information of the sub-tables created in all the groups.

The sub-table information marked with 1 is currently being used, and the sub-table number is inserted into the current sub-table number buffer according to the group number key. The cache records the sub-table number of the sub-table with cmrrent_f lag of 1 in each group. , go to step A6 ;

 To group the sub-tables in the current sub-table number buffer, you need to check whether it exceeds the total number of rows in the sub-table;

A6, using the timer and waiting, if the timer does not timeout, then waits, when the timer expires, if the signal of the application exit is received, the processing flow of the historical data sub-table management is exited; otherwise, the process proceeds to step A7; A7, traversing the current sub-table number sequence buffer of the packet, if not traversed, then go to step ,8, if it is suitable, then go to step A6;

 A8, reading the sub-table number in the current group, constructing the name of the sub-table according to the group number and the sub-table number of the current group, and obtaining the number of sub-table data lines;

 Λ 9, if the number of sub-table data rows is greater than the total number of sub-tables in the sub-table rules ro - count, then the process is changed to AM, otherwise it is transferred to step A7;

 A 10 , create a new sub-table, the sub-table number of the new sub-table is the sub-table number value of the sub-table whose original current^flag is 1 in the group where the sub-table is located, and then add 1 in the database sub-table information table. Dividing the information of the table, and inserting the information of the new sub-table into the information table of the sub-group table, and then changing the serial number of the current sub-table in the current sub-table number buffer of the packet to the sub-table number of the new sub-table;

 Al 1, determining whether the total number of sub-tables in the current group is greater than the total number of sub-tables in the group split-count, if it is greater, then go to step A12, if it is less, then go to step A7;

 A12. In the update database sub-table information table, the sub-table number of the sub-segment in the sub-segment is marked with the use flag in_ use_flag, indicating that the sub-table will not be used in data insertion and query, and will be historical number « The transfer management process processes, and deletes the record of the part table from the packet table information cache, and proceeds to step Al l. On the basis of the above technical solution, as shown in FIG. 5, the processing flow package of the historical data insertion management has the following steps:

 Bl, determining the group number of the data service packet according to the content of the inserted data, obtaining the sub-table number according to the current sub-table number of the packet, the root vibration table and the data logical correspondence, rearranging the inserted data content, and depositing the packet by the group number The serial number is the key, and the data memory is inserted into the map of the value;

 132, traverse the map in B1, exit when the traversal is completed, otherwise go to the step

B3. According to the current group number and the sub-table number, construct the sub-table name, insert the data into the current sub-table, and go to step B2. Based on the above technical solution, as shown in FIG. 6, the historical data query management The processing flow includes the following steps:

 The sub-table information marked with 1 has been used in the CU to read the database sub-table information table;

 C2, sub-query conditions, get a list of table names of all relevant sub-component tables;

 C3. The club queries all the sub-tables in C2 according to the query conditions. Based on the foregoing technical solution, as shown in FIG. 7, the processing flow of the historical data transfer management includes the following steps:

 D1. All the table information records in the database table information table that have been marked with 0 are read by the group number, and are saved to the ap with the group number as the key and the table number list as the value;

 D2, traversing the map obtained in D1, and exiting after the traversal is completed;

 D3. Construct a sub-table name according to the current group number and the sub-table number, and directly use the batch tool to directly export all the rows in the sub-table of the specified sub-table name;

 D4. Delete the sub-table information from the database sub-table information table according to the group number and the sub-table number;

 D5. Remove the sub-table from the database according to the name of the sub-table, and transfer to D2. Figure 1 shows the state of the partial table that may appear after being divided according to the rules of the table. The table rules are usually stored in a static file (for example: stored in the data table table rule file), and the vertical representation is divided according to the business function. A grouping, such as a circle, divides M groups. There may be multiple sub-tables in each group, and the sub-table numbers start from 1 and are arranged backwards. The number of sub-tables in each group is not necessarily the same, but is determined according to the quantity of actual data. The method τ of the group number and the sub-sequence number pair is used to uniquely identify a sub-table, for example, (2, 1) identifies the group 2 φ sequence number. A sub-table of 1. The following is a specific embodiment.

 Figure 4 shows a method for managing the massive historical data of the network management

 Al. Read the data table sub-table rules from the static file, and read the sub-table information using the tag in se-flag to 1 from the database table information table.

Sub-table management only manages the use of sub-tables marked with 1 and has used the score marked 0 The table is made up of data transfer functions.

 A2, traverse the sub-table information read from the database sub-table information table according to the group number, and complete the conversion A6, otherwise go to step A3.

 The traversal performs the initialization of the component table information cache and the packet current table number sequence buffer, and performs the scheduled table management step after completion.

A3. If there is no sub-table in the packet with the packet number 1 in A2, the process proceeds to step A4. If the packet with the packet number 1 has at least one sub-table, the process proceeds to step A5 .

 When the network management system is used for the first time, the historical data does not create any sub-tables, so it will enter 歩骡 A4.

 A4. Create a new serial number tab le no for the 1 minute table, update the database sub-table information table, and set the sub-table number to 1. The currently used tag is currentj!ag is L. Only one sub-table in each group is currently in use. The flag ciirrent_flag is 1, which identifies the sub-table accessed during data insertion.

 A5, updating the sub-group table information cache, increasing the sub-table information traversed in A2 or the sub-table information created in A4; and updating the current sub-table number sequence buffer of the group, the cache is used to record the sub-tables that each group is using. Serial number.

 Αβ, the current sub-table number buffer of the packet obtained in the timed Α5, if the timer is too large, the sub-table management exit signal is not received, then the process proceeds to Α7, and if the sub-table management exit signal is received, the sub-table management flow is exited. According to the empirical data, set the timer over time check-interval to 1 hour.

 A7, the current subgroup table serial number cache, if not traversed, then go to step A8, if the traversal is completed, go to step A6;

 Check that all the sub-tables marked with 1 are currently being used. After entering the next check-up timer waiting period, the value of this period is the estimated value. When checking the sub-table capacity, the data insertion is not available' To prevent frequent inspections, This time period is preferably set to be longer, and it is necessary to ensure that the data rows of the sub-tables between the two checkpoints exceed the total number of sub-tables allowed by the sub-table rules so that count is not too much.

A8. According to the group number and the sub-sequence number of the current group, obtain the number of table data rows that are currently using the flag current_flag is 1, table_row_count, and assume that the group number is % sub-table number is 3;

 A9. If the tab le-row_count in A8 is greater than the total number of row-counts in the sub-table rule φ sub-table, then go to step A 10, otherwise turn to step A7;

 A10, create a group number of 2, a sub-table number of 4 new sub-table, set the current-flag of the sub-table number 4 to 1, injse jlag to 1, and insert the sub-table information with the sub-table number 4 into the database sub-table information. In the table, the current_flag of the sub-table number 3 is modified to 0 in the sub-table information table, the sub-table information cache is updated, the sub-table information of the sub-table number 4 is inserted, and the packet in the current sub-table number buffer is updated. The corresponding current sub-table number is 4;

 After creating a new sub-table, you need to update the database sub-table information table. The newly added sub-table must be the currently used sub-table. At the same time, you need to update the sub-component table information cache and group the current sub-table number buffer to obtain the points when the data is inserted. The classification of the table.

 Al l, according to the packet table information cache to determine whether the total number of sub-tables in the current group is greater than the total number of sub-tables in the sub-table rule split-count, if it is greater, then go to step A12, if it is less, then step A7;

 A12. Update the sub-table information of the sub-table number corresponding to the group number in the database sub-table information table, and set the indentation flag of the sub-table to incise_flag to 0, indicating that the sub-table is not used during the query. Moreover, the data needs to be transferred, and then the record of the part table is deleted from the packet table information cache, and the process goes to step A11. For example, in group 2, if spl it — TOunt is 3, the iiuse_flag of the part table whose sub-table number is 1 is set to 0, and then the sub-table number is 1 in the packet table information cache. Table information. Figure 5 shows the processing flow of historical data insertion, including the following steps:

Bl, determining the group number of the data service packet according to the content of the inserted data, obtaining the sub-table number currently marked with 1 according to the current sub-table information cache of the packet, rearranging the insertion digital sequence according to the sub-table corresponding to the data, and inserting into the grouping The number of the sub-table is the key, and the data content of the inserted data is the value of the dish ap;

 B2, traversing map, exiting when traversing is completed, and not completing step Β3.

Β3, the root group number is the sub-table number to obtain the sub-table name, and the number is inserted into the sub-table. Due to the existence of the component table, a batch of data may be written to different sub-tables of different groups. In order to prevent ih from switching the data used in the data writing, the data is corresponding to the data according to the sub-table and the data before the insertion. Rearrange, reduce the time used for table switching, so as to ensure the order of the data, but also to improve the efficiency of data insertion. Figure 6 shows the processing flow of historical data query, including the following 歩« :

 C1. Reading the database Φ The sub-table information table uses the sub-table information with the tag in-use_flag 1;

 C2, parsing the query conditions, analyzing the groupings that need to be included, and then setting all the sub-tables in the relevant group to the list of the target sub-tables, and using (X, Y) to identify a unique sub-table, where X The identification group number, Y identifies the sub-table number in the group. When the group number to be queried includes 1, 2, 3, the list of sub-tables that need to be queried is - (1, 1), (2, 2), (2, 3), (2, 4), (3, 1), (3, 2), (3, 3);

 C3. Traverse all the sub-tables. If the query results have priority requirements, the list in C2 needs to be prioritized, such as: (1, 1), (2, 4), (2, 3) , (2, 2), (3, 3), (3, 2), (3, 1), and then serially check who; if the query results have no priority requirements, then the list in C2 has no order, check The query can be executed in parallel, merge all the query results, and return. 圏7 shows the processing flow of historical data transfer, including the following steps:

 Dl, read all the table information in the database table information table that has been marked with in_use_flag 0, and save it to the map with the group number as the key and the table number list as the value, for example, the group number is 2. Group φ, the sub-table number of the sub-table is 1, 2, the used mark of the sub-table is 0;

 02, in turn, the imp obtained in D1, the obtained group number and the list number list, and then the table number list, if not completed, then go to step U3. Complete the traversal and exit;

03. If the current sub-table is a sub-table of the sub-table number 1 in the group 2, the constructed table name is TJIISDATAJ-1, and all the records in the sub-table are directly exported to the external file through the database tool or the SQL script: D4. Delete the information that the sub-table number is 2 and the sub-sequence number is 1 from the database sub-table information table Φ;

 D5. Delete the sub-table T- HISDATA— 2—1 from the database, and go to step D2. Contents not described in detail in this specification are prior art known to those skilled in the art.

Claims

Claim
 A method for managing massive historical data of a network management system, characterized in that: the database is divided into two parts by a secondary table, and the data stored in a single table storing the massive historical data of the network management is segmented and saved to the same plurality of nodes. In the sub-table, the capacity of the single table is reduced, thereby improving the processing efficiency of the single table, and the following management is performed for each sub-table:
 A historical data table management, B historical data insertion management, C historical data query management, D historical data transfer management.
 2. The network management massive historical data management method according to claim 1, wherein the second-level sub-table rule uses the file format of the "1 format, wherein:
 The historical data type data_type indicates the type of data stored in the data table. The total number of groups is group_count, which indicates the total number of groupings of such digital cameras. The total number of sub-tables in the group sp lit-count indicates the number of sub-tables in each group. The maximum value, the total number of rows in the sub-table, ro^ count, indicates the total number of sub-tables. When the data volume of a sub-table exceeds this value, a new sub-table needs to be created.
 3. The method for managing massive data management data of a network management system according to claim 2, wherein the processing flow of the historical data distribution table management comprises the following steps:
 Al, read the file that records the secondary table rules of the data table, obtain the historical data type data_type, the total number of groups group_count, the total number of sub-tables in the group sp l it-count, and the total number of rows F3⁄4 row-count ;
Read the database table information table, get the group number group no of all the sub-tables, the table number table JM is currently using the tag current_flag and the used tag in^use^flag, save the information to a group number as the key The sub-table information list is a value of the ma object, and the list of the sub-table information includes the sub-table number table- n0 , the current mark-current_flag is currently used, and the ffi flag is in-use-flag, where the map key is taken The value ranges from 1 to the total number of groups grwip^ count. If there is no sub-table information corresponding to a key in the database table information table, the list of sub-table information corresponding to the key in ap is empty;
A2, traversing the map object obtained by A1; A3, according to the results of the appropriate, if the list of sub-table information corresponding to the current group number in the map object is empty, then go to step M, otherwise, go to step A5;
A4. Create a usable sub-table in the grouping of the current group number, and assign the sub-sequence number to 1. Set the sub-table currently using the tag as 1, the used tag is 1, and insert the information of the sub-table into the database sub-table information. After the table, go to step A5 ;
 A5. The sub-table information contained in the database sub-table information table is inserted into the sub-component table information cache by the group number as a key, and the cache is used to record the information of the sub-tables created in all the groups.
The sub-table information marked with 1 is currently being used, the group number is the key, and the sub-table number is inserted into the current sub-table number buffer of the packet, and the cache records the sub-table number of the sub-table with the curretrt_f lag of 1 in each group. , go to step A6 ;
 To group the sub-tables in the current sub-table number buffer, you need to check whether it exceeds the total number of rows in the sub-table;
A6, using the timer and waiting, if the timer has not timed out, it waits for a while, when the timer expires, if the signal of the application exit is received, the processing flow of the historical data distribution table management is exited; otherwise, the process proceeds to step A7 ;
A7, traversing the current sub-table serial number cache of the packet, if not traversed, then go to step A8, if the traversal is completed, then step A6 ;
 Λ8, reading the sub-table number in the current group, constructing the name of the sub-table according to the group number and the sub-table number of the current group, and obtaining the number of sub-table data lines;
 Λ 9, if the number of sub-table data rows is greater than the total number of rows in the sub-table rules row count, then go to step Α 10, otherwise transfer 歩 7;
 Α10. Create a new sub-table. The sub-table number of the new sub-table is the sub-table number of the sub-table Φ. The current table with the current-flag of 1 is incremented by 1 and then inserted in the database sub-table information table. The information of the table, and inserting the information of the new sub-table into the packet information cache, and then changing the sub-table number corresponding to the current group of the current sub-table number buffer Φ of the packet to the sub-table number of the new sub-table;
A1 K determines whether the total number of sub-tables in the current group is greater than the total number of sub-tables in the group split-coiurt, if it is greater, then step A12, if not, then go to step A7; A12. In the update database table information table, the used mark in^use^fla of the minimum part table of the sub-table in the group is 0, and the sub-table is not used in data insertion and query, and will be used by history. The data transfer management process processes, and deletes the records of the sub-table from the sub-component table information cache, and proceeds to step A11.
 4. The network management massive historical data management method according to claim 3, wherein the processing flow of the historical data insertion management comprises the following steps:
 B. According to the content of the inserted data, determine the group number of the data grouping, obtain the sub-sequence number according to the current sub-table number of the grouping, and rearrange the content of the sub-division according to the correspondence between the sub-table and the data, and store the content by the group number and The sub-table number is the key, and the inserted data is saved as a map of values;
 B2, traverse the map in B1, exit when the traversal is completed, otherwise go to step B3;
 B3. According to the current group number and the sub-table number, construct the sub-table name, insert the number into the current sub-table, and go to step B2.
 The method for managing the massive historical data of the network management system according to claim 3, wherein the processing flow of the historical data query management includes the following steps - ci, reading the database sub-table information table, using the marked 1 sub-table information;
C2, analyzing the query conditions, and obtaining a list of table names of all relevant sub-component tables;
C3, the rod digging query conditions query all the sub-tables in C2.
 The method for managing the massive data management of the network management system according to claim 3, wherein the processing flow of the historical data transfer management comprises the following steps:
 Dl, reading all the table information records in the database table information table that have been marked with 0 by the group number, and saving to the map with the group number as the key and the table number list as the value;
 02, traversing the obtained ap, exit after the traversal is completed;
 D3. Construct a sub-table name according to the current grouping number and the sub-sequence number, and directly export all the row records in the sub-table of the specified sub-table name by using a batch processing tool.
 M. Delete the sub-table information from the database sub-table information table according to the group number and the sub-table number;
 D5. Delete the sub-table from the database according to the sub-table name, and go to step D2.
PCT/CN2013/079993 2012-10-19 2013-07-24 Method for managing mass historical data of network management WO2014059808A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201210401421.6 2012-10-19
CN201210401421.6A CN102867071B (en) 2012-10-19 2012-10-19 Management method for massive network management historical data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PH12014502368A PH12014502368A1 (en) 2012-10-19 2014-10-22 Method for managing mass historical data of network management

Publications (1)

Publication Number Publication Date
WO2014059808A1 true WO2014059808A1 (en) 2014-04-24

Family

ID=47445940

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2013/079993 WO2014059808A1 (en) 2012-10-19 2013-07-24 Method for managing mass historical data of network management

Country Status (3)

Country Link
CN (1) CN102867071B (en)
PH (1) PH12014502368A1 (en)
WO (1) WO2014059808A1 (en)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102867071B (en) * 2012-10-19 2015-04-29 烽火通信科技股份有限公司 Management method for massive network management historical data
CN103810238A (en) * 2013-11-20 2014-05-21 国家电网公司 Large-data-volume Excel file exporting method based on asynchronous processing
CN103812691A (en) * 2014-01-14 2014-05-21 浪潮电子信息产业股份有限公司 Log statistical system and method for cluster load balancing system
CN104360997B (en) * 2014-04-01 2017-11-14 芜湖齐创自动化系统有限公司 A kind of big data bleach-out process based on structured database
CN104035974B (en) * 2014-05-22 2017-02-15 烽火通信科技股份有限公司 General Web network management object selector and Web network management selecting method
CN104182546B (en) * 2014-09-09 2017-10-27 北京国双科技有限公司 The data query method and device of database
CN104715076B (en) * 2015-04-13 2019-03-12 东信和平科技股份有限公司 A kind of data processing of multithread and device
CN106294423A (en) * 2015-05-25 2017-01-04 阿里巴巴集团控股有限公司 Data base divides wiring method and the device of table
CN104965873A (en) * 2015-06-10 2015-10-07 努比亚技术有限公司 Paging inquiring method and apparatus
CN105095393B (en) * 2015-06-30 2018-11-16 努比亚技术有限公司 A kind of date storage method and device
CN105930504B (en) * 2016-05-10 2019-04-16 烽火通信科技股份有限公司 A kind of network management timing file cocurrent processing system and concurrent processing method
CN106897365A (en) * 2017-01-11 2017-06-27 北京市天元网络技术股份有限公司 Data processing method and device
CN106886568B8 (en) * 2017-01-12 2018-05-01 掌阅科技股份有限公司 One kind divides table method, apparatus and electronic equipment
CN107633097B (en) * 2017-10-16 2018-09-25 马上消费金融股份有限公司 A kind of dynamic capacity-expanding method and system of database table
CN110222115A (en) * 2019-04-30 2019-09-10 武汉达梦数据库有限公司 The database synchronization method and equipment loaded based on table initialisation packet

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030154206A1 (en) * 2002-02-08 2003-08-14 Sanyo Electric Co., Ltd. File data storage management method, file data storage device, program executing processing for storing file data, and storage medium
CN101149744A (en) * 2007-10-18 2008-03-26 中兴通讯股份有限公司 Method for executing uniform ordering for multiple data source
CN101566986A (en) * 2008-04-21 2009-10-28 阿里巴巴集团控股有限公司 Method and device for processing data in online business processing
CN102542054A (en) * 2011-12-29 2012-07-04 厦门雅迅网络股份有限公司 Method utilizing cache tables to improve insertion performance of data in database
CN102867071A (en) * 2012-10-19 2013-01-09 烽火通信科技股份有限公司 Management method for massive network management historical data

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101216821B (en) * 2007-01-05 2010-06-16 中兴通讯股份有限公司 Data acquisition system storage management method
CN102033882B (en) * 2009-09-25 2013-07-03 中兴通讯股份有限公司 Method and system for storing performance data
CN101697152A (en) * 2009-10-23 2010-04-21 金蝶软件(中国)有限公司; Database storage system and method and device for splitting data thereof

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030154206A1 (en) * 2002-02-08 2003-08-14 Sanyo Electric Co., Ltd. File data storage management method, file data storage device, program executing processing for storing file data, and storage medium
CN101149744A (en) * 2007-10-18 2008-03-26 中兴通讯股份有限公司 Method for executing uniform ordering for multiple data source
CN101566986A (en) * 2008-04-21 2009-10-28 阿里巴巴集团控股有限公司 Method and device for processing data in online business processing
CN102542054A (en) * 2011-12-29 2012-07-04 厦门雅迅网络股份有限公司 Method utilizing cache tables to improve insertion performance of data in database
CN102867071A (en) * 2012-10-19 2013-01-09 烽火通信科技股份有限公司 Management method for massive network management historical data

Also Published As

Publication number Publication date
CN102867071B (en) 2015-04-29
PH12014502368A1 (en) 2015-01-12
CN102867071A (en) 2013-01-09

Similar Documents

Publication Publication Date Title
EP2633413B1 (en) Low ram space, high-throughput persistent key-value store using secondary memory
US7725437B2 (en) Providing an index for a data store
JP5377318B2 (en) Storage management of individually accessible data units
CN104067216B (en) System and method for implementing expansible data storage service
US7885932B2 (en) Managing storage of individually accessible data units
US20030009443A1 (en) Generic data aggregation
CN103177055B (en) It is stored as row storage and row stores the hybrid database table of the two
US8359316B2 (en) Database table look-up
EP0472736A1 (en) Sampled data storage and editing system
US7730045B2 (en) Monitoring usage of components in a database index
Golab et al. Data stream management
US20190266195A1 (en) Filtering queried data on data stores
JPH0628231A (en) Method for storing and maintaining data base to be entered in computer system and data base management system
JP2002244898A (en) Database managing program and database system
US9811570B2 (en) Managing storage of data for range-based searching
CN101315628A (en) Internal memory database system and method and device for implementing internal memory data base
EP0586550A4 (en) Relational data base memory utilization analyzer
JP2001022617A (en) File managing method
CN101216821A (en) Data acquisition system storage management method
Nath et al. Online maintenance of very large random samples on flash storage
JP5922716B2 (en) Handling storage of individually accessible data units
US7263520B2 (en) Fast aggregation of compressed data using full table scans
CN101533408B (en) Processing method and processing device of mass data
EP2946333B1 (en) Efficient query processing using histograms in a columnar database
JP3584630B2 (en) Classification and aggregation processing method in database processing system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13846463

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct app. not ent. europ. phase

Ref document number: 13846463

Country of ref document: EP

Kind code of ref document: A1