CN102098170B - Data acquisition optimization method and system - Google Patents
Data acquisition optimization method and system Download PDFInfo
- Publication number
- CN102098170B CN102098170B CN200910252748.XA CN200910252748A CN102098170B CN 102098170 B CN102098170 B CN 102098170B CN 200910252748 A CN200910252748 A CN 200910252748A CN 102098170 B CN102098170 B CN 102098170B
- Authority
- CN
- China
- Prior art keywords
- network element
- play amount
- time
- omc
- newly
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention discloses a data acquisition optimization method and a data acquisition optimization system. The method comprises the following steps that: an operation and maintenance centre (OMC) determines a uniform minimal report particle size for each network element, configures the offset of a report time for each network element according to the minimal report particle size, and transmits an offset-containing performance task to each network element; and after receiving the performance task, the network element determines the report time for reporting performance data according to the offset in the performance task. By adopting the data acquisition optimization method and the data acquisition optimization system provided by the invention, tremendous pressure, caused by large amount of file transmission, of the OMC can be avoided and the load of a server can be reduced.
Description
Technical field
The present invention relates to the communications field, relate in particular to a kind of optimized data collection method and system.
Background technology
In OMC (Operations & Maintenance Center, operation maintenance center) total demand, be the performance index data that can gather and manage 10000 network elements to the requirement of performance module.The acquisition granularity can be set by the user, and option comprises 15 minutes and 30 minutes two kinds of granularities.
The main flow process of data acquisition is as follows:
First in OMC, create a performance task, the main contents of task comprise: carry out data acquisition to which network element, will gather the granularity of which performance index, collection, and the beginning gathering and the time of end;
Performance task is issued to after network element, and network element is according to setting of task, and each granularity is collected and statistic property index, and after a acquisition granularity finishes, active reporting is to OMC, until the end of task.
Wherein, reporting of the performance data of collection is that OMC is as ftp server by FTP (File Transfer Protocol, file transfer protocol (FTP)) agreement, and network element uploads to OMC by the data of collection by FTP.The network element performance deposit data of each the acquisition granularity is in a text.Text is compressed, and zip (a kind of compressed format of the computer documents) file after compression is exactly the carrier of transmission.OMC receives after zip file, first decompresses, then the data text file after decompressing is imported in database, and so far, once complete performance index collecting flowchart finishes.
Mainly there are following two point defects in the solution of current data acquisition: one, and the concurrency that FTP uploads is excessive, will have 10000 network elements and upload simultaneously, and OMC system assignment is limited to the FTP number of connection of performance management module, only has 30 at most time; Its two, transmission is frequent, the minimum the acquisition granularity that will support is at present 15 minutes, that is to say under limiting case every 15 minutes, just has 10000 network element upload files.
Under this pattern, often can cause network element to fight for the situation of FTP resource.Once certain network element does not obtain FTP and uploads link, it can, in a period of time, after normally several seconds, ask FTP link again so.This will cause ftp server congested for a long time, and system resource takies too highly, but also after there will be a acquisition granularity to finish, still have some network elements to fail to obtain FTP link, thereby cannot complete uploading of performance index data.
Summary of the invention
The technical problem to be solved in the present invention is to provide a kind of optimized data collection method and system, avoids heap file to be transferred to the immense pressure that OMC causes, and reduces the load of server.
In order to address the above problem, the invention provides a kind of optimized data collection method, described method comprises:
Operation maintenance center (OMC) determines that for each network element unified minimum reports granularity, and reporting granularity according to described minimum is the side-play amount calling time in each network element configuration, and described side-play amount is included in and in performance task, is handed down to each network element;
Described network element receives after described performance task, according to described side-play amount wherein determine reporting performance data on call time.
Further, described OMC as required total number of the network element of reporting performance data determine that described minimum reports granularity.
It is further, described that to report granularity according to described minimum be that the side-play amount calling time in each network element configuration refers to:
All network elements that need reporting performance data are divided into multiple side-play amount groups by described OMC, and in same side-play amount group, the side-play amount of each network element is identical;
And the maximum offset of described side-play amount group should be less than or equal to described minimum and report granularity.
Further, the side-play amount of the network element of adjacent side-play amount group differs a period of time T, and this period of time T has at least been greater than the time of a performance transfer of data.
Further, in described OMC, be provided with network element offset information table, for storing the corresponding relation of each network element and its side-play amount;
Described OMC comprises two queues: allocation offsets amount queue, for storing the network element of allocation offsets amount and the corresponding relation of its side-play amount; Side-play amount queue to be allocated, for storing side-play amount to be allocated.
Further, in the time that a certain network element is deleted, described OMC is by described network element from deleting the queue of allocation offsets amount, and the side-play amount that it is used is stored in queue to be allocated; Leave out the record of network element described in network element offset information table in database simultaneously.
Further, the corresponding relation of storing in described network element offset information table comprises the corresponding relation of the logical address of network element and the side-play amount of this network element;
In the time that network element moves group, the logical address of this network element in described network element offset information table is updated to the new logical address of this network element.
Further, in the time increasing network element newly in system, be described newly-increased network element allocation offsets amount in the following way:
Search in described side-play amount queue to be allocated and whether have available planned number, if had, from side-play amount queue to be allocated, be an available planned number of its distribution, otherwise, then search in current maximum offset group whether have available planned number, if had, for this network element distributes current maximum offset, otherwise, newly-built side-play amount group, from this newly-built side-play amount group, be an available planned number of this network element distribution, and the corresponding relation of this network element and side-play amount is added in described network element offset information table.
The present invention also provides a kind of optimized data collection system, comprising: OMC, and network element,
Described OMC is used for, and determines unified minimum report granularity for each network element, and reporting granularity according to described minimum is the side-play amount calling time in each network element configuration, and described side-play amount is included in and in performance task, is handed down to each network element;
Described network element is used for, receive after described performance task, according to described side-play amount wherein determine reporting performance data on call time.
Further, described OMC is used for, side-play amount for calling time in each network element configuration in the following way: all network elements that need reporting performance data are divided into multiple side-play amount groups, in same side-play amount group, the side-play amount of each network element is identical, the side-play amount of the network element of adjacent side-play amount group differs a period of time T, this period of time T has at least been greater than the time of a performance transfer of data, and the maximum offset of described side-play amount group should be less than or equal to described minimum and reports granularity; And,
For the corresponding relation of the each network element of storage system and its side-play amount; When in system when newly-increased network element, for newly-increased network element allocation offsets amount, store the corresponding relation of this newly-increased network element and its side-play amount, and the side-play amount of distribution is included in and in performance task, is handed down to this newly-increased network element; In the time that a certain network element is deleted in system, delete this network element of storage and the corresponding relation of its side-play amount.
Compared with prior art, above-mentioned embodiment at least has following beneficial effect:
The NE quantity that need to manage for OMC is large, transmission data file is situation frequently, the present invention provides a kind of prioritization scheme to the collecting flowchart of performance index, has alleviated to a certain extent heap file and has been transferred to the immense pressure that OMC causes, and has reduced server load; And design and realization are rationally, are not minimizing transmission quantity simply; By the ability to bear of client's actual demand and server, and the execution efficiency of OMC system reaches a kind of meeting point of the best.
Brief description of the drawings
Fig. 1 is the schematic block diagram of the optimized data collection system of the embodiment of the present invention;
Fig. 2 is the newly-built network element side-play amount allocation flow schematic diagram of the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawings and the specific embodiments the enforcement of technical solution of the present invention is described in further detail.
The present invention mainly adopts following technical scheme:
Step 1, OMC is that each network element determines that unified minimum reports granularity; And to report granularity be the side-play amount calling time in each network element configuration according to the described minimum of determining, and be included in and in performance task, be handed down to each network element;
Step 2, each network element is according to calling time in described side-play amount specified data, that is and, after calling time on arriving, network element again etc. after shift time to be specified, reports performance file.
Particularly, can, according to the total number of network element that needs uploading data, automatically determine the granularity that reports of each network element, for example:
1000 below network element, it is 15 minutes that minimum reports granularity;
1001 to 5000 network elements, it is 60 minutes that minimum reports granularity;
5000 more than network element, it is 120 minutes that minimum reports granularity.
Further, report granularity to determine maximum offset, OMC can be in conjunction with reporting granularity to configure the side-play amount of each network element.For example, report the relation of granularity and maximum offset as follows:
Reporting granularity is 15 minutes, and the maximum offset of allowing is 12 minutes;
Reporting granularity is 60 minutes, when the maximum offset of allowing 55 minutes;
Reporting granularity is 120 minutes, when the maximum offset of allowing 110 minutes.
Wherein, in the time calculating side-play amount, be first responsible for calculating the concrete time of each network element upload file by OMC;
According to statistics, completing a FTP transmission approximately needs the time of 6 seconds.In these 6 seconds, it is time-consuming a lot of that network element and OMC set up the process of FTP link, really not long for transmitting time of data.Hence one can see that, transmits a zip who comprises a text, and a zip who comprises four texts of transmission, and time phase difference used is few.That is, one reports the data that report a acquisition granularity in granularity, and the time used with the data that report four the acquisition granularities equates substantially.
Then, report granularity to unifiedly calculate the side-play amount of each reported by network elements file by OMC basis, for example, for the side-play amount of each network element configuration is respectively: 0, T, 2T, 3T.。。, be offset successively a period of time T, wherein this cycle T is 12 seconds.
Adopt technique scheme, can make all network elements on call time and be evenly distributed on unified reporting in granularity, can effectively avoid the pressure of FTP resource.
Below in conjunction with drawings and the specific embodiments, the enforcement of technical solution of the present invention is described in further detail.
With reference to Fig. 1, in the embodiment of the present invention, OMC is as network management service, the network element of all lower volatility tasks (the present embodiment middle finger base station) is divided into multiple side-play amount groups, maximum 30 network elements of each side-play amount group, in same side-play amount group, the side-play amount of network element is identical, i.e. the upload file simultaneously of network element in same side-play amount group.The side-play amount of the network element of adjacent side-play amount group differs a period of time T, and this period of time T has at least been greater than the time of a performance transfer of data.And the maximum offset of allowing should be not more than minimum and report granularity.
In the present embodiment, the allocative decision of side-play amount comprises:
A, the setting of side-play amount
In the present embodiment, be 0 second by the side-play amount of first side-play amount group, the side-play amount of second side-play amount group is 12 seconds, and the side-play amount of the 3rd side-play amount group is 24 seconds, and the side-play amount of n side-play amount group is 12 × (n-1) second, by that analogy.Be equivalent to for each network element distribution time of 12 seconds is for reported data.
B, the storage of side-play amount
A network element offset information table is set in the performance database of OMC, for storing the corresponding relation of network element and side-play amount, for example, can stores the corresponding relation of logical address and the side-play amount of network element.After startup of server, the network element of storing in this network element offset information table in database and the corresponding relation of side-play amount are read in internal memory.
In OMC internal memory, set up 2 queues: one of them queue is for depositing the side-play amount of having distributed and the corresponding relation of network element, and this queue is after startup of server, and reading out data loads from database; Another is for depositing all scrappy side-play amounts to be allocated.
C, the maintenance of network element offset information table and the reallocation of side-play amount
When network element in network deleted, or newly-increased network element, or network element moves while group, JMS (the Java Messaging Service changing about network element sending by monitoring configuration management, Java messenger service) message, safeguard the offset information in internal memory, revise the relevant information in network element offset information table simultaneously, ensure that database is consistent with the information in internal memory.
When the Configuration Manager of OMC sends certain network element after deleted message to performance management module, performance management module is deleted this network element from the queue of allocation offsets amount, and the side-play amount that it is used is put in queue to be allocated; Leave out the respective record in network element offset information table in database simultaneously.
In the time that network element moves group (the group is here different from the side-play amount group in the present invention), the logical address of network element will change, now, only need upgrade the logical address of corresponding network element in network element offset information table, and without its side-play amount is done to any change.
In the time newly increasing network element in system, can be handled as follows: first from side-play amount queue to be allocated, search for, if do not have availablely in queue to be allocated, the side-play amount of newly-increased network element increases progressively successively.For example, current maximum offset is 24 seconds, and the side-play amount that has 28 network elements is 24 seconds.As increased 5 network elements, the side-play amount that so wherein has 2 network elements is 24 seconds again, and another 3 is 36 seconds, and is recorded in database.
Taking the handling process that newly increases network element in system as example, the allocation flow of side-play amount of the present invention is elaborated below.
With reference to figure 2, in the time newly increasing network element in system, corresponding handling process can comprise the steps:
Step 1, OMC carries out the preparation of performance task for increasing new network element newly;
Step 2, searches in side-play amount queue to be allocated whether have available planned number, if had, from side-play amount queue to be allocated, is an available planned number of this network element distribution, and goes to step 5, otherwise, carry out next step;
Step 3, searches in current maximum offset group whether have available planned number, if had, for this network element distributes current maximum offset, and goes to step 5; Otherwise, carry out next step;
Step 4, newly-built side-play amount group, the side-play amount of this side-play amount group increases progressively according to aforementioned manner, for this network element distributes an available planned number, and is recorded in database, carries out next step;
Step 5, sends performance task to this newly-increased network element.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any amendment of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (7)
1. an optimized data collection method, is characterized in that, described method comprises:
The OMC of operation maintenance center is that each network element determines that unified minimum reports granularity, and reporting granularity according to described minimum is the side-play amount calling time in each network element configuration, and described side-play amount is included in and in performance task, is handed down to each network element;
Described network element receives after described performance task, according to described side-play amount wherein determine reporting performance data on call time;
It is described that to report granularity according to described minimum be that the side-play amount calling time in each network element configuration refers to:
All network elements that need reporting performance data are divided into multiple side-play amount groups by described OMC, and in same side-play amount group, the side-play amount of each network element is identical;
And the maximum offset of described side-play amount group is less than or equal to described minimum and reports granularity;
The side-play amount of the network element of adjacent side-play amount group differs a period of time T, and this period of time T has at least been greater than the time of a performance transfer of data.
2. the method for claim 1, is characterized in that,
Described OMC as required total number of the network element of reporting performance data determines that described minimum reports granularity.
3. method as claimed in claim 2, is characterized in that,
In described OMC, be provided with network element offset information table, for storing the corresponding relation of each network element and its side-play amount;
Described OMC comprises two queues: allocation offsets amount queue, for storing the network element of allocation offsets amount and the corresponding relation of its side-play amount; Side-play amount queue to be allocated, for storing side-play amount to be allocated.
4. method as claimed in claim 3, is characterized in that,
In the time that a certain network element is deleted, described OMC is by described network element from deleting the queue of allocation offsets amount, and the side-play amount that it is used is stored in queue to be allocated; Leave out the record of network element described in network element offset information table in database simultaneously.
5. method as claimed in claim 3, is characterized in that,
The corresponding relation of storing in described network element offset information table comprises the corresponding relation of the logical address of network element and the side-play amount of this network element;
In the time that network element moves group, the logical address of this network element in described network element offset information table is updated to the new logical address of this network element.
6. method as claimed in claim 3, is characterized in that,
In the time increasing network element newly in system, be described newly-increased network element allocation offsets amount in the following way:
Search in described side-play amount queue to be allocated and whether have available planned number, if had, from side-play amount queue to be allocated, be an available planned number of its distribution, otherwise, then search in current maximum offset group whether have available planned number, if had, for this network element distributes current maximum offset, otherwise, newly-built side-play amount group, from this newly-built side-play amount group, be an available planned number of this network element distribution, and the corresponding relation of this network element and side-play amount is added in described network element offset information table.
7. an optimized data collection system, is characterized in that, comprising: OMC, and network element,
Described OMC is used for, and determines unified minimum report granularity for each network element, and reporting granularity according to described minimum is the side-play amount calling time in each network element configuration, and described side-play amount is included in and in performance task, is handed down to each network element;
Described network element is used for, receive after described performance task, according to described side-play amount wherein determine reporting performance data on call time;
Described OMC is used for, side-play amount for calling time in each network element configuration in the following way: all network elements that need reporting performance data are divided into multiple side-play amount groups, in same side-play amount group, the side-play amount of each network element is identical, the side-play amount of the network element of adjacent side-play amount group differs a period of time T, this period of time T has at least been greater than the time of a performance transfer of data, and the maximum offset of described side-play amount group should be less than or equal to described minimum and reports granularity; And,
For the corresponding relation of the each network element of storage system and its side-play amount; When in system when newly-increased network element, for newly-increased network element allocation offsets amount, store the corresponding relation of this newly-increased network element and its side-play amount, and the side-play amount of distribution is included in and in performance task, is handed down to this newly-increased network element; In the time that a certain network element is deleted in system, delete this network element of storage and the corresponding relation of its side-play amount.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200910252748.XA CN102098170B (en) | 2009-12-09 | 2009-12-09 | Data acquisition optimization method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200910252748.XA CN102098170B (en) | 2009-12-09 | 2009-12-09 | Data acquisition optimization method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102098170A CN102098170A (en) | 2011-06-15 |
| CN102098170B true CN102098170B (en) | 2014-09-10 |
Family
ID=44131048
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200910252748.XA Active CN102098170B (en) | 2009-12-09 | 2009-12-09 | Data acquisition optimization method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102098170B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102571977B (en) * | 2012-02-06 | 2015-02-04 | 大唐移动通信设备有限公司 | Method and device for transmitting CDL files, and method and device for determining data integrity |
| CN103023887B (en) * | 2012-11-26 | 2016-05-25 | 大唐移动通信设备有限公司 | A kind of method and apparatus based on File Transfer Protocol transfer files |
| CN104254102A (en) * | 2013-06-26 | 2014-12-31 | 中兴通讯股份有限公司 | Measurement report reporting method, communication node and system |
| CN103795803B (en) * | 2014-02-14 | 2018-10-26 | 浪潮天元通信信息系统有限公司 | A kind of method of across a network environment remote deployment distributed data acquisition node |
| CN103942287B (en) * | 2014-04-10 | 2017-06-06 | 大唐移动通信设备有限公司 | Mass data processing method, database server and application server |
| CN109639843A (en) * | 2019-02-26 | 2019-04-16 | 青岛海信移动通信技术股份有限公司 | Data transmission method and device |
-
2009
- 2009-12-09 CN CN200910252748.XA patent/CN102098170B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN102098170A (en) | 2011-06-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102098170B (en) | Data acquisition optimization method and system | |
| CN103905537A (en) | System for managing industry real-time data storage in distributed environment | |
| CN107968802B (en) | Resource scheduling method and device and filtering type scheduler | |
| CN100484017C (en) | Method for statistics of mass performance data in network element management system | |
| CN101710901B (en) | Distributed type storage system having p2p function and method thereof | |
| US9015229B1 (en) | Determining placement of user data to optimize resource utilization for distributed systems | |
| CN101431734B (en) | Method and system for group sending short messages | |
| CN101039212B (en) | Fast data storage method | |
| CN102375837B (en) | Data acquiring system and method | |
| CN105940396A (en) | Hierarchical chunking of objects in a distributed storage system | |
| CN103152393A (en) | Charging method and charging system for cloud computing | |
| CN105025053A (en) | Distributed file upload method based on cloud storage technology and system | |
| CN110209345A (en) | The method and device of data storage | |
| CN101937474A (en) | Mass data query method and device | |
| CN111835562B (en) | Log uploading method, log inquiring method and system | |
| CN107844524A (en) | Data processing method, data processing equipment, computer equipment and storage medium | |
| CN102404760B (en) | Method and device for real-time measurement of system performance | |
| CN116777182B (en) | Task dispatch method for semiconductor wafer manufacturing | |
| CN102480489A (en) | Logging method and device used in distributed environment | |
| CN113867957A (en) | Method and device for realizing elastic expansion of number of cross-cluster containers | |
| CN111858722A (en) | Big data application system and method based on Internet of things | |
| CN103530335A (en) | In-stockroom operation method and device of electric power measurement acquisition system | |
| CN103258389B (en) | The method of self-aided terminal upload file, system and self-aided terminal | |
| CN106713512B (en) | Efficient data transmission method and system for dynamically balancing network bandwidth of Internet of things | |
| WO2017215415A1 (en) | Resource control method and apparatus, and iptv server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |