CN101119191A - Data consistency collecting method - Google Patents
Data consistency collecting method Download PDFInfo
- Publication number
- CN101119191A CN101119191A CNA2007100493850A CN200710049385A CN101119191A CN 101119191 A CN101119191 A CN 101119191A CN A2007100493850 A CNA2007100493850 A CN A2007100493850A CN 200710049385 A CN200710049385 A CN 200710049385A CN 101119191 A CN101119191 A CN 101119191A
- Authority
- CN
- China
- Prior art keywords
- data
- network element
- consistent
- collecting
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present discloses a data conformity collecting method. The present invention is characterized in that a data collecting method which combines a population parameter collecting with an increment collecting to collect data from element locations. The population parameter collecting requires that the data that element locations output keep conformity. The increment collecting is directed to the data that requires data to keep conformity, and the collecting just collects the data that has changed from former colleting time to the present collecting time. The present invention is not only can ensure the allsidedness and precision of data collecting process is between element locations, but also can ensure the timeliness of data colleting process which requires conformity between element locations. Therefore the detection and process of the realtime property and precision of discord data systems is greatly enhanced, and the process function of the discord data systems is also enhanced. The present invention is better applicable to fields of communication, banking, insurance, stock share exchange, various payment systems and network account checking. The present invention is more applicable to the systems and methods of data conformity matching synchronization.
Description
Technical field
The present invention relates to communication and network data acquisition method field, particularly a kind of data consistency collecting method.
Background technology
Along with communication, increasing of web service product, accept the variation of channel, network business platform complicated, usually need the common a certain business that realizes that cooperates of a plurality of network elements, some data of the network element that these cooperatively interact must be consistent, but, because network failure, the transfer of data fault, disturb, the operation flow of accident and enterprise product design and the reasons such as imperfection of management system, the inconsistent situation of data is on the rise in the network, the business that causes thus, charge and service problem, caused the customer complaint increase, customer satisfaction reduces, the business event losses of revenues.At present, also there are not the good technical means to solve the inconsistent detection of data, handle and relevant problem of management, normally operator finds just that by client's complaint the data between network element are inconsistent passively, initiate processing at last, keep the data consistent of business support net unit and Data-carrying net unit data between network element.
How on one's own initiative, find that in time the inconsistent situation of related network elements data is one of final key technologies of improving solution network data consistency problem in the network.And how comprehensively, accurately and timely the data that require between acquisition network element to be consistent are the first steps that solve these a series of problems.Particularly when having lot of data to gather and to handle, carry out once comprehensively that data acquisition process may need to expend a large amount of time and resource, how to accomplish comprehensively, gather accurately and timely extremely important especially.
Summary of the invention
The present invention provides the method that can require the data that are consistent comprehensively, accurately and timely between acquisition network element for addressing the above problem, and is specially adapted to the comparison synchro system of data consistency.
Technical scheme of the present invention is as follows:
Data consistency collecting method is characterized in that: the data acquisition modes that employing full dose collection and incremental crawler combine is from the network element image data; Described full dose collection requires network element to derive the data that all requirements are consistent; Described incremental crawler is at the data that require to be consistent, and only gathers the data that last acquisition time changes in the current acquisition time.
Described by full dose collection and incremental crawler combination, be meant when the deal with data consistency problem, adopt the full dose collection to guarantee the comprehensive and accuracy of data acquisition, do not omit as far as possible in the fiducial time section and require to be consistent but actual inconsistent data between network element; Adopt incremental crawler to guarantee the promptness of data acquisition, so just the data that can avoid repeated acquisition to be consistent are wasted the resource and the time of system.
Described network element is meant the node that has data storage function in the network, comprises the network element and the network element that is used to realize professional service carrying network of the business supporting network that is used for service handling and management.
Described data acquisition is a process of obtaining data from network element.
Described full dose collection and incremental crawler are realized the collection of the data that requirement is consistent by file mode or on-line mode.
Described on-line mode refers to set up network and connects, and collects the mode that requires the data that are consistent by Network Transmission; Described file mode refers to the data that require to be consistent are preserved with file mode, and the mode by document copying or file transfer collects the data that requirement is consistent again.
Described on-line mode comprises gathering to connect to be set up and transmission course, collection connects foundation and is meant that initiating the collection request by comparison synchro system or network element extremely sets up the process of gathering connection, gather the difference that connects according to acquisition protocols, set up different connections, as ftp, http, tcp/ip or custom protocol etc.; Described transmission is the process that the data that require between network element to be consistent are transmitted between network element.
Beneficial effect of the present invention is as follows:
The present invention adopts the method for full dose and increment combination, comprehensive, accuracy both can have been guaranteed to the data acquisition process that requires between network element to be consistent, can guarantee the promptness of the data acquisition process that requirement is consistent between the data network element again, thereby strengthen the real-time and the accuracy of detection, the inconsistent system of deal with data greatly, improve its handling property.The present invention is specially adapted to fields such as communication, bank, insurance, share certificate transaction, various fare payment system, network accounting checking; The present invention also is specially adapted to data consistency comparison synchro system and method.
Description of drawings
Fig. 1 is a data acquisition process process schematic diagram of the present invention
Fig. 2 is the function framework schematic diagram of the data consistency comparison synchro system that the present invention was suitable for
Fig. 3 is the flow chart of data processing figure of the data consistency comparison synchro system that the present invention was suitable for
Embodiment
Embodiment 1
As shown in Figure 1, data consistency collecting method, the data acquisition modes that employing full dose collection and incremental crawler combine is from the network element image data; Described full dose collection requires network element to derive the data that all requirements are consistent; Described incremental crawler is at the data that require to be consistent, and only gathers the data that last acquisition time changes in the current acquisition time.
Described by full dose collection and incremental crawler combination, be meant when the deal with data consistency problem, adopt the full dose collection to guarantee the comprehensive and accuracy of data acquisition, do not omit as far as possible in the fiducial time section and require to be consistent but actual inconsistent data between network element; Adopt incremental crawler, guarantee the promptness of data acquisition, the data that can avoid repeated acquisition to be consistent are wasted the resource and the time of system like this.
Described network element is meant the node that has data storage function in the network, comprises the network element and the network element that is used to realize professional service carrying network of the business supporting network that is used for service handling and management.
Described data acquisition is a process of obtaining data from network element.
Described full dose collection and incremental crawler are realized the collection of the data that requirement is consistent by file mode or on-line mode.
Described on-line mode refers to set up network and connects, and collects the mode that requires the data that are consistent by Network Transmission; Described file mode refers to the data that require to be consistent are preserved with file mode, and the mode by document copying or file transfer collects the data that requirement is consistent again.
Described on-line mode comprises gathering to connect to be set up and transmission course, the collection connection is set up and is meant that initiating collection by system or network element asks extremely to set up the process of gathering connection, gather the difference that connects according to acquisition protocols, set up different connections, as ftp, http, tcp/ip or custom protocol etc.; Described transmission is the data transmission procedure of data between network element that requirement is consistent between network element.
Embodiment 2
Data consistency collecting method, when being applicable to moving communicating field, the data acquisition modes that employing full dose collection and incremental crawler combine is from the network element image data, and described full dose collection requires network element to derive the data that all requirements are consistent; Described incremental crawler is at the data that require to be consistent, and only gathers the data that last acquisition time changes in the current acquisition time.
Described network element is a network element such as HLR/AUC, SCP/SMP, DSMP in network element BOSS and the service carrying network in the business supporting network.
Described data acquisition is a process of obtaining data from network elements such as BOSS, HLR/AUC, SCP/SMP, DSMP.
Described full dose collection and incremental crawler are realized the collection of the data that requirement is consistent by file mode or on-line mode.
Described on-line mode refers to set up network and connects, and collects the mode that requires the data that are consistent by Network Transmission; Described file mode refers to the data that require to be consistent are preserved with file mode, and the mode by document copying or file transfer collects the data that requirement is consistent again.
Described incremental crawler adopts dual mode, and a kind of is on-line mode, is undertaken by the OLRT query interface, requires network element to provide Subscriber Number, office data sign to carry out the inquiry of user profile, business bureau's data; Another kind of then mode by file requires network element to provide to have taken place after last the collection file of the data that change.
The real-time inquiry of described incremental crawler can be adopted different inquiry modes at its interface for different network elements:
1.HLR/AUC, realize the inquiry of incremental portion data by the real-time command interface.
2.SCP/SMP, realize the inquiry of incremental portion data by the MML instruction interface.
3.DSMP,, realize the inquiry of incremental portion data by real-time query interface.
4. for other provincial business network element, realize inquiry, the existing query interface of utilization or the newly-built real-time query interface realization of incremental portion data by corresponding query interface.
5. for the shared network element in the whole nation, query statement carries out transfer by one-level BOSS hinge, utilizes existing query interface or newly-built real-time query interface to realize.
During the deal with data consistency problem, adopt the full dose collection to guarantee the comprehensive and accuracy of data acquisition, accomplished not omit in the fiducial time section and require to be consistent but actual inconsistent data between network element; And in follow-up processing, adopt incremental crawler, only gather the data that last acquisition time changes in the current acquisition time, the data of having avoided repeated acquisition to be consistent like this, the resource of waste system and time, thereby the promptness of assurance data acquisition.
Embodiment 3
Shown in Fig. 2～3, the synchronous system of data consistency comparison that the present invention is suitable for comprises display layer, data storage processing layer and interface layer; Described display layer is used for man-machine data interaction, represents various Data Management Analysis results; Described data storage processing layer is used for the processing logic of various data, comprise data acquisition module, data preprocessing module, data comparing module, data simultaneous module and supplementary module, data acquisition module is used for image data, data preprocessing module is used for data are put in order, changed and resolve, the data comparing module is used for the data comparison, and data simultaneous module is carried out Synchronous Processing according to comparison result; Described system interface layer is used for the communication interaction of system and each network element.
Described display layer is to provide a personal-machine mutual portal interface to the operation/maintained/managed personnel that use native system, the personnel of various using systems can add up extraction, monitoring, alarm and corresponding various configuration setting operation by this door, and the various rules that relate to for the system handles function, parameter configuration can realize by this door simultaneously.
The supplementary module of described data storage processing layer is by checking in module, scheduler module, statistical module, analysis module, system management module and other modules one to a plurality of combinations/integrated forming.
Described system management module is used for the management to the various settings of system running environment, comprises setting and system's running log management of process scheduling, the automatic start-stop setting of process, data backup; Configuration management is used for the management maintenance to the configuration data of each functional module of system, and what comprise allocation list, configuration file increases, deletes, changes, looks into operation; User management is used for the management to all operations personnel's of using system job number, password, authority, the job number rights management mechanism of layering is adopted in user management, the job number of each level has identical authority, and the management of authority is carried out authority by the user of last layer time and given, changes and cancel; The alarm setting of system management module and processing are the unusual generation warning information that system monitoring is arrived, and behind the generation warning information, in time notify in the alarm monitoring interface display and by the mode of setting.
Described system management module also has the function of selecting the automatic/hand pattern, and when selecting automatic mode, system can finish the comparison synchronizing process of data consistency automatically, generates synchrodata and processing automatically according to the Synchronous Processing algorithm; Select manual mode, then can finish the comparison synchronizing process of data consistency step by step by manual intervention.
Alarm level and alarm mode can be set in described alarm monitoring interface, and the alarm mode includes note, Email, sound, light, page-images, text, forms mode etc.
Described system interface layer comprises the data acquisition interface that is used between each network element, is used for synchronous Data synchronization interface to each network element, and the data distribution interface.
The hardware configuration that described data consistency is compared synchronous plateform system comprises memory device, FC-SWITCH and database server, WEB server, backup server, database/application/interface server and switch.
Data consistency comparison method for synchronous, its concrete steps are as follows:
A, system communicate alternately by interface layer and network element, data acquisition module carries out data acquisition according to interface standard and parameter adaptation from network element, by the situation of data preprocessing module according to network element data, the network element data that collects is carried out preliminary treatment, carry out dissection process then and obtain useful data;
B, data comparing module are compared to the data that steps A obtains, and inconsistent data are recorded and are kept in the difference tabulation;
C, according to the described difference tabulation that obtains of step B, data simultaneous module is verified the validity of data difference in conjunction with the validity proof rule, to difference is that active data generates corresponding synchrodata in conjunction with synchronization rules, and the interface of synchrodata by correspondence is sent to related network elements, in related network elements, carry out corresponding increase/deletion/modification data manipulation then, make that data are consistent between network element.
Described network element is meant the node that has data storage function in the network, comprises the business support network element that is used for service handling and management and is used to realize professional service bearer network element.
Described data acquisition is a process of obtaining data from network element, and the data acquisition modes that employing full dose collection and incremental crawler combine is from the network element image data; Described full dose collection requires network element to derive the data that all requirements are consistent; Described incremental crawler is at the data that require to be consistent, and only gathers the data that last acquisition time changes in the current acquisition time.
Described employing full dose is gathered and the incremental crawler combination, be meant when the deal with data consistency problem, adopt the full dose collection to guarantee the comprehensive and accuracy of data acquisition, do not omit as far as possible in the fiducial time section and require to be consistent but actual inconsistent data between network element; Adopt incremental crawler to guarantee the promptness of data acquisition, so just the data that can avoid repeated acquisition to be consistent are wasted the resource and the time of system.
Described full dose collection and incremental crawler are realized the collection of the data that requirement is consistent by file mode or on-line mode.
Described on-line mode refers to set up network and connects, and collects the mode that requires the data that are consistent by Network Transmission; Described file mode refers to the data that require to be consistent are preserved with file mode, and the mode by document copying or file transfer collects the data that requirement is consistent again.
Described on-line mode comprises gathering to connect to be set up and transmission course, the collection connection is set up and is meant that initiating collection by system or network element asks extremely to set up the process of gathering connection, gather the difference that connects according to acquisition protocols, set up different connections, as ftp, http, tcp/ip or custom protocol etc.
Described preliminary treatment is when the network element data of collection can't be directly used in comparison, and network element data is handled, and with the process that is used to compare, preliminary treatment comprises zero extremely multinomial operation in the operations such as filtration, verification, reparation, format conversion.The filtration of described data is processes of data invalid or useless in the image data being investigated filtration; Described data check is the process of the data that collect being carried out accuracy and the rational judgement of logic; Described data reparation is to repair abnormal data according to the attribute of data itself and Data Source; Described format conversion is that network element data can be used for the process that data are compared from being converted to of form that can't be used for the data comparison.
The process of the corresponding relation of the described parsing data that to be the network element data that is consistent of the requirement that will collect be identified for comparing according to the data implication.
Described comparison is that the data after resolving are compared, and finds the also process of record data consistency situation then.
The described network element data that collects carries out preliminary treatment and parsing, carries out the data comparison then; Perhaps also can carry out gathering again after the preliminary treatment to network element data earlier, and then resolve, carry out the data comparison at last.
Described difference tabulation is meant the record set that can reflect network element data consistency situation that generates by comparison, the inconsistent network element data that ordinary circumstance difference tabulation is found during for comparison and in the positional information of network element, as find that the A customer data is inconsistent when comparing, then will compare both sides separately the A customer data and in difference list records of positional information generation of network element.
Described validation verification adopts the checking of one-sided checking, bilateral or exempts a kind of in the proof rule and verify to multiple rule; One-sided checking refers to whether needs are carried out Synchronous Processing for data difference according to the tabulation of the network element data of side checking difference; The bilateral checking is meant whether needs are carried out Synchronous Processing to the data difference of tabulating according to the network element data checking difference of both sides; Exempt proof rule and be meant that the acquiescence variance data is effective, directly do not carry out Synchronous Processing thereby do not carry out any checking.
Among the step C, described related network elements is meant the inconsistent network element of back data of comparing; Corresponding just be meant with synchronization rules corresponding to.
After finishing secondary data comparison and data sync, system generates statistical report automatically and shows, simultaneously according to the result, can also carry out alarming processing and notice, and generates the historical statistics report, carries out statistical query by statistical report.
After result after comparison is handled is saved in corresponding storehouse table, adopt comparison to check that the equilibrium relation of judgment data and daily record verify the integrality of entire comparison process and the accuracy of comparison; And when occur in data comparisons and the data synchronization process unusual the time, according to the processing policy that sets in advance handle remove unusual, and generation abnormality data table and daily record.
What the data balancing that described comparison is checked concerned totally is:
The data total amount of comparison inlet=comparison outlet pass through that data bulk+comparison exports do not pass through data bulk.
The designed english term table of comparisons of the present invention is as follows:
| Abbreviation | The English description | Chinese is described |
| BOSS | Business Operation Support System | Business operation support system |
| MISC | Mobile Information System center | Mobile Information System center |
| SCP | Service Control Point | Service control point |
| HLR | Home Location Register | Attaching position register |
| AUC | Authentication Centre | AUC |
| SMP | Sevice Managerment Point | The service management point |
| DSMP | Data Service Magagerment Platform | Service Provisioning Administration Core Environment |
| ftp | File Transfer Protocol | File transfer protocol (FTP) |
| http | Hypertext Transfer Protocol | HTML (Hypertext Markup Language) |
| tcp/ip | Transmission Control Protocol/Internet Protocol | Be used for one group of communications protocol of network, comprise IP and TCP |
Table 1
Claims (6)
1. data consistency collecting method is characterized in that: adopt data acquisition modes that full dose collection and incremental crawler combine from the network element image data; Described full dose collection requires network element to derive the data that all requirements are consistent; Described incremental crawler is at the data that require to be consistent, and only gathers the data that last acquisition time changes in the current acquisition time.
2. data consistency collecting method according to claim 1 is characterized in that: described network element is meant the node that has data storage function in the network.
3. data consistency collecting method according to claim 1 is characterized in that: described data acquisition is a process of obtaining data from network element.
4. data consistency collecting method according to claim 1 is characterized in that: described full dose collection and incremental crawler are by realize requiring the being consistent collection of data of file mode or on-line mode.
5. data consistency collecting method according to claim 1 is characterized in that: described on-line mode refers to set up network and connects, and collects the mode that requires the data that are consistent by Network Transmission.File mode refers to the data that require to be consistent are preserved with file mode, and the mode by document copying or file transfer collects the data that requirement is consistent again.
6. data consistency collecting method according to claim 1, it is characterized in that: described on-line mode comprises gathering to connect to be set up and transmission course, the collection connection is set up and is meant that initiating collection by business support network element or service bearer network element asks extremely to set up the process of gathering connection, gather the difference that connects according to acquisition protocols, set up different connections; Described transmission is the process transmitted of data between network element that requirement is consistent between network element.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2007100493850A CN101119191A (en) | 2007-06-27 | 2007-06-27 | Data consistency collecting method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2007100493850A CN101119191A (en) | 2007-06-27 | 2007-06-27 | Data consistency collecting method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101119191A true CN101119191A (en) | 2008-02-06 |
Family
ID=39055147
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2007100493850A Pending CN101119191A (en) | 2007-06-27 | 2007-06-27 | Data consistency collecting method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101119191A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101876991B (en) * | 2009-11-16 | 2012-09-05 | 中兴通讯股份有限公司 | Database processing method and device for web television statistical analysis system |
| CN102903059A (en) * | 2012-09-25 | 2013-01-30 | 上海证券交易所 | Quotation data consistency protective method based on timer shaft in data transmission |
| CN107948288A (en) * | 2018-01-08 | 2018-04-20 | 中山路得斯空调有限公司 | A kind of data acquisition for saving flow and polishing method |
| CN109819019A (en) * | 2018-12-29 | 2019-05-28 | 中国科学院计算技术研究所 | Monitoring and statistical analysis technique and system for the acquisition of large scale network data |
-
2007
- 2007-06-27 CN CNA2007100493850A patent/CN101119191A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101876991B (en) * | 2009-11-16 | 2012-09-05 | 中兴通讯股份有限公司 | Database processing method and device for web television statistical analysis system |
| CN102903059A (en) * | 2012-09-25 | 2013-01-30 | 上海证券交易所 | Quotation data consistency protective method based on timer shaft in data transmission |
| CN102903059B (en) * | 2012-09-25 | 2016-01-13 | 上海证券交易所 | Based on the market data consistency protective method of time shaft in a kind of data transmission |
| CN107948288A (en) * | 2018-01-08 | 2018-04-20 | 中山路得斯空调有限公司 | A kind of data acquisition for saving flow and polishing method |
| CN107948288B (en) * | 2018-01-08 | 2020-06-30 | 中山路得斯空调有限公司 | Data acquisition and complement method capable of saving flow |
| CN109819019A (en) * | 2018-12-29 | 2019-05-28 | 中国科学院计算技术研究所 | Monitoring and statistical analysis technique and system for the acquisition of large scale network data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101094051B (en) | System and method for synchronizing comparison of data consistency | |
| CN101079749A (en) | Data consistency detection method | |
| CN101079683A (en) | Data consistency processing method | |
| CN100546260C (en) | The inconsistent detection processing method of data | |
| CN102097859B (en) | Power grid dispatching intelligentized and instruction informationized working platform | |
| EP3076346A1 (en) | Systems and methods for improved demand response management system (drms) | |
| CN100471135C (en) | Device, system and method for realizing business distribution and synchronization | |
| CN101174973A (en) | Network safety control construction | |
| CN101471962A (en) | Device management portal, system and method | |
| CN104599111B (en) | A kind of business management method and device | |
| CN102035660A (en) | Internet data center (IDC) network-based service processing method, equipment and system | |
| CN102508773B (en) | Method and device for monitoring WEB service system simulation based on Internet explorer (IE) kernel | |
| CN101119191A (en) | Data consistency collecting method | |
| CN109787844A (en) | A kind of distribution master station communication fault fast positioning system | |
| CN101482960A (en) | Method and system for marketing in bank hall | |
| CN102298738A (en) | intelligent audit management system | |
| CN201733328U (en) | Enterprise data maintaining device and system based on bank system | |
| CN109947844B (en) | Medical data management system based on medical block chain | |
| CN102930392A (en) | System for running information of transformer substation | |
| CN101488203A (en) | Method and system for real-time browsing production technique document in production field | |
| CN111146868A (en) | Power communication machine room dynamic loop monitoring standardized access method and system | |
| Yohapriyadharsini et al. | Multi-server Markovian heterogeneous arrivals queue with two kinds of working vacations and impatient customers | |
| CN106254173A (en) | The monitoring system of public transport automatic fare collection terminal and monitoring method | |
| CN110675283B (en) | Power distribution equipment synchronous management system and method | |
| CN101996355A (en) | System and method for immediately updating, verifying and releasing electronic information and file |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Open date: 20080206 |