CN1809108A - Filter based call ticket memory repetition elimination method - Google Patents
Filter based call ticket memory repetition elimination method Download PDFInfo
- Publication number
- CN1809108A CN1809108A CN 200610038375 CN200610038375A CN1809108A CN 1809108 A CN1809108 A CN 1809108A CN 200610038375 CN200610038375 CN 200610038375 CN 200610038375 A CN200610038375 A CN 200610038375A CN 1809108 A CN1809108 A CN 1809108A
- Authority
- CN
- China
- Prior art keywords
- row
- filter
- internal memory
- ticket
- heavy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This invention relates to call record ranking method based on filter slice, which processes the record into set interface output call list files and time key order stored in memory to be filtered by filter slice; wherein, when one list does not find relative time, then the list self-ranging and copying adds to the filter to eliminate following list; the filter slice length is matched with one only time section; when the call time is within the length to operate rank process; if the length is outside length then entering special channel for directly re-ranked.
Description
Technical field
The invention belongs to the extensive accurately data handling utility technology category of telecom operators.
Background technology
Can telecom operators use the record of communication products to carry out accurate charging to the client in service operation, is directly connected to client's satisfaction, also be directly connected to operator and constantly enlarge market.For the accuracy that guarantees to charge, the communication service recorder (ticket) that must guarantee to enter telecommunication charging system had not both had omission is not had repetition yet: for guaranteeing not omit, need check data source; For guaranteeing not repeat, will arrange heavily and handle.Data processing field in enormous quantities in other industry is used, as processing to data such as transaction records in the ecommerce, the demand that the same row of existence heavily handles.
Present telecommunication charging system utilizes the limited characteristic of tables of data unique index in the relational database to arrange heavily processing when data being arranged heavily mostly.The heavy processing procedure of this row has following 2 kinds of specific implementations (is example with the oracle database):
Mode 1:
1. will need to arrange heavy data file and be inserted into the database interface table;
2. calling storing process carries out database and looks into heavily (data of the interface table data with row's heavy filtration table are compared, find out heavy single and normal ticket);
3. the duplicate record in the interface table is inserted heavy single table;
4. for the normal recordings in the interface table, need on the one hand to insert formal tablet menu, need the heavy information of the row of extraction to insert row's heavy filtration table on the other hand, these 2 work must be as 1 affairs.
Mode 2:
1, reads the file that needs row heavy, according to configuration information, the heavy information of row that extracts every record (is the discriminant information of duplicate record, as converse the time started, calling number, called number, the duration of call), these are arranged heavy information merge into a key message field, and generate corresponding subregion field information;
2, utilize batch copy function of ORACLE OCI, these batch data are inserted in the heavy data field of row,, can determine the heavy data field of row fast according to the graded properties of ORACLE;
3, when the heavy data field of the copy row of being inserted in batches, if violate unique index, i.e. there are duplicate record data (promptly heavy single) in explanation, and can return duplicate record row hereof number this moment from copying in batches;
4, according to the heavy single file that returns number, record corresponding in the file is chosen and to be generated the weight monofile, remaining generates normal inventory file.
The saving routing table memory consumption way of CN200310118453.6 Border Gateway Protocol: after this locality is received or self generated route, with Loc-Rib as a public shared Buffer Pool, preserve all existing routing iinformations, each bar route in the establishment Loc-Rib information bank and the mapping relations of Adj-Rib-Out, route is handed over into the filtration of carrying out local policy to the arbitration decisions module, if passed through the decision-making of arbitration modules, give other Peer with the advertising of route that this locality sees fit.Owing to set up the route in the Adj-Rib-Out information bank and the corresponding relation of a certain route in the Loc-Rib information bank, Adj-Rib-Out need not to safeguard that portion treats to send the copy of route again, sharing of realization information saved memory, can save the memory consumption of bgp routing table significantly.
Expression formula in the CN03810570.5 management database system provides a kind of method of managing expression formula, and it comprises that reception comprises first inquiry of conditional expression.This expression formula is represented as the data of row in the table then.Second inquiry is received, first set of its required standard, and carry out second inquiry whether to satisfy the first set selection data of standard at least based on the expression formula in these row.In one embodiment, second set of the further required standard of second inquiry carries out wherein that second inquiry comprises based on data in a plurality of row but not whether expression formula satisfies second standard.The definition special index is with the big expression formula set of effective filtration, and this index can be in the establishment that lists of storage expression formula.A kind of is method as the expression formula value of storage in table, and it is the classification of the judgement in each expression formula, and judges the set of categorical filtering expression formula based on this.
In a word, there is following point in the heavy technology of row based on database:
1, when the ticket amount of needs execution row heavy filtration was huge, treatment effeciency was low;
2, because the heavy process of row is operated database table, use and owe to adjust the capability operation complexity flexibly;
3, row's principal characteristic can depend on high performance database, and the large-scale relational database platform of essential buying high price is invested higher when implementing.
Along with developing rapidly of communication service, data volume sharply increases, and present telecommunication charging system need be handled several hundred million even tens records common every month, and professional going up requires nearest 3 months record data are arranged heavily processing usually.In this case, the heavy technology of above-mentioned row based on database can't satisfy the real-time processing requirements of data in enormous quantities, the heavy processing bottleneck that has become telecommunication charging system of row on performance and efficient.
Summary of the invention
The objective of the invention is: based on the internal memory of filter row weight technology is exactly problems such as inefficiency, complicated operation, scale of investment at traditional data row weight technology is big, proposes a kind ofly to imitate minimizing and arrange the interface of re-computation amount, minimizing database, break away from this method of internal memory rearrangement based on filter to the dependence of database.
Technical solution of the present invention is: based on the ticket row internal memory weighing method of filter, it is characterized in that becoming by the promptly crucial array of arranging heavy record field of the conversation ticket writing file of agreement interface output after the CDR file preliminary treatment, the array of the heavy record field of key row of conversation ticket time, be kept in the internal memory, filter heavy single with the filter form, do not find heavy single time of its correspondence when a ticket, then self arranging of this ticket heavily copies in this filter of adding, is used to prepare the heavy single of eliminating this ticket afterwards; The length of filter is a unique time period of configuration; When by the air time of row's hard words list within this length, then arrange heavily and handle, if not within this length, then enter specific channel; Directly arrange retry at internal memory.
Internal memory row heavily adopts the modular structure structure.
Filter net sheet is big or small customizable in the realization host memory, effectively reduces row's re-computation amount.This method can be used as independently configurable application module and is present in application platform, and the way that the heavy output of internal memory row was imported, accepted to overall applicability by the mode of file or memory interface is arranged the realization that weighs function.This technical support is carried out the configuration that multi-channel parallel is handled according to concrete the application.The heavy technology of internal memory row has considered that by being the setting of the affair mechanism of unit by file, the cleaning and the recovery of field reappears in the row of guaranteeing under the abnormal conditions such as outage.Reduce the interface of database, directly arrange retry at internal memory.
Effect: the actual use at telecommunication charging system shows, arranges heavy technology based on the internal memory of filter and improves the heavy efficient of row at least more than 5 times, and can reduce the investment of buying large-scale relational database.
It below is the test case of the heavy efficient of row.
Test environment: 1) application server: IBM P650 8CPU 8G MEM
/CPU?TYPE：PowerPC_POWER4
2) database server: IBM P650 8CPU 8G MEM
/CPU?TYPE：PowerPC_POWER4
3) in the ORACLE9i database, set up a cover row heavy filtration table, heavy single table, warehousing interface table, formal tablet menu.
Tested object: number of files to be put in storage is: 20; 50000 records of each file.
Method of testing: file is positioned over row's reentry storehouse input directory, starts 1 row's reentry storehouse process.The file number of monitoring input directory is monitored the tablet menu data acknowledgment number simultaneously until being zero.
Test result:
A, based on the row of the database library test of reentrying:
System is consuming time: 220 seconds
Row reentry storehouse speed: 1000000 ÷, 220 pacts=4545/second
B, based on the internal memory of filter row reentry library test:
System is consuming time: 43.5 seconds
Row reentry storehouse speed: 1000000 ÷, 43.5 pacts=22989/second
Test result: approximately be based on 5 times of database row reentry storehouse speed based on the internal memory of filter row reentry storehouse speed.
Note, comprised heavy and 2 processes of warehouse-in of row in the above-mentioned test process, and the required time of data loading is identical basically for 2 kinds of methods.So in fact, arranging heavy speed based on the internal memory of filter is based on database certainly and arranges more than 5 times of heavy speed.
Characteristics of the present invention are: 1, realize the big or small customizable of filter net sheet in the host memory, effectively reduce the re-computation amount of arranging; 2, reduce the interface of database, directly arrange retry, improve performance greatly at internal memory; 3, break away from dependence, small investment, be convenient to transplant database.
Description of drawings
Fig. 1 is that the present invention arranges heavy processing procedure framework schematic diagram
Embodiment
Internal memory rearrangement based on filter adopts the componentization design philosophy to design.This method can be used as independently configurable application module and is present in application platform, and the way that the heavy output of internal memory row was imported, accepted to overall applicability by the mode of file or memory interface is arranged the realization that weighs function.This method support is carried out the configuration that multi-channel parallel is handled according to concrete the application.
Several key object class designs in the internal memory rearrangement are as follows:
CDR file after the preliminary treatment: this document is that preliminary treatment is used through ways such as translation, code system conversions, to the CDR file that switch spues, becomes the message registration file by the output of agreement interface after the processing.May there be duplicate record in this document.Record in this document is generally arranged by the air time order, also exists special circumstances to arrange not according to the air time order.
Filter: the array of the heavy record field of crucial row in certain air time, be kept in the internal memory, play the heavy single effect of filtering.When a ticket is not found heavy single time of its correspondence, then self row of this ticket heavily copy add in this filter, be used to prepare the heavy single of eliminating this ticket afterwards.The length of filter is a unique time period of configuration.When by the air time of row's hard words list within this length, then arrange heavily and handle, if not within this length, then enter specific channel, the filter that accesses the corresponding time period is handled.
Repeat ticket: when one day ticket enters filter and compares, find to have the ticket of repetition, judge that then this ticket for repeating ticket, lists independent heavy single channel in and handle.Other tickets are proceeded row and are heavily handled.
Row is heavy, and processing procedure is as follows: (referring to Fig. 1)
1) reads file after the preliminary treatment,, extract the information of a ticket corresponding field, the row of being processed into weight information according to configuration;
2), determine the filter of the important employing of row according to the heavy information of row of this ticket.
3) if this filter in internal memory, is then arranged heavily to be handled, if not in internal memory, then the heavy file of correspondence row of this filter of search from disk file reads in internal memory and handles.
4) carry out subsequent treatment according to the heavy result of row: if not heavy single, then replenish filter, export normal ticket according to the heavy information of row of this ticket; If heavy single, do not replenish filter, this ticket is output as the repetition ticket.And the relevant daily record of record.
The scope of filter net sheet is: can carry out the parameter setting, performance according to machine is made amendment, and is traditionally arranged to be to two day of current time, for the CDR file in this time period not, scheduling virtual memory or be stored in data in the disk changes to internal memory and arranges heavily and handle.
Support to carry out the configuration that multi-channel parallel is handled: because row heavily is relevant with switch according to concrete the application, so when the heavy passage of configuration row need same switch ticket down must be configured in and arrange heavily below the same passage, other can carry out any channel arrangement as required and make up.
For example: 10 switches are arranged at present, 5 passages of plan configuration; 1,2, No. 3 switches can be put a passage, and 4,5 can put a passage ... but No. 1 switch cannot not only be placed on a passage but also be placed on two passages.
Internal memory row is heavily by being that the setting of the mechanism of unit is by file: internal memory row be heavily by can dynamically arranging by the heavy key element of row, can dispose different filters according to the difference of each service needed.
Claims (8)
1, arranges the internal memory rearrangement based on the ticket of filter, it is characterized in that becoming by the promptly crucial array of arranging heavy record field of the conversation ticket writing file of agreement interface output after the CDR file preliminary treatment, the array of the heavy record field of key row of conversation ticket time, be kept in the internal memory, filter heavy single with the filter form, when a ticket is not found heavy single time of its correspondence, then self row of this ticket heavily copy add in this filter, be used to prepare the heavy single of eliminating this ticket afterwards; The length of filter is a unique time period of configuration; When by the air time of row's hard words list within this length, then arrange heavily and handle, if not within this length, then enter specific channel; Directly arrange retry at internal memory.
2, by the described ticket row internal memory rearrangement based on filter of claim 1, the ticket writing file that it is characterized in that conversing is arranged by the air time order.
3, by the described ticket row internal memory rearrangement of claim 1 based on filter, the scope that it is characterized in that filter net sheet is: carry out the parameter setting, performance according to machine is made amendment, be traditionally arranged to be to two day of current time, for the CDR file in this time period not, scheduling virtual memory or be stored in data in the disk changes to internal memory and arranges heavily and handle.
4, by the described ticket row internal memory rearrangement of claim 1 based on filter, it is characterized in that this method is present in application platform as configurable application module independently, the heavy output of internal memory row is imported, accepted to overall applicability by the mode of file or memory interface.
5, by the described ticket row internal memory rearrangement of claim 1, it is characterized in that supporting carrying out the configuration that multi-channel parallel is handled according to concrete the application based on filter.
6,, it is characterized in that internal memory row is heavily by being the setting of the mechanism of unit by file by the described ticket row internal memory rearrangement based on filter of claim 1.
7, by the described ticket row internal memory rearrangement based on filter of claim 1, when it is characterized in that the heavy passage of the row of configuration need same switch under ticket must be configured in and arrange weight below the same passage.
8, by the described ticket row internal memory rearrangement of claim 1 based on filter, it is characterized in that internal memory row is heavily by being the setting of the mechanism of unit by file, internal memory row be heavily by can dynamically arranging by the heavy key element of row, can dispose different filters according to the difference of each service needed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200610038375 CN1809108A (en) | 2006-02-20 | 2006-02-20 | Filter based call ticket memory repetition elimination method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200610038375 CN1809108A (en) | 2006-02-20 | 2006-02-20 | Filter based call ticket memory repetition elimination method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN1809108A true CN1809108A (en) | 2006-07-26 |
Family
ID=36840784
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 200610038375 Pending CN1809108A (en) | 2006-02-20 | 2006-02-20 | Filter based call ticket memory repetition elimination method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN1809108A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101350869B (en) * | 2007-07-19 | 2011-08-24 | 中国电信股份有限公司 | Method and apparatus for removing repeat of telecom charging based on index and hash |
| CN102169499A (en) * | 2011-04-18 | 2011-08-31 | 北京神州数码思特奇信息技术股份有限公司 | Repeated ticket removing method |
| CN102591946A (en) * | 2010-12-28 | 2012-07-18 | 微软公司 | Using index partitioning and reconciliation for data deduplication |
| CN105225192A (en) * | 2015-09-09 | 2016-01-06 | 北京思特奇信息技术股份有限公司 | A kind of based on detailed reconciliation auditing method and system |
| CN105930396A (en) * | 2016-04-15 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Database based duplicate removal method and system |
| CN106599326A (en) * | 2017-01-23 | 2017-04-26 | 北京思特奇信息技术股份有限公司 | Duplication eliminating method and system for recorded data under cloud architecture |
| CN112069510A (en) * | 2020-07-24 | 2020-12-11 | 北京思特奇信息技术股份有限公司 | Data encryption and de-duplication method |
| CN116599782A (en) * | 2023-06-05 | 2023-08-15 | 深圳众投互联信息技术有限公司 | Multi-dimensional voice call ticket data filtering system based on data analysis |
-
2006
- 2006-02-20 CN CN 200610038375 patent/CN1809108A/en active Pending
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101350869B (en) * | 2007-07-19 | 2011-08-24 | 中国电信股份有限公司 | Method and apparatus for removing repeat of telecom charging based on index and hash |
| US9785666B2 (en) | 2010-12-28 | 2017-10-10 | Microsoft Technology Licensing, Llc | Using index partitioning and reconciliation for data deduplication |
| CN102591946A (en) * | 2010-12-28 | 2012-07-18 | 微软公司 | Using index partitioning and reconciliation for data deduplication |
| CN102169499A (en) * | 2011-04-18 | 2011-08-31 | 北京神州数码思特奇信息技术股份有限公司 | Repeated ticket removing method |
| CN105225192A (en) * | 2015-09-09 | 2016-01-06 | 北京思特奇信息技术股份有限公司 | A kind of based on detailed reconciliation auditing method and system |
| CN105225192B (en) * | 2015-09-09 | 2018-10-19 | 北京思特奇信息技术股份有限公司 | One kind is based on detailed reconciliation auditing method and system |
| CN105930396A (en) * | 2016-04-15 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Database based duplicate removal method and system |
| CN105930396B (en) * | 2016-04-15 | 2019-04-09 | 北京思特奇信息技术股份有限公司 | A kind of repetition removing method and system based on database |
| CN106599326A (en) * | 2017-01-23 | 2017-04-26 | 北京思特奇信息技术股份有限公司 | Duplication eliminating method and system for recorded data under cloud architecture |
| CN106599326B (en) * | 2017-01-23 | 2020-02-04 | 北京思特奇信息技术股份有限公司 | Recorded data duplication eliminating processing method and system under cloud architecture |
| CN112069510A (en) * | 2020-07-24 | 2020-12-11 | 北京思特奇信息技术股份有限公司 | Data encryption and de-duplication method |
| CN112069510B (en) * | 2020-07-24 | 2024-01-30 | 北京思特奇信息技术股份有限公司 | Data encryption and duplication elimination method |
| CN116599782A (en) * | 2023-06-05 | 2023-08-15 | 深圳众投互联信息技术有限公司 | Multi-dimensional voice call ticket data filtering system based on data analysis |
| CN116599782B (en) * | 2023-06-05 | 2024-04-02 | 深圳众投互联信息技术有限公司 | Multi-dimensional voice call ticket data filtering system based on data analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11449562B2 (en) | Enterprise data processing | |
| CN1809108A (en) | Filter based call ticket memory repetition elimination method | |
| CN111459985B (en) | Identification information processing method and device | |
| US7831662B2 (en) | Multi-user database system and method for resource usage tracking | |
| CN105164674A (en) | Queries involving multiple databases and execution engines | |
| CN100578498C (en) | Data integral service system and method | |
| CN101251861B (en) | Method for loading and inquiring magnanimity data | |
| US20110004622A1 (en) | Method and apparatus for gathering and organizing information pertaining to an entity | |
| CN101937474A (en) | Mass data query method and device | |
| CN107193898B (en) | The inquiry sharing method and system of log data stream based on stepped multiplexing | |
| CN101086732A (en) | A high magnitude of data management method | |
| CN1959676A (en) | Apparatus and method for materialized query table journaling in a computer database system | |
| CN1859505B (en) | Bill checking system and method | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN109376142A (en) | Data migration method and terminal device | |
| CN1717686A (en) | Automatic data consolidation | |
| CN112650736B (en) | Data migration method and device | |
| CN103902592A (en) | Method and system for realizing analytic functions based on MapReduce | |
| CN103198146B (en) | Real-time event filtering method and real-time event filtering system oriented to network stream data | |
| CN110941757A (en) | Big data based policy information query pushing system and method | |
| CN107291938A (en) | Order Query System and method | |
| CN106844497A (en) | The check device and method of a kind of database code | |
| CN110502529B (en) | Data processing method, device, server and storage medium | |
| CN1897629B (en) | Mass toll-ticket fast cross rearrangement based on memory | |
| CN101431760A (en) | Method and system for implementing business report |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Assignee: LIAN Technology (Nanjing) Co., Ltd. Assignor: Linkage System Integration Co., Ltd. Contract fulfillment period: 2009.6.23 to 2027.8.30 contract change Contract record no.: 2009320001548 Denomination of invention: Filter based call ticket memory repetition elimination method License type: exclusive license Record date: 2009.8.17 |
|
| LIC | Patent licence contract for exploitation submitted for record |
Free format text: EXCLUSIVE LICENSE; TIME LIMIT OF IMPLEMENTING CONTACT: 2009.6.23 TO 2027.8.30; CHANGE OF CONTRACT Name of requester: LIANCHUANG SCIENCE ( NANJING ) CO., LTD. Effective date: 20090817 |
|
| ASS | Succession or assignment of patent right |
Owner name: LIANCHUANG SCIENCE ( NANJING ) CO., LTD. Free format text: FORMER OWNER: NANJING LIANCHUANG SCIENCE CO., LTD. Effective date: 20091211 |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20091211 Address after: 16 F, 12 Huai gate, Nanjing City, Jiangsu Province, China: 210013 Applicant after: LIAN Technology (Nanjing) Co., Ltd. Address before: 16 F, 12 Huai gate, Nanjing City, Jiangsu Province, China: 210013 Applicant before: Linkage System Integration Co., Ltd. |
|
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Open date: 20060726 |