CN104750703B - A kind of method and apparatus for improving data accuracy - Google Patents
A kind of method and apparatus for improving data accuracy Download PDFInfo
- Publication number
- CN104750703B CN104750703B CN201310738426.2A CN201310738426A CN104750703B CN 104750703 B CN104750703 B CN 104750703B CN 201310738426 A CN201310738426 A CN 201310738426A CN 104750703 B CN104750703 B CN 104750703B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- dealing
- variance
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention discloses it is a kind of improve data accuracy method, including：Data generation text file is exported from memory database, the text file of generation is imported into physical database backup table, data in physical database backup table are once compared with corresponding data in physical database, obtains the first variance data after primary compare；After delay time threshold value, the corresponding data of first variance data are read from memory database and physical database respectively and carry out secondary comparison, obtain the second variance data after secondary comparison；When the sum of second variance data finally performs non-setting period time no more than alarm threshold and dealing of abnormal data program, start dealing of abnormal data program.The present invention further simultaneously discloses a kind of device for improving data accuracy.
Description
Technical field
The present invention relates to database management technology more particularly to a kind of method and apparatus for improving data accuracy.
Background technology
Business support system is that foundation user handles or the system using telecommunication service to user's charging, the system include：
Service handling module, memory database, dealing of abnormal data program, physical database, SMS transmission module etc.；The reality of the system
Existing flow includes：Service handling module receives the data from transacting business terminal and network element and receives number according to set service interfacing
After rule process is carried out, data are written in memory database and physical database simultaneously；In memory database and physics number
After completing read-write operation according to library, other processes that data needed for business are submitted to service handling module continue at next link
Reason.
In the prior art, business support system, which is completed to handle business, needs multiple application processes serially to perform, when after
During one stepping journey operation exception, the data being written from back process into memory database are unable to rollback, in this way, in will making
Deposit data library constantly accumulates wrong data, so as to reduce the data accuracy of memory database.Also, due in memory database
The service handling module that the data of mistake can be still submitted in business support system continues with, in this way, normally resulting in two
Kind consequence, process exception exits or process is ossified.Obviously, the wrong data of memory database can be to the stabilization of business support system
Property causes serious threat.
Invention content
In view of this, the main purpose of the embodiment of the present invention is to provide a kind of method and apparatus for improving data accuracy,
It can effectively solve the problems, such as to cause to retain wrong data, Jin Erti in the prior art due to the data of memory database are unable to rollback
The accuracy of data in high memory database.
In order to achieve the above objectives, the technical proposal of the invention is realized in this way：
An embodiment of the present invention provides a kind of method for improving data accuracy, this method includes：
Data generation text file is exported from memory database, the text file of generation is imported into physical data library backup
Table is once compared data in physical database backup table with corresponding data in physical database, after obtaining primary compare
The first variance data；
After delay time threshold value, first variance data is read from memory database and physical database respectively and is corresponded to
Data carry out secondary comparison, obtain the second variance data after secondary comparison；
When the sum for determining second variance data is finally performed no more than alarm threshold and dealing of abnormal data program
Time it is non-setting the period when, start dealing of abnormal data program.
In said program, the method further includes：
When the sum for determining second variance data is more than alarm threshold, warning information is sent out, user is notified to handle；
When the sum for determining second variance data is not more than alarm threshold, and dealing of abnormal data program finally performs
When time is the setting period, warning information is sent out, user is notified to handle.
In said program, the alarm threshold is nearly three months memory databases data exception caused by being unable to rollback,
The average value of daily abnormal data number to be treated.
In said program, the time threshold is that service handling module writes an internal storage data simultaneously in business support system
Maximum duration needed for library and physical database.
The embodiment of the present invention additionally provides a kind of device for improving data accuracy, which includes：Comparison process module,
Time delay module, judging treatmenting module；Wherein,
The comparison process module, for exporting data generation text file from memory database, by the text of generation
File imports physical database backup table；Corresponding data in data in physical database backup table and physical database is carried out one
Secondary comparison obtains the first variance data after primary compare；After delay time threshold value, respectively from memory database and physical data
The corresponding data of first variance data are read in library and carry out secondary comparison, obtain the second variance data after secondary comparison；
The time delay module, for delay time threshold value；
The judging treatmenting module, for determining the sum of second variance data no more than alarm threshold and abnormal number
Finally performed according to processing routine the time it is non-setting the period when, start dealing of abnormal data program.
In said program, the judging treatmenting module includes judgment module, dealing of abnormal data module；Wherein,
The judgment module, at determining the sum of second variance data no more than alarm threshold and abnormal data
Reason program finally perform the time it is non-setting the period when, trigger dealing of abnormal data module；
The dealing of abnormal data module, for starting dealing of abnormal data program.
In said program, the judgment module is additionally operable to：
When determine second variance data sum be more than alarm threshold, send out warning information, user notified to handle；
When the sum for determining second variance data is not more than alarm threshold, and dealing of abnormal data program finally performs
When time is the setting period, warning information is sent out, user is notified to handle.
In said program, the judging treatmenting module further includes counting module, for recording the alarm threshold；The announcement
Alert threshold value is nearly three months memory databases data exception caused by being unable to rollback, daily abnormal data number to be treated
Average value.
The method and apparatus of raising data accuracy that the embodiment of the present invention is provided, to memory database and physical data
Data in library carry out primary comparison and secondary comparison, obtain the variance data after secondary comparison；It is total when the variance data
When number finally performs non-setting period time no more than alarm threshold and dealing of abnormal data program, start dealing of abnormal data
Program.In this way, it can effectively solve to cause to retain wrong data in the prior art due to the data of memory database are unable to rollback
Problem, it is achieved thereby that in daily automatic inspection and processing memory database wrong data function, and then improve memory number
According to the accuracy of data in library.
Description of the drawings
Fig. 1 is that the method that the embodiment of the present invention improves data accuracy realizes flow diagram；
Fig. 2 is the composition structure diagram for the device that the embodiment of the present invention improves data accuracy.
Specific embodiment
In embodiments of the present invention：Data generation text file is exported from memory database, by the text file of generation
Physical database backup table is imported, data in physical database backup table are once compared with corresponding data in physical database
It is right, obtain the first variance data after primary compare；After delay time threshold value, respectively from memory database and physical database
It reads the corresponding data of first variance data and carries out secondary comparison, obtain the second variance data after secondary comparison；Work as institute
When stating the sum of variance data and finally performing non-setting period time no more than alarm threshold and dealing of abnormal data program, open
Dynamic dealing of abnormal data program.
Here, the primary comparison and secondary comparison can be that timing carries out, and such as setting compares the period；Can also be in industry
Business carries out when accepting module processor operation exception.
Wherein, first variance data, the second variance data refer both to physical database and internal storage data after comparison
There is which data to have differences in library；The timing progress can be that setting compares the period, and period length can be according to practical need
Setting is asked, such as：It is set as two hours, four hours etc..
Here, the setting period can be adjusted as needed, such as：When high to data alignment requirements, it can incite somebody to action
The setting period is taken as nearest 8 hours or 12 hours；When not high to data alignment requirements, it is small the setting period can be taken as nearest 24
When, 2 days or 3 days.
Here, the startup dealing of abnormal data program is：It is parameter according to type of service and major key, from physical database
The correct data corresponding to variance data after the middle secondary comparison of extraction, replaces the error number in corresponding memory database
According to；Wherein, the major key is the identification field for distinguishing different data in database.
Below in conjunction with the accompanying drawings and specific embodiment the present invention is further described in more detail.
Fig. 1 is that the method that the embodiment of the present invention improves data accuracy realizes flow diagram, as shown in Figure 1, of the invention
The method that embodiment improves data accuracy includes：
Step S100：Data generation text file is exported from memory database, the text file of generation is imported into physics
DB Backup table is once compared the data in physical database backup table and physical database corresponding data, is obtained
The first variance data after primary comparison；
Here, the export data generation text file includes：Can be parameter according to type of service and time, from memory
Data are exported in database, generate multiple text files；Wherein, type of service is opened an account including user, is transferred ownership, preferential handle, becomes
More title, application shutdown etc..
Specifically, according to the default rule of corresponding field and form between physical database data and memory database data
The text file of generation is imported into physical database backup table.
Further, before the text file generated is imported physical database backup table, physical database need to be emptied
Backup table.
Here, using type of service and major key as parameter, by the data in physical database backup table and physical database pair
Data is answered to be compared one by one, determine the first variance data.
Step S101：After delay time threshold value, the first difference number is read from memory database and physical database respectively
Secondary comparison is carried out according to corresponding data, obtains the second variance data after secondary comparison；
Here, the time threshold writes a memory database and object simultaneously for service handling module in business support system
Manage the maximum duration needed for database.
Correspondingly, the corresponding data of the first variance data are read from memory database and physical database and carry out secondary ratio
It is right, data correspondence is also carried out as the specific tables of data of parameter selection using type of service and major key.
Here, the primary comparison in step S100 and step S101 and secondary comparison, specific comparison process uses can
The multiple threads mechanism of configuration, and Thread Count is divided according to type of service and business datum amount, to ensure comparison process
Treatment effeciency.
Step S102：Step S101 the second variance datas obtained are judged, when determining second variance data
Sum when finally performing non-setting period time no more than alarm threshold and dealing of abnormal data program, start at abnormal data
Manage program；
Here, dealing of abnormal data program is specially：Using type of service as parameter, extracted from physical database and the
Correct data corresponding to two variance datas replaces wrong data in corresponding memory database.
Further, the method for the embodiment of the present invention further includes：
When the sum for determining second variance data is more than alarm threshold, warning information is sent out, user is notified to handle；
Wherein, the warning information can include：The record number of second variance data and comparison variance data result generated time.
When the sum for determining second variance data is not more than alarm threshold, and dealing of abnormal data program finally performs
When time is the setting period, warning information is sent out, user is notified to handle；Wherein, the warning information can include：It is currently processed
Time, last processing time and the setting time segment limit.
Wherein, alarm threshold is nearly three months memory databases data exception caused by being unable to rollback, needs to locate daily
The average value of the abnormal data number of reason.
Fig. 2 is the device composition structure diagram that the embodiment of the present invention improves data accuracy, as shown in Fig. 2, of the invention
The device that embodiment improves data accuracy includes：Comparison process module 20, time delay module 22, judging treatmenting module 21；Wherein,
Comparison process module 20, for exporting data generation text file from memory database, by the text text of generation
Part imports physical database backup table, corresponding data in data in physical database backup table and physical database is carried out primary
It compares, obtains the first variance data after primary compare；After delay time threshold value, respectively from memory database and physical database
The middle corresponding data of first variance data that read carry out secondary comparison, obtain the second variance data after secondary comparison；
Here, before the text file generated is imported physical database backup table, comparison process module 20 is additionally operable to
Empty physical database backup table.
Specifically, comparison process module 20 according to corresponding field between physical database data and memory database data with
The text file of generation is imported physical database backup table by the default rule of form.
The time delay module 22, for delay time threshold value；
Here, the time threshold writes a memory database and object simultaneously for service handling module in business support system
Manage the maximum duration needed for database.
Judging treatmenting module 21, for being not more than alarm threshold and abnormal number when the sum for determining second variance data
Finally performed according to processing routine the time it is non-setting the period when, start dealing of abnormal data program.
Here, the judging treatmenting module includes judgment module 211, dealing of abnormal data module 212；Wherein,
The judgment module 211, for determining the sum of second variance data no more than alarm threshold and abnormal number
When non-setting period time is finally performed according to processing routine, triggering dealing of abnormal data module 212；
The dealing of abnormal data module 212, for starting dealing of abnormal data program.
Here, the startup of dealing of abnormal data module 212 dealing of abnormal data program is specially：Using type of service and major key as
Parameter extracts the correct data corresponding to the second variance data from physical database, replaces corresponding internal storage data
Wrong data in library.
Further, judgment module 211 is additionally operable to：
When the sum for determining second variance data is more than alarm threshold, alarm message is sent out, user is notified to handle；
Wherein, the warning information can include：The record number of second variance data and comparison variance data result generated time.
When the sum for determining second variance data is not more than alarm threshold, and dealing of abnormal data program finally performs
When time is the setting period, warning information is sent out, user is notified to handle；Wherein, the warning information can include：It is currently processed
Time, last processing time and the setting time segment limit.
Here, judging treatmenting module 21 further includes counting module 210, for recording the alarm threshold；The alarm threshold
Be worth for nearly three months memory databases data exception caused by being unable to rollback, daily abnormal data number to be treated it is flat
Mean value.
In practical applications, the comparison process module 20, judgment module 211 and dealing of abnormal data module 212 are equal
It can be by being located at the central processing unit of terminal（CPU）, microprocessor（MPU）, digital signal processor（DSP）Or field-programmable
Gate array（FPGA）It realizes；The time delay module 22 and counting module 210 respectively can be real by memory, delayer, counter
It is existing.
The foregoing is only a preferred embodiment of the present invention, is not intended to limit the scope of the present invention.
Claims (7)
- A kind of 1. method for improving data accuracy, which is characterized in that the method includes：Data generation text file is exported from memory database, the text file of generation is imported into physical database backup table, Data in physical database backup table with corresponding data in physical database are once compared, obtain the after primary compare One variance data；After delay time threshold value, the corresponding number of first variance data is read from memory database and physical database respectively According to secondary comparison is carried out, the second variance data after secondary comparison is obtained；The time threshold is that service handling module writes a memory database and physical database simultaneously in business support system Required maximum duration；When the sum for determining second variance data finally performs the time no more than alarm threshold and dealing of abnormal data program During the non-setting period, start dealing of abnormal data program；The time span of the setting period requires to be configured according to data calibration.
- 2. according to the method described in claim 1, it is characterized in that, the method further includes：When the sum for determining second variance data is more than alarm threshold, warning information is sent out, user is notified to handle；When the sum for determining second variance data is not more than alarm threshold, and dealing of abnormal data program finally performs the time During for the setting period, warning information is sent out, user is notified to handle.
- 3. method according to claim 1 or 2, which is characterized in that the alarm threshold is nearly three months memory databases The data exception caused by being unable to rollback, the average value of daily abnormal data number to be treated.
- 4. a kind of device for improving data accuracy, which is characterized in that described device includes：Comparison process module, time delay module, Judging treatmenting module；Wherein,For exporting data generation text file from memory database, the text file of generation is imported for comparison process module Physical database backup table；Data in physical database backup table are once compared with corresponding data in physical database, Obtain the first variance data after primary compare；After delay time threshold value, read from memory database and physical database respectively The corresponding data of first variance data is taken to carry out secondary comparison, obtain the second variance data after secondary comparison；Time delay module, for delay time threshold value；The time threshold is that service handling module is write simultaneously in business support system Maximum duration needed for memory database and physical database；Judging treatmenting module, for determining that the sum of second variance data is not more than alarm threshold and dealing of abnormal data journey Sequence finally perform the time it is non-setting the period when, start dealing of abnormal data program；The time span of the setting period is according to number It is configured according to alignment requirements.
- 5. device according to claim 4, which is characterized in that the judging treatmenting module includes judgment module, abnormal number According to processing module；Wherein,The judgment module, for determining that the sum of second variance data is not more than alarm threshold and dealing of abnormal data journey Sequence finally perform the time it is non-setting the period when, trigger dealing of abnormal data module；The dealing of abnormal data module, for starting dealing of abnormal data program.
- 6. device according to claim 5, which is characterized in that the judgment module is additionally operable to：When determine second variance data sum be more than alarm threshold, send out warning information, user notified to handle；When the sum for determining second variance data is not more than alarm threshold, and dealing of abnormal data program finally performs the time During for the setting period, warning information is sent out, user is notified to handle.
- 7. according to claim 4 to 6 any one of them device, which is characterized in that the judging treatmenting module further includes counting Module, for recording the alarm threshold；Alarm threshold number caused by being unable to rollback for nearly three months memory databases According to exception, the average value of daily abnormal data number to be treated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310738426.2A CN104750703B (en) | 2013-12-26 | 2013-12-26 | A kind of method and apparatus for improving data accuracy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310738426.2A CN104750703B (en) | 2013-12-26 | 2013-12-26 | A kind of method and apparatus for improving data accuracy |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104750703A CN104750703A (en) | 2015-07-01 |
| CN104750703B true CN104750703B (en) | 2018-06-26 |
Family
ID=53590408
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310738426.2A Active CN104750703B (en) | 2013-12-26 | 2013-12-26 | A kind of method and apparatus for improving data accuracy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104750703B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108647936B (en) * | 2018-05-08 | 2022-07-08 | 中国平安人寿保险股份有限公司 | Commission issuing method, apparatus, computer device and storage medium |
| CN108805098A (en) * | 2018-06-21 | 2018-11-13 | 云城（北京）数据科技有限公司 | The comparison method of paper document and electronic document, device and system |
| CN109284301A (en) * | 2018-08-31 | 2019-01-29 | 阿里巴巴集团控股有限公司 | Verification of data method and device |
| CN109146427A (en) * | 2018-08-31 | 2019-01-04 | 万翼科技有限公司 | Mail communication method, device and the computer readable storage medium of calibration |
| CN110175090A (en) * | 2019-05-29 | 2019-08-27 | 数字中融信息技术河北有限公司 | A kind of data managing method handled with comparison and be capable of backup and reduction |
| CN110287200B (en) * | 2019-07-02 | 2022-03-25 | 贵阳货车帮科技有限公司 | Account checking method, account checking system, computer equipment and storage medium |
| CN111125008B (en) * | 2019-12-20 | 2022-04-22 | 浪潮电子信息产业股份有限公司 | Dynamic repairing method and system of abnormal inode and related components |
| CN111324620A (en) * | 2020-02-18 | 2020-06-23 | 中国联合网络通信集团有限公司 | Data processing method, device and storage medium |
| CN111949728A (en) * | 2020-07-14 | 2020-11-17 | 北京思特奇信息技术股份有限公司 | Dynamic data difference comparison method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858745A (en) * | 2006-04-05 | 2006-11-08 | 华为技术有限公司 | Processing method for realizing consistency of internal storage data and data bank data service |
| CN101079683A (en) * | 2007-06-27 | 2007-11-28 | 中国移动通信集团四川有限公司 | Data consistency processing method |
| CN101998354A (en) * | 2009-08-24 | 2011-03-30 | 中国移动通信集团甘肃有限公司 | Method and device for checking user data consistency |
| KR20130008333A (en) * | 2011-07-12 | 2013-01-22 | (주)선재소프트 | Convergence database management system using database manager , and the method |
-
2013
- 2013-12-26 CN CN201310738426.2A patent/CN104750703B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858745A (en) * | 2006-04-05 | 2006-11-08 | 华为技术有限公司 | Processing method for realizing consistency of internal storage data and data bank data service |
| CN101079683A (en) * | 2007-06-27 | 2007-11-28 | 中国移动通信集团四川有限公司 | Data consistency processing method |
| CN101998354A (en) * | 2009-08-24 | 2011-03-30 | 中国移动通信集团甘肃有限公司 | Method and device for checking user data consistency |
| KR20130008333A (en) * | 2011-07-12 | 2013-01-22 | (주)선재소프트 | Convergence database management system using database manager , and the method |
Non-Patent Citations (1)
| Title |
|---|
| 基于HA结构的HLR系统容灾中心技术;景旭等;《计算机技术与发展》;20070731;第16卷(第7期);第2.2、2.3、3.3节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104750703A (en) | 2015-07-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104750703B (en) | A kind of method and apparatus for improving data accuracy | |
| CN109472566B (en) | Block dynamic generation method, device, computer equipment and storage medium | |
| WO2019062189A1 (en) | Electronic device, method and system for conducting data table filing processing, and storage medium | |
| CN112115026B (en) | Server cluster monitoring method and device, electronic equipment and readable storage medium | |
| CN109611815B (en) | Energy consumption alarm management method and device for gas steam boiler | |
| EP3796182A1 (en) | Data processing method and apparatus, and computer-readable storage medium | |
| WO2012048555A1 (en) | Method and device for importing data into database | |
| CN101676881A (en) | Log management server | |
| CN113141410B (en) | Dynamically adjusted QPS control method, system, device and storage medium | |
| CN103412929A (en) | Mass data storage method | |
| CN105022676B (en) | A kind of memory database redoes log the restoration methods and device of file | |
| CN111291022A (en) | Data storage system based on block chain | |
| CN107480283A (en) | Realize the method, apparatus and storage system of big data quick storage | |
| CN107766512B (en) | Log data storage method and log data storage system | |
| CN106294470A (en) | The method that real-time incremental log information based on cutting daily record reads | |
| CN110413492A (en) | Health evaluating method, electronic equipment and the computer program product of storage dish | |
| CN108038127A (en) | A kind of method, apparatus, terminal device and the storage medium of data synchronization | |
| CN109039695B (en) | Service fault processing method, device and equipment | |
| CN107315806B (en) | Embedded storage method and device based on file system | |
| CN109656945A (en) | A kind of information processing method | |
| CN106294630A (en) | Multimedia messages recommends method, device and multimedia system | |
| CN106599326B (en) | Recorded data duplication eliminating processing method and system under cloud architecture | |
| CN113765504B (en) | Universal dynamic delay jitter elimination method, device, terminal and storage medium | |
| CN107748711A (en) | Method, terminal device and the storage medium of Automatic Optimal Storm degree of parallelisms | |
| CN110244096B (en) | Method for automatically discovering and processing electric meter full code in electric energy metering system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |