CN108319694A - A kind of automatic cleaning method and device of historical data - Google Patents
A kind of automatic cleaning method and device of historical data Download PDFInfo
- Publication number
- CN108319694A CN108319694A CN201810104814.8A CN201810104814A CN108319694A CN 108319694 A CN108319694 A CN 108319694A CN 201810104814 A CN201810104814 A CN 201810104814A CN 108319694 A CN108319694 A CN 108319694A
- Authority
- CN
- China
- Prior art keywords
- data
- clearance
- main table
- cleaning
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The automatic cleaning method and device of a kind of historical data disclosed in the present application, data scrubbing master control parameter list, cleaning rule table, rule of consistency table and tables of data for clearance are read, every cleaning rule of the main table of current data for clearance, the current corresponding contingency table of the main table of data for clearance and the current main table of data for clearance is obtained；According to every cleaning rule of the current main table of data for clearance and the current main table of data for clearance, the relevant historical data of the current main table of data for clearance and the corresponding contingency table of the current main table of data for clearance is cleared up.Data purge is carried out automatically by configurable cleaning rule, and ensures the data consistency between tables of data in the scale removal process of historical data.
Description
Technical field
The present invention relates to data scrubbing technical field, more particularly to a kind of automatic cleaning method of historical data and
Device.
Background technology
Business datum in computer system usually all can be accumulated constantly.If the excessive business historical data of scale is too late
Shi Qingli can occupy excessive disk space, can also influence system responsiveness energy.So to the history number more than certain time limit
It is very necessary according to being cleared up in time.
In existing application system, generally two methods is used to carry out data purge：
Method 1：For each tables of data for needing to carry out data scrubbing, SQL TRUNCATE or SQL are assigned by hand
DELETE SQL DROP CREATE TABLE orders, to carry out data scrubbing；
Method 2：Data scrubbing is realized with the method for programming.Defined in liquidation procedures and using database cursor, and from
Main table sets out, and to associated each tables of data, is deleted one by one record.
Shortcoming and defect existing for method 1：This method is difficult to ensure the consistency between tables of data, is easy error, causes
Data corruption.
Shortcoming and defect existing for method 2：Without configurability.It realizes dependent on programming, is specifically answered for each
With system, data scrubbing logic will be realized respectively；Without versatility and portability.Once needing to adjust data scrubbing rule
Then or table range is cleared up in increase, it is necessary to be changed program, therefore maintenance work becomes very complicated, is easy error.
Invention content
In view of this, the present invention provides a kind of automatic cleaning method of historical data and device, by configurable cleaning
Rule and rule of consistency, the automatic cleaning work for executing historical data, improve the cleaning efficiency of historical data.
In order to achieve the above-mentioned object of the invention, specific technical solution provided by the invention is as follows：
A kind of automatic cleaning method of historical data, including：
Data scrubbing master control parameter list, cleaning rule table, rule of consistency table and tables of data for clearance are read, is obtained current
Every of data main table, the currently corresponding contingency table of the main table of data for clearance and the current main table of data for clearance for clearance is clear
Reason rule；
According to every cleaning rule of the current main table of data for clearance and the current main table of data for clearance, successively to working as
The relevant historical data of the preceding corresponding contingency table of the main table of data for clearance and the current main table of data for clearance is cleared up.
Preferably, the reading data scrubbing master control parameter list, cleaning rule table, rule of consistency table and data for clearance
Table obtains the main table of current data for clearance, the current corresponding contingency table of the main table of data for clearance and current data master for clearance
Every cleaning rule of table, including：
Data scrubbing master control parameter list is read, data purge control parameter of overall importance is obtained, the historical data is clear
It includes that cleaning execution controls the mark that comes into force, the last cleaning date and cleaning frequency to manage control parameter of overall importance；
When the cleaning, which executes control, comes into force mark to come into force, according to current date, the last cleaning date
With cleaning frequency, judge currently whether need to clear up historical data；
If so, reading tables of data for clearance, the current main table of data for clearance is obtained, cleaning rule table is read, obtained current
Every cleaning rule of the main table of data for clearance, and rule of consistency table is read, it is corresponding to obtain the current main table of data for clearance
Contingency table.
Preferably, the basis currently advise by the main table of data for clearance and currently every cleaning of the main table of data for clearance
Then, successively to the relevant historical data of the current corresponding contingency table of the main table of data for clearance and the current main table of data for clearance
It is cleared up, including：
According to the tables of data for clearance and DDT data definition table, the major key of the current main table of data for clearance is determined；
According to the major key of the current main table of data for clearance, the SQL for deleting the current main table historical data of data for clearance is generated
Sentence；
According to the currently corresponding contingency table of the main table of data for clearance, generates delete the main table correspondence of current data for clearance respectively
Contingency table historical data SQL statement；
The SQL statement for deleting the corresponding contingency table historical data of the current main table of data for clearance is executed successively and is deleted current
The SQL statement of the main table historical data of data for clearance.
Preferably, the corresponding number of the corresponding each contingency table of the current main table of data for clearance, each number indicate
The incidence relation intensity of respective associated table and the current main table of data for clearance；
It is described to execute the SQL statement for deleting the corresponding contingency table historical data of the current main table of data for clearance and deletion successively
The SQL statement of the current main table historical data of data for clearance, specially：
It is executed successively from big to small according to the number and deletes the corresponding contingency table history number of the current main table of data for clearance
According to SQL statement, then execute the SQL statement for deleting the current main table historical data of data for clearance.
Preferably, the method further includes：
In the scale removal process of historical data, record cleaning data logging.
Preferably, the method further includes：
In the scale removal process of historical data, scale removal process daily record is recorded.
A kind of automatic cleaning apparatus of historical data, including：
Reading unit, for reading data scrubbing master control parameter list, cleaning rule table, rule of consistency table and number for clearance
According to table, the main table of current data for clearance, the current corresponding contingency table of the main table of data for clearance and current data for clearance are obtained
Every cleaning rule of main table；
Unit is cleared up, for every cleaning according to the current main table of data for clearance and the current main table of data for clearance
Rule, successively to the relevant historical number of the current corresponding contingency table of the main table of data for clearance and the current main table of data for clearance
According to being cleared up.
Preferably, the reading unit includes：
First reading subunit obtains data purge control of overall importance for reading data scrubbing master control parameter list
Parameter, the data purge control parameter of overall importance include that cleaning execution control comes into force mark, the last cleaning date
With cleaning frequency, primary cleaning date and cleaning frequency；
Judgment sub-unit, for when the cleaning execute control come into force mark for come into force when, according to current date, it is described most
Nearly primary cleaning date and cleaning frequency, judge currently whether need to clear up historical data；If so, son list is read in triggering second
Member；
Second reading subunit obtains the current main table of data for clearance, reads clear for reading tables of data for clearance
Rule list is managed, every cleaning rule of the current main table of data for clearance is obtained, and reads rule of consistency table, acquisition currently waits for clear
Manage the corresponding contingency table of the main table of data.
Preferably, the cleaning unit includes：
Determination subelement, for according to the tables of data for clearance and DDT data definition table, determining current data master for clearance
The major key of table；
First generates subelement, for the major key according to the current main table of data for clearance, generates and deletes current number for clearance
According to the SQL statement of main table historical data；
Second generates subelement, works as according to the currently corresponding contingency table of the main table of data for clearance, generating to delete respectively
The SQL statement of the corresponding contingency table historical data of the preceding main table of data for clearance；
Subelement is executed, the corresponding contingency table historical data of the current main table of data for clearance is deleted for executing successively
SQL statement and the SQL statement for deleting the current main table historical data of data for clearance.
Preferably, described device further includes：
Data log record unit is cleared up, in the scale removal process of historical data, record to clear up data logging.
Preferably, described device further includes：
Scale removal process logging unit, in the scale removal process of historical data, recording scale removal process daily record.
Compared with the existing technology, beneficial effects of the present invention are as follows：
The automatic cleaning method and device of a kind of historical data disclosed by the invention, reading data scrubbing master control parameter list,
Cleaning rule table, rule of consistency table and tables of data for clearance obtain the main table of current data for clearance, current data master for clearance
Every cleaning rule of the corresponding contingency table of table and the current main table of data for clearance；According to the main table of current data for clearance, with
And every cleaning rule of the current main table of data for clearance, to the current main table of data for clearance and current data master for clearance
The relevant historical data of the corresponding contingency table of table is cleared up.Data purge is carried out automatically by configurable cleaning rule,
And ensure the data consistency between tables of data in the scale removal process of historical data.
Description of the drawings
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of invention for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is a kind of automatic cleaning method flow chart of historical data disclosed by the embodiments of the present invention；
Fig. 2 is the automatic cleaning method flow chart of another historical data disclosed by the embodiments of the present invention；
Fig. 3 is data scrubbing master control parameter list schematic diagram disclosed by the embodiments of the present invention；
Fig. 4 is that rule of consistency disclosed by the embodiments of the present invention indicates to be intended to；
Fig. 5 is tables of data schematic diagram for clearance disclosed by the embodiments of the present invention；
Fig. 6 is that cleaning rule disclosed by the embodiments of the present invention indicates to be intended to；
Fig. 7 is that data table definitions disclosed by the embodiments of the present invention indicate to be intended to；
Fig. 8 is the automatic cleaning method flow chart of another historical data disclosed by the embodiments of the present invention；
Fig. 9 is that cleaning data logging disclosed by the embodiments of the present invention indicates to be intended to；
Figure 10 is scale removal process log sheet schematic diagram disclosed by the embodiments of the present invention；
Figure 11 is a kind of automatic cleaning apparatus structural schematic diagram of historical data disclosed by the embodiments of the present invention.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Referring to Fig. 1, present embodiment discloses a kind of automatic cleaning method of historical data, following steps are specifically included：
S101：Data scrubbing master control parameter list, cleaning rule table, rule of consistency table and tables of data for clearance are read, is obtained
To current data main table, the currently corresponding contingency table of the main table of data for clearance and the current main table of data for clearance for clearance
Every cleaning rule；
Referring to Fig. 2, a kind of preferred embodiments of S101 include：
S201：Data scrubbing master control parameter list is read, data purge control parameter of overall importance, the history number are obtained
Include that cleaning execution controls the mark that comes into force, the last cleaning date and cleaning frequency according to control parameter of overall importance is cleared up；
S202：When the cleaning, which executes control, comes into force mark to come into force, cleared up according to current date, described the last time
Date and cleaning frequency, judge currently whether need to clear up historical data；
If so, executing S203, S203：Tables of data for clearance is read, the current main table of data for clearance is obtained, reads cleaning rule
Then table, obtains every cleaning rule of the current main table of data for clearance, and reads rule of consistency table, obtains current number for clearance
According to the corresponding contingency table of main table；
S202 is executed if it is not, returning.
It should be noted that the data scrubbing master control parameter list include system banner, cleaning execute control come into force mark,
Consistency check whether by, whether record cleaning rule, whether record be cleaned data table structure definition, data whether
It has backed up, the last time clears up the date, cleaning frequency, each COMMIT records number, the modification date, modification time, changes correspondence
Accounting date, modification teller and mandate teller etc., are data scrubbing master control parameter list schematic diagram referring specifically to Fig. 3, Fig. 3.
Rule of consistency table defines the relevance between main table and contingency table, and data scrubbing is carried out to be directed to contingency table
When ensure tables of data between consistency.Rule of consistency table is to be generated in the process of development with form of scripts, under version
Hair, does not provide transaction maintenance function.
Main table refer to contingency table data to this table there are the tables of data of dependence, the external key of contingency table is the master of main table
Key.For example, illustrate so that simplest department, office worker, office worker check card the data model that three tables of record are formed as an example, for
Relationship between department table and staff table, then department table is main table, and staff table is contingency table.When deleting department table record, it is necessary to
Ensure that office worker's record has been not present under the department.
The principles and requirements of data consistency are：
1, the data between main table and contingency table are the data by the same major key+be keyed outside, if main table records
It is cleaned, then the data of contingency table also must while or be cleaned in advance；
If 2, there is also main table record also there must be the record of contingency table.
Rule of consistency table includes main table name, association table name, incidence relation, incidence relation packet class and incidence relation point
The fields such as serial number in group group are that rule of consistency indicates to be intended to referring specifically to Fig. 4, Fig. 4.
Referring to Fig. 5, Fig. 5 is tables of data schematic diagram for clearance.It has recorded whether to record in tables of data for clearance and clear up
Cheng Zhi, the field informations such as the data being cleaned whether are recorded.
The tables of data that can only be directed in " tables of data for clearance " defines its data scrubbing rule.Between same record
Condition is the relationship with (AND).Each table can define a plurality of regular record.It is between a plurality of record or the relationship of (OR).
Typically define cleaning rule for main table, the data scrubbing of contingency table is the data that are cleaned according to main table into
Row cleaning.The concrete structure of cleaning rule table please refers to Fig. 6.
S102：According to every cleaning rule of the current main table of data for clearance and the current main table of data for clearance, successively
The relevant historical data of the current corresponding contingency table of the main table of data for clearance and the current main table of data for clearance is carried out clear
Reason.
Referring to Fig. 2, a kind of preferred embodiments of S102 are as follows：
S204：According to the tables of data for clearance and DDT data definition table, the major key of the current main table of data for clearance is determined；
S205：According to the major key of the current main table of data for clearance, generates and delete the current main table historical data of data for clearance
SQL statement；
S206：According to the currently corresponding contingency table of the main table of data for clearance, generates delete current data master for clearance respectively
The SQL statement of the corresponding contingency table historical data of table；
S207：The SQL statement for deleting the corresponding contingency table historical data of the current main table of data for clearance is executed successively and is deleted
Except the SQL statement of the current main table historical data of data for clearance.
It should be noted that when each application system version updating is installed, changed in case of data list structure, Ying Jiang
The total data table definition write-in data table definitions table of latest edition.Referring to Fig. 7, having recorded corresponding version in data table definitions table
This number, main table name, data row name, type, length, whether major key, whether can the fields such as sky.
Preferably, when the corresponding each contingency table of the current main table of data for clearance corresponds to a number, each number table
Show the incidence relation intensity of respective associated table and the current main table of data for clearance；The current main table of data for clearance is deleted in the execution
The SQL statement of historical data and every SQL statement for deleting the corresponding contingency table historical data of the current main table of data for clearance, tool
Body is：
It is executed successively from big to small according to the number and deletes the corresponding contingency table history number of the current main table of data for clearance
According to SQL statement, then execute the SQL statement for deleting the current main table historical data of data for clearance.
It should be noted that the incidence relation of the bigger expression respective associated table of number and the current main table of data for clearance is more
It is weak.
A kind of automatic cleaning method of historical data disclosed in the present embodiment reads data scrubbing master control parameter list, cleaning
Rule list, rule of consistency table and tables of data for clearance obtain the main table of current data for clearance, the current main table pair of data for clearance
Every cleaning rule of the contingency table and the current main table of data for clearance answered；According to the currently main table of data for clearance and work as
Every cleaning rule of the preceding main table of data for clearance, to the current main table of data for clearance and the current main table pair of data for clearance
The relevant historical data for the contingency table answered is cleared up.Data purge is carried out automatically by configurable cleaning rule, and
Ensure the data consistency between tables of data in the scale removal process of historical data.
Referring to Fig. 8, present embodiment discloses the automatic cleaning method of another historical data, following step is specifically included
Suddenly：
S301：Data scrubbing master control parameter list, cleaning rule table, rule of consistency table and tables of data for clearance are read, is obtained
To current data main table, the currently corresponding contingency table of the main table of data for clearance and the current main table of data for clearance for clearance
Every cleaning rule；
S302：According to every cleaning rule of the current main table of data for clearance and the current main table of data for clearance, successively
The relevant historical data of the current corresponding contingency table of the main table of data for clearance and the current main table of data for clearance is carried out clear
Reason；
S303：In the scale removal process of historical data, record cleaning data logging；
It indicates to be intended to referring to Fig. 9, Fig. 9 is cleaning data logging, clears up in data logging table and have recorded cleaning data day
Will specifically includes the fields such as cleaning batch, data table name, cleaning date, clearance time, major key field title, Major key.
S304：In the scale removal process of historical data, scale removal process daily record is recorded.
Referring to Fig. 10, Figure 10 is scale removal process log sheet schematic diagram, scale removal process is had recorded in scale removal process log sheet
Daily record specifically includes cleaning batch, cleaning table name, cleaning date, data backup band mark, cleaning operation, cleaning record number, cleaning
The fields such as time started, cleaning end time.
The automatic cleaning method of historical data disclosed in the present embodiment, in the scale removal process of historical data, record cleaning
Data logging and scale removal process daily record, it is extensive in order to carry out backtracking and data according to cleaning data logging and scale removal process daily record
It is multiple.
Based on a kind of automatic cleaning method of historical data disclosed in above-described embodiment, 1 is please referred to Fig.1, the present embodiment pair
A kind of automatic cleaning apparatus of historical data should be disclosed, including：
Reading unit 401, for reading data scrubbing master control parameter list, cleaning rule table, rule of consistency table and waiting for clear
Tables of data is managed, the main table of current data for clearance, the corresponding contingency table of the current main table of data for clearance and current for clearance are obtained
Every cleaning rule of the main table of data；
Unit 402 is cleared up, every for the current main table of data for clearance of basis and the current main table of data for clearance is clear
Reason rule, successively to the relevant historical of the current corresponding contingency table of the main table of data for clearance and the current main table of data for clearance
Data are cleared up.
Preferably, the reading unit 401 includes：
First reading subunit obtains data purge control of overall importance for reading data scrubbing master control parameter list
Parameter, the data purge control parameter of overall importance include that cleaning execution control comes into force mark, the last cleaning date
With cleaning frequency；
Judgment sub-unit, for when the cleaning execute control come into force mark for come into force when, according to current date, it is described most
Nearly primary cleaning date and cleaning frequency, judge currently whether need to clear up historical data；If so, son list is read in triggering second
Member；
Second reading subunit obtains the current main table of data for clearance, reads clear for reading tables of data for clearance
Rule list is managed, every cleaning rule of the current main table of data for clearance is obtained, and reads rule of consistency table, acquisition currently waits for clear
Manage the corresponding contingency table of the main table of data.
Preferably, the cleaning unit 402 includes：
Determination subelement, for according to the tables of data for clearance and DDT data definition table, determining current data master for clearance
The major key of table；
First generates subelement, for the major key according to the current main table of data for clearance, generates and deletes current number for clearance
According to the SQL statement of main table historical data；
Second generates subelement, works as according to the currently corresponding contingency table of the main table of data for clearance, generating to delete respectively
The SQL statement of the corresponding contingency table historical data of the preceding main table of data for clearance；
Subelement is executed, the corresponding contingency table historical data of the current main table of data for clearance is deleted for executing successively
SQL statement and the SQL statement for deleting the current main table historical data of data for clearance.
Preferably, described device further includes：
Data log record unit is cleared up, in the scale removal process of historical data, record to clear up data logging.
Preferably, described device further includes：
Scale removal process logging unit, in the scale removal process of historical data, recording scale removal process daily record.
A kind of automatic cleaning apparatus of historical data disclosed in the present embodiment reads data scrubbing master control parameter list, cleaning
Rule list, rule of consistency table and tables of data for clearance obtain the main table of current data for clearance, the current main table pair of data for clearance
Every cleaning rule of the contingency table and the current main table of data for clearance answered；According to the currently main table of data for clearance and work as
Every cleaning rule of the preceding main table of data for clearance, to the current main table of data for clearance and the current main table pair of data for clearance
The relevant historical data for the contingency table answered is cleared up.Data purge is carried out automatically by configurable cleaning rule, and
Ensure the data consistency between tables of data in the scale removal process of historical data.
The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention.
Various modifications to these embodiments will be apparent to those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, of the invention
It is not intended to be limited to the embodiments shown herein, and is to fit to and the principles and novel features disclosed herein phase one
The widest range caused.
Claims (11)
1. a kind of automatic cleaning method of historical data, which is characterized in that including：
Data scrubbing master control parameter list, cleaning rule table, rule of consistency table and tables of data for clearance are read, is currently waited for clear
Manage every cleaning rule of the main table of data, the current corresponding contingency table of the main table of data for clearance and the current main table of data for clearance
Then；
According to every cleaning rule of the current main table of data for clearance and the current main table of data for clearance, successively to currently waiting for
The relevant historical data of the corresponding contingency table of the cleaning main table of data and the current main table of data for clearance is cleared up.
2. according to the method described in claim 1, it is characterized in that, the reading data scrubbing master control parameter list, cleaning rule
It is corresponding to obtain the main table of current data for clearance, the current main table of data for clearance for table, rule of consistency table and tables of data for clearance
Every cleaning rule of contingency table and the current main table of data for clearance, including：
Data scrubbing master control parameter list is read, data purge control parameter of overall importance is obtained, the data purge is complete
Office's property control parameter includes that cleaning execution controls the mark that comes into force, the last cleaning date and cleaning frequency；
When the cleaning, which executes control, comes into force mark to come into force, according to current date, the last cleaning date and clear
Frequency is managed, judges currently whether need to clear up historical data；
If so, reading tables of data for clearance, the current main table of data for clearance is obtained, reads cleaning rule table, acquisition currently waits for clear
Every cleaning rule of the main table of data is managed, and reads rule of consistency table, obtains the corresponding association of the current main table of data for clearance
Table.
3. according to the method described in claim 1, it is characterized in that, the current main table of data for clearance and current of the basis
Every cleaning rule of the main table of data for clearance is waited for the corresponding contingency table of the current main table of data for clearance and currently successively
The relevant historical data of the cleaning main table of data is cleared up, including：
According to the tables of data for clearance and DDT data definition table, the major key of the current main table of data for clearance is determined；
According to the major key of the current main table of data for clearance, the SQL statement for deleting the current main table historical data of data for clearance is generated；
According to the currently corresponding contingency table of the main table of data for clearance, generates delete the corresponding pass of the current main table of data for clearance respectively
Join the SQL statement of table historical data；
Executing deletion successively, currently the SQL statement of the corresponding contingency table historical data of the main table of data for clearance and deletion currently wait for clearly
Manage the SQL statement of the main table historical data of data.
4. according to the method described in claim 3, it is characterized in that, the corresponding each contingency table of the current main table of data for clearance
A corresponding number, each number indicate the incidence relation intensity of respective associated table and the current main table of data for clearance；
It is described to execute the SQL statement for deleting the corresponding contingency table historical data of the current main table of data for clearance successively and delete current
The SQL statement of the main table historical data of data for clearance, specially：
It is executed successively from big to small according to the number and deletes the corresponding contingency table historical data of the current main table of data for clearance
SQL statement, then execute the SQL statement for deleting the current main table historical data of data for clearance.
5. according to the method described in claim 1, it is characterized in that, the method further includes：
In the scale removal process of historical data, record cleaning data logging.
6. according to the method described in claim 1, it is characterized in that, the method further includes：
In the scale removal process of historical data, scale removal process daily record is recorded.
7. a kind of automatic cleaning apparatus of historical data, which is characterized in that including：
Reading unit, for reading data scrubbing master control parameter list, cleaning rule table, rule of consistency table and data for clearance
Table obtains the main table of current data for clearance, the current corresponding contingency table of the main table of data for clearance and current data master for clearance
Every cleaning rule of table；
Unit is cleared up, every cleaning rule according to the current main table of data for clearance and the current main table of data for clearance is used for,
The relevant historical data of the current corresponding contingency table of the main table of data for clearance and the current main table of data for clearance is carried out successively
Cleaning.
8. device according to claim 7, which is characterized in that the reading unit includes：
First reading subunit obtains data purge control parameter of overall importance for reading data scrubbing master control parameter list,
The data purge control parameter of overall importance includes that cleaning execution controls the mark that comes into force, the last cleaning date and cleaning
Frequency；
Judgment sub-unit, for when the cleaning execute control come into force mark for come into force when, according to current date, described nearest one
Secondary cleaning date and cleaning frequency, judge currently whether need to clear up historical data；If so, the second reading subunit of triggering；
Second reading subunit obtains the current main table of data for clearance, reads cleaning rule for reading tables of data for clearance
Then table, obtains every cleaning rule of the current main table of data for clearance, and reads rule of consistency table, obtains current number for clearance
According to the corresponding contingency table of main table.
9. device according to claim 7, which is characterized in that the cleaning unit includes：
Determination subelement, for according to the tables of data for clearance and DDT data definition table, determining the current main table of data for clearance
Major key；
First generates subelement, for the major key according to the current main table of data for clearance, generates and deletes current data master for clearance
The SQL statement of table historical data；
Second generates subelement, is currently waited for for according to the currently corresponding contingency table of the main table of data for clearance, generating to delete respectively
Clear up the SQL statement of the corresponding contingency table historical data of the main table of data；
Subelement is executed, for executing the SQL languages for deleting the corresponding contingency table historical data of the current main table of data for clearance successively
Sentence and the SQL statement for deleting the current main table historical data of data for clearance.
10. device according to claim 7, which is characterized in that described device further includes：
Data log record unit is cleared up, in the scale removal process of historical data, record to clear up data logging.
11. device according to claim 7, which is characterized in that described device further includes：
Scale removal process logging unit, in the scale removal process of historical data, recording scale removal process daily record.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810104814.8A CN108319694B (en) | 2018-02-02 | 2018-02-02 | Automatic historical data cleaning method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810104814.8A CN108319694B (en) | 2018-02-02 | 2018-02-02 | Automatic historical data cleaning method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108319694A true CN108319694A (en) | 2018-07-24 |
| CN108319694B CN108319694B (en) | 2023-01-20 |
Family
ID=62891542
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810104814.8A Active CN108319694B (en) | 2018-02-02 | 2018-02-02 | Automatic historical data cleaning method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108319694B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222039A (en) * | 2019-05-07 | 2019-09-10 | 平安科技（深圳）有限公司 | Data storage and garbage data cleaning method, device, equipment and storage medium |
| CN111597180A (en) * | 2020-05-19 | 2020-08-28 | 山东汇贸电子口岸有限公司 | Data cleaning method of OTRS system based on storage process |
| CN112559511A (en) * | 2021-02-25 | 2021-03-26 | 江苏苏宁银行股份有限公司 | Deposit system historical data cleaning method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090083341A1 (en) * | 2007-09-21 | 2009-03-26 | International Business Machines Corporation | Ensuring that the archival data deleted in relational source table is already stored in relational target table |
| CN103092631A (en) * | 2007-04-06 | 2013-05-08 | 西安万年科技实业有限公司 | Database application system development platform and development method |
| CN104036001A (en) * | 2014-06-13 | 2014-09-10 | 上海新炬网络技术有限公司 | Dynamic hotlist priority scheduling based quick data cleaning method |
-
2018
- 2018-02-02 CN CN201810104814.8A patent/CN108319694B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092631A (en) * | 2007-04-06 | 2013-05-08 | 西安万年科技实业有限公司 | Database application system development platform and development method |
| US20090083341A1 (en) * | 2007-09-21 | 2009-03-26 | International Business Machines Corporation | Ensuring that the archival data deleted in relational source table is already stored in relational target table |
| CN104036001A (en) * | 2014-06-13 | 2014-09-10 | 上海新炬网络技术有限公司 | Dynamic hotlist priority scheduling based quick data cleaning method |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222039A (en) * | 2019-05-07 | 2019-09-10 | 平安科技（深圳）有限公司 | Data storage and garbage data cleaning method, device, equipment and storage medium |
| CN110222039B (en) * | 2019-05-07 | 2023-09-29 | 平安科技（深圳）有限公司 | Data storage and garbage data cleaning method, device, equipment and storage medium |
| CN111597180A (en) * | 2020-05-19 | 2020-08-28 | 山东汇贸电子口岸有限公司 | Data cleaning method of OTRS system based on storage process |
| CN112559511A (en) * | 2021-02-25 | 2021-03-26 | 江苏苏宁银行股份有限公司 | Deposit system historical data cleaning method and system |
| CN112559511B (en) * | 2021-02-25 | 2021-06-01 | 江苏苏宁银行股份有限公司 | Deposit system historical data cleaning method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108319694B (en) | 2023-01-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106030533B (en) | It is executed by split process and retries affairs automatically | |
| CN108959400B (en) | Bank system historical data cleaning method and device | |
| JPH05173873A (en) | Data processor and storage management method | |
| US7698319B2 (en) | Database system management method, database system, database device, and backup program | |
| CN108319694A (en) | A kind of automatic cleaning method and device of historical data | |
| US8380663B2 (en) | Data integrity in a database environment through background synchronization | |
| JP2007188277A (en) | Storage control apparatus and data management method thereof | |
| CN103593449B (en) | A kind of database resource recovery method and system | |
| US9009175B2 (en) | System and method for database migration and validation | |
| JP6342070B2 (en) | Job management apparatus, job management method, and job management program | |
| CN109783315A (en) | A kind of database platform automation method for inspecting and system | |
| CN103970834A (en) | Recovery method for incremental data synchronization fault in isomerous database synchronizing system | |
| JP2011258122A (en) | Data transfer unit, data transfer method, data transfer program, and data cooperation system | |
| CN102902763A (en) | Method and device for relating and retrieving information processing data and processing information tasks | |
| CN105095255A (en) | Data index creating method and device | |
| CN104182436A (en) | Method and device for cleaning databases | |
| CN111221698A (en) | Task data acquisition method and device | |
| CN105373339B (en) | A kind of copy method and system of hard disc data | |
| US7266564B2 (en) | Control system and method for management items | |
| US8639657B2 (en) | Reorganizing table-based data objects | |
| CN104123104B (en) | Daily record control system and method | |
| AU2015265595B2 (en) | System and method for dynamic collection of system management data in a mainframe computing environment | |
| CN103279408A (en) | RAID (redundant array of inexpensive disk) performance testing method | |
| US20090063578A1 (en) | Management of data needed to resolve pointer errors in heirarchical database management systems | |
| CN102682038A (en) | Database change method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |