CN113190545A - Method, device, equipment and computer readable medium for cleaning data - Google Patents
Method, device, equipment and computer readable medium for cleaning data Download PDFInfo
- Publication number
- CN113190545A CN113190545A CN202110577795.2A CN202110577795A CN113190545A CN 113190545 A CN113190545 A CN 113190545A CN 202110577795 A CN202110577795 A CN 202110577795A CN 113190545 A CN113190545 A CN 113190545A
- Authority
- CN
- China
- Prior art keywords
- transaction
- preset
- storage period
- transaction detail
- date
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004140 cleaning Methods 0.000 title claims abstract description 86
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000003860 storage Methods 0.000 claims abstract description 167
- 238000012545 processing Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012216 screening Methods 0.000 claims description 9
- 230000002776 aggregation Effects 0.000 claims description 8
- 238000004220 aggregation Methods 0.000 claims description 8
- 230000007246 mechanism Effects 0.000 claims description 7
- 238000012360 testing method Methods 0.000 claims description 5
- 230000008520 organization Effects 0.000 claims description 2
- 230000004931 aggregating effect Effects 0.000 claims 1
- 238000004519 manufacturing process Methods 0.000 abstract description 13
- 238000010586 diagram Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- RWSOTUBLDIXVET-UHFFFAOYSA-N Dihydrogen sulfide Chemical compound S RWSOTUBLDIXVET-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000009530 blood pressure measurement Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000010926 purge Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Technology Law (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a method, a device, equipment and a computer readable medium for clearing data, and relates to the technical field of big data. One embodiment of the method comprises: deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account; storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period; and taking the stored transaction detail and the summarized transaction detail as the completion cleaning data for storage. The method and the device can automatically clear the transaction details of the private account so as to avoid the production risk.
Description
Technical Field
The present invention relates to the field of big data technologies, and in particular, to a method, an apparatus, a device, and a computer-readable medium for cleaning data.
Background
At present, no mechanism for clearing the transaction details of the private account exists in the production related to banking business. Wherein the private current account includes a private debit card and a private passbook.
Because the transaction details of the private account increase along with the frequency of network transaction, and the transaction details of the private account occupy excessive storage space, the storage space occupied by the transaction details needs to be cleared urgently.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: manual cleaning of the transaction details of the private account needs to occupy a large amount of working time of data center workers, and excessive manual intervention can cause production risks.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, a device, and a computer-readable medium for clearing data, which can automatically clear details of a private account transaction to avoid a production risk.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method of cleaning data, including:
deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account;
storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period;
and taking the stored transaction detail and the summarized transaction detail as the completion cleaning data for storage.
The deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account comprises the following steps:
screening out the logged-out user based on the transaction detail quantity and the type of the current account according to the preset cleaning date;
and deleting the transaction detail of the cancelled user in the transaction detail of the database to obtain the transaction detail of the registered user.
Screening out the cancelled users based on the transaction detail quantity and the type of the current account according to the preset cleaning date, wherein the screening comprises the following steps:
taking the current account established before the preset cleaning date as a primary logout user;
in a preset cleaning time period, the initially selected logout users with the transaction detail quantity smaller than a preset logout transaction detail quantity threshold value are used as quantity logout users;
taking the initially selected logout user of which the type of the current account belongs to the account seller as a type logout user;
the logged-off users comprise the number logged-off users and the type logged-off users.
The method further comprises the following steps:
and updating the preset cleaning date and/or the preset cleaning time period.
The storing the transaction details of the registered user in a preset storage period and summarizing the transaction details of the registered user outside the preset storage period includes:
dividing the transaction details of the registered user into the transaction details in the preset storage period and the transaction details outside the preset storage period according to the preset storage period and the current date;
and according to the registered user, storing the transaction details in the preset storage period, and summarizing the transaction details outside the preset storage period.
Dividing the transaction details of the registered user into the transaction details in the preset storage period and the transaction details outside the preset storage period according to the preset storage period and the current date, and the method comprises the following steps:
determining a division date according to the preset storage period and the current date;
and taking the transaction detail of the registered user after the division date as the transaction detail in the preset storage period, and taking the transaction detail of the registered user before the division date as the transaction detail outside the preset storage period.
The preset storage period is determined based on the transaction detail amount of the registered user.
The storing the transaction details in the preset storage period and the summarizing the transaction details outside the preset storage period according to the registered user comprises the following steps:
establishing a registered user index according to the identification of the registered user;
and storing the transaction details in the preset storage period according to the registered user index, and summarizing the transaction details outside the preset storage period.
The step of saving the transaction details in the preset storage period comprises the following steps:
and saving the transaction details in the preset storage period by taking the day as a unit.
The collecting the transaction details outside the preset storage period comprises:
establishing one or more transaction general details of the summary period according to the transaction details outside the preset storage period;
and storing the transaction detail, and establishing a corresponding relation between the transaction detail and the transaction detail outside the preset storage period so as to fetch the transaction detail outside the preset storage period.
The transaction summary includes one or more of a total expenditure amount, a total income amount, and a start-stop period of the aggregation period.
The storing the saved transaction detail and the summarized transaction detail as the completion cleaning data storage comprises the following steps:
and according to the mechanism to which the registered user belongs, taking the stored transaction detail and the summarized transaction detail as the completed clearing data for storage.
The storing the saved transaction detail and the summarized transaction detail as the completion cleaning data storage comprises the following steps:
and according to the preset processing date, storing the stored transaction detail and the summarized transaction detail as the completed cleaning data.
The preset processing date comprises one or more of a peak date, a rest date, a pressure test date, a non-working date and a rest date.
The method further comprises the following steps:
and clearing the historical transaction detail of the user according to the affiliated mechanism of the user and the number of the transaction detail.
The clearing of the historical transaction details of the user comprises:
and clearing the historical transaction detail of the user according to the preset maximum transaction detail.
The clearing of the historical transaction detail of the user according to the affiliated mechanism of the user and the number of the transaction detail comprises the following steps:
and according to the preset date for cleaning the historical transaction detail, cleaning the historical transaction detail of the user according to the affiliated organization and the number of the transaction detail of the user.
According to a second aspect of the embodiments of the present invention, there is provided an apparatus for cleaning data, including:
the deleting module is used for deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account;
the storage module is used for storing the transaction details of the registered user in a preset storage period and summarizing the transaction details of the registered user outside the preset storage period;
and the storage module is used for storing the saved transaction details and the summarized transaction details as the finished cleaning data.
According to a third aspect of the embodiments of the present invention, there is provided an electronic device for cleaning data, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method as described above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium, on which a computer program is stored, which when executed by a processor, implements the method as described above.
One embodiment of the above invention has the following advantages or benefits: deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account; storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period; and taking the stored transaction detail and the summarized transaction detail as the completion cleaning data for storage. The method can automatically clear the transaction details of the private account based on the preset parameters so as to avoid the production risk.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a method of cleaning data according to an embodiment of the invention;
FIG. 2 is a schematic flow diagram for obtaining transaction details for a registered user, according to an embodiment of the invention;
FIG. 3 is a flowchart illustrating screening of logged-out users according to an embodiment of the present invention;
FIG. 4 is a schematic flow diagram for processing transaction details for a registered user according to an embodiment of the invention;
FIG. 5 is a schematic flow diagram for partitioning transaction details of registered users, according to an embodiment of the invention;
FIG. 6 is a schematic flow chart illustrating the process of saving transaction details within a predetermined storage period and summarizing transaction details outside the predetermined storage period according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a main structure of an apparatus for cleaning data according to an embodiment of the present invention;
FIG. 8 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 9 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
At present, the detail of the private life bankbook is not cleared all the time in a long time. Although the private debit card details are cleared all the time, the database is still always at the edge of the alarm. It is expected that only billions of passbook details will be in urgent need of cleaning.
The amount of detail of the private account is too large, the detail is cleaned almost 20 days each month under the condition of considering non-functions, and particularly, manual intervention is needed to implement when various special dates such as website promotion, pressure measurement, production and the like are met, so that much workload is brought to a data center. And further, a large amount of working time of data center workers is occupied, and production risks exist due to excessive manual intervention.
In order to solve the technical problems that a large amount of working time of data center workers is occupied and production risks exist due to excessive manual intervention, the following technical scheme in the embodiment of the invention can be adopted.
Referring to fig. 1, fig. 1 is a schematic diagram of a main flow of a method for clearing data according to an embodiment of the present invention, which stores transaction details of a registered user in a preset storage period through preset parameters, and summarizes transaction details of the registered user outside the preset storage period. As shown in fig. 1, the method specifically comprises the following steps:
s101, deleting the transaction detail of the logged-out user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account.
The most effective method for cleaning data is to clean redundant private account details from the source and regularly clean the private account details daily. The details of the private account are controlled within a reasonable data range, the system pressure is reduced, and the risk is reduced.
The reason why the detail of the private passbook cannot be cleared by deleting the detail of the private debit card as well as the detail of the private debit card includes the following two aspects:
in a first aspect:
the private debit card details are moved down because of the inquiry, the host only keeps the details for 3 months, and the details exceeding 3 months are directly deleted. If history details are required, the move down component can be queried. But the detail of the private deposit book is not moved down, and all the detail is stored in the private deposit book.
In a second aspect:
the private account details are associated with the passbook. When a private account is traded, the customer has not yet completed the statement with the passbook. When the bankbook is subjected to registration next time, the registration history needs to be detailed. If the private account details are deleted directly, the registration details are discontinuous, and poor customer experience is brought.
In the process of implementing the embodiment of the invention, according to the analysis of the test environment on all the private account, a large part of data of the private account is data which is transferred to outside the business. These accounts are similar to those that account with private account data debits, and there is substantially less chance that customers will continue to transact on the counter. And in many scenarios, no account outside the business has a bankbook.
Then, in the embodiment of the present invention, the above-mentioned characteristics of the private account can be adopted to divide the user into a logged-out user and a registered user. Different processing modes are adopted for the transaction details of the logged-out user and the registered user.
The transaction detail of the cancelled user can be directly deleted; and storing the transaction details of the registered user according to the condition.
In one embodiment of the invention, the transaction detail of the registered user can be obtained by deleting the transaction detail of the cancelled user in the database based on the preset clearing date, the transaction detail quantity and the type of the current account.
Referring to fig. 2, fig. 2 is a schematic flowchart of obtaining transaction details of a registered user according to an embodiment of the present invention, which specifically includes the following steps:
s201, screening out the logged-out users according to a preset cleaning date and based on the transaction detail quantity and the type of the current account.
The preset cleaning date is the basis preset for screening the logged-out users. Generally, a current account established before a preset cleaning date has a high possibility of belonging to a logged-out user; accordingly, it is less likely that an account established alive after the preset cleanup date belongs to a logged-off user.
The transaction detail amount is the amount of transaction detail for the current account over a period of time. The more the transaction detail quantity is, the higher the possibility that the current account belongs to the logged-off user is; the smaller the number of transaction details, the less likely the current account will belong to a logged off user.
The type of current account is the type of current account in the banking system. According to the preset type, whether the current account belongs to the logged-off user can be judged.
Referring to fig. 3, fig. 3 is a schematic flowchart of a process of screening out a logged-out user according to an embodiment of the present invention, which specifically includes the following steps:
s301, taking a current account established before a preset cleaning date as a primary logout user.
In the embodiment of the invention, the preset cleaning date is used as the basis for determining the initial selection of the logout user. As an example, a current account established before a preset cleaning date is used as a primary logout user, and the preset cleaning date does not contain the current day of the preset cleaning date.
S302, in a preset cleaning time period, the initially selected logout users with the transaction detail quantity smaller than a preset logout transaction detail quantity threshold value are used as quantity logout users.
On the basis of initially selecting the logout users, different logout users can be screened out based on the transaction detail quantity and the types of current accounts.
The preset purge time period is a preset time period from the current date. As an example, the preset cleaning period is 6 months. Of course, the preset cleaning time period may be set in advance according to actual conditions. And subsequently, according to the actual scene of production, if the details in more time need to be kept, the method can be finished by adjusting the preset cleaning time period.
And in a preset cleaning time period, if the transaction detail quantity is less than a preset transaction detail quantity logout threshold value, the transaction of the user is not frequent, and the initially selected logout user with the transaction detail quantity less than the preset transaction detail quantity logout threshold value is used as a quantity logout user.
S303, the type of the current account belongs to the initial logout user of the seller, and the current account is used as the type logout user.
The types of the current account comprise a plurality of types, and if the type of the current account belongs to a sales account, the initially selected logout user is used as a type logout user.
It should be noted that the execution of S302 and S303 is not sequential, and may also be executed simultaneously.
S304, the logged-off users comprise number logged-off users and type logged-off users.
After determining the number and type of logged-off users, it is then determined which users have logged off. Logged-off users include both number logged-off users and type logged-off users.
In the embodiment of fig. 3, a number of logged-off users and types of logged-off users are determined based on the initial selection of logged-off users, and then the logged-off users are screened out.
And deleting the transaction detail of the cancelled user in the database, thereby effectively reducing invalid data in the database.
In one embodiment of the present invention, in order to adjust the manner of cleaning data in time, the preset cleaning date may be updated, and/or the preset cleaning time period may be updated.
S202, deleting the transaction detail of the logged-out user in the transaction detail of the database to obtain the transaction detail of the registered user.
The transaction details of all users are included in the transaction details of the database. After the user is confirmed to be logged off, the transaction detail of the user can be deleted, and the rest transaction detail is the transaction detail of the registered user.
And S102, storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period.
The transaction details of the registered user are data that needs to be saved. But the number of all transaction details of the registered user is larger, and the possibility that the recent transaction details are inquired is higher for the registered user; less likely will be the long-lived transaction detail queried.
Therefore, the preset storage period is used as the basis for processing the transaction details. A time period before the current date of the storage period is preset. As an example, the preset storage period is 6 months. That is, the current date is 7 months and 15 days, the preset storage period is 1 month and 15 days to 7 months and 15 days.
In one embodiment of the invention, the predetermined storage period is determined based on the number of transaction details of the registered user. As an example, during peak hours of a transaction for a registered user, such as: at the end of the year, the preset storage period is set to 3 months. During off-peak hours of registered users, such as: in the first half of the year, the preset storage period is set to 6 months.
And storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period.
Referring to fig. 4, fig. 4 is a schematic flowchart of processing transaction details of a registered user according to an embodiment of the present invention, which specifically includes the following steps:
s401, dividing the transaction details of the registered user into transaction details in a preset storage period and transaction details outside the preset storage period according to the preset storage period and the current date.
Firstly, according to a preset storage period and a current date, dividing the transaction details of the registered user into the transaction details in the preset storage period and the transaction details outside the preset storage period.
Referring to fig. 5, fig. 5 is a schematic flowchart of dividing transaction details of registered users according to an embodiment of the present invention, which specifically includes the following steps:
s501, determining a division date according to a preset storage period and the current date.
And taking the current date as the last day of the preset storage period, and further determining the division date based on the preset storage period. That is, the date is divided as the first day of the preset storage period, and the current date is the last day of the preset storage period.
S502, taking the transaction detail of the registered user after the division date as the transaction detail in the preset storage period, and taking the transaction detail of the registered user before the division date as the transaction detail outside the preset storage period.
After the division date is determined, the transaction detail is divided by the division date. Specifically, the transaction detail of the registered user after the division date is used as the transaction detail in the preset storage period, and the transaction detail of the registered user before the division date is used as the transaction detail outside the preset storage period.
In the embodiment of fig. 5, the transaction detail is divided by using the division date, which lays a foundation for further classifying and processing the transaction detail.
S402, according to the registered user, storing the transaction details in the preset storage period, and summarizing the transaction details outside the preset storage period.
For each registered user, there are corresponding transaction details within the preset storage period and transaction details outside the preset storage period. Then, the transaction details within the preset storage period may be saved and the transaction details outside the preset storage period may be summarized according to the registered user.
Referring to fig. 6, fig. 6 is a schematic flowchart of a process of saving transaction details in a preset storage period and summarizing transaction details outside the preset storage period according to an embodiment of the present invention, which specifically includes the following steps:
s601, establishing a registered user index according to the identification of the registered user.
The identification of each registered user is different, and for each registered user, the transaction details in the preset storage period need to be stored, and the transaction details outside the preset storage period need to be summarized.
And establishing a registered user index according to the identification of the registered user so as to distinguish different registered users.
And S602, storing the transaction details in the preset storage period by using the registered user index, and summarizing the transaction details outside the preset storage period.
And according to the registered user index, on one hand, storing the transaction details in the preset storage period, and on the other hand, summarizing the transaction details outside the preset storage period.
In one embodiment of the invention, the transaction details within the preset storage period are saved in units of days. That is, the transaction details of the registered user in the preset storage period are saved by day. Transaction details can be queried by date.
In an embodiment of the present invention, if the possibility that the transaction details outside the preset storage period are queried is low, an aggregation period may be established, and the transaction details are stored in the aggregation period. Wherein, the summary period can be preset. As one example, the aggregation period is 1 week.
Specifically, the transaction summary details of one or more summary periods are established according to the transaction details outside the preset storage period. As one example, the transaction summary for the aggregation period includes one or more of a total expenditure amount, a total income amount, and an aggregation period start-stop time.
And storing the transaction summary in a database, and establishing a corresponding relation between the transaction summary and the transaction detail outside the preset storage period so as to fetch the transaction detail outside the preset storage period.
Therefore, the transaction detail can be easily inquired, and when further demands exist, the transaction detail outside the preset storage period can be called based on the corresponding relation between the transaction detail and the transaction detail outside the preset storage period. As an example, the transaction details outside the preset storage period are stored in a server different from the transaction summary.
In the embodiment of fig. 6, the transaction details are stored in the registered user index, and the transaction details are summarized, so that the transaction details can be called in time.
And S103, storing the saved transaction detail and the summarized transaction detail as the completion cleaning data.
And storing the saved transaction detail and the summarized transaction detail as the completion cleaning data. In the embodiment of the invention, because the number of the subordinate registered users of different organizations is different, the stored transaction detail and the summarized transaction detail can be used as the clearing data storage according to the organizations to which the registered users belong. In this way, the data store is cleared according to the institution to which the registered user belongs.
As an example, data of subordinate registered users of Beijing division are cleaned. And clearing data of subordinate registered users of Tianjin division.
Even if private accounts are cleared every month, the amount of detail for a month is considerable given the hundreds of millions of private accounts. The working time is scheduled to be cleared about 20 days per month. For the monthly processing date of the private account, technicians are required to exclude the next month of commissioning and production environment arrangement, etc.
In one embodiment of the invention, the stored transaction detail and the aggregated transaction detail are stored as the completion clearing data according to a preset processing date. The preset processing date comprises one or more of a peak date, a rest date, a stress test date, a non-working date and a rest date.
In one embodiment of the invention, there are nearly billions of historical transaction details that need to be cleared on a daily monthly basis when private account details are cleared, but it is contemplated that if manually cleared, at least 3 to 4 technicians need to be scheduled for manual clearing and monitoring each time. And in the stage of clearing the historical transaction detail, clearing the historical transaction detail of the user according to the mechanism to which the user belongs and the number of the transaction detail.
Specifically, the historical transaction detail of the user is cleared according to the transaction detail with the preset maximum number. The cleaning purpose is achieved in limited resources and time on the basis of ensuring that the existing data is not influenced.
In one embodiment of the invention, considering that the cleaning history detail does not respond to normal operation of cleaning daily data as much as possible, the date of cleaning the history detail is preset, and the date of cleaning the history detail is preset to avoid the peak period of cleaning the daily data.
And cleaning the historical transaction detail of the user according to the preset date for cleaning the historical transaction detail and the number of the affiliated institutions and the transaction detail of the user.
In the embodiment, based on the preset cleaning date, the transaction detail quantity and the type of the current account, the transaction detail of the cancelled user in the database is deleted to obtain the transaction detail of the registered user; storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period; and taking the stored transaction detail and the summarized transaction detail as the completion cleaning data for storage. The method can automatically clear the transaction details of the private account based on the preset parameters so as to avoid the production risk.
Referring to fig. 7, fig. 7 is a schematic diagram of a main structure of an apparatus for cleaning data according to an embodiment of the present invention, where the apparatus for cleaning data may implement a method for cleaning data, and as shown in fig. 7, the apparatus for cleaning data specifically includes:
a deleting module 701, configured to delete the transaction details of the cancelled user in the database to obtain the transaction details of the registered user based on a preset clearing date, the transaction details number, and the type of the current account;
a saving module 702, configured to save the transaction details of the registered user in a preset storage period, and summarize the transaction details of the registered user outside the preset storage period;
and the storage module 703 is configured to store the saved transaction details and the summarized transaction details as the completed cleaning data.
In an embodiment of the present invention, the deleting module 701 is specifically configured to, according to the preset cleaning date, screen out the logged-out user based on the transaction detail amount and the type of the current account;
and deleting the transaction detail of the cancelled user in the transaction detail of the database to obtain the transaction detail of the registered user.
In an embodiment of the present invention, the deleting module 701 is specifically configured to use a current account established before the preset cleaning date as an initially logged-out user;
in a preset cleaning time period, the initially selected logout users with the transaction detail quantity smaller than a preset logout transaction detail quantity threshold value are used as quantity logout users;
taking the initially selected logout user of which the type of the current account belongs to the account seller as a type logout user;
the logged-off users comprise the number logged-off users and the type logged-off users.
In an embodiment of the present invention, the deleting module 701 is further configured to update the preset cleaning date and/or the preset cleaning time period.
In an embodiment of the present invention, the saving module 702 is specifically configured to divide the transaction details of the registered user into the transaction details in the preset storage period and the transaction details outside the preset storage period according to the preset storage period and the current date;
and according to the registered user, storing the transaction details in the preset storage period, and summarizing the transaction details outside the preset storage period.
In an embodiment of the present invention, the saving module 702 is specifically configured to determine a division date according to the preset storage period and the current date;
and taking the transaction detail of the registered user after the division date as the transaction detail in the preset storage period, and taking the transaction detail of the registered user before the division date as the transaction detail outside the preset storage period.
In one embodiment of the invention, the predetermined storage period is determined based on the transaction detail amount of the registered user.
In an embodiment of the present invention, the storing module 702 is specifically configured to establish a registered user index according to the identifier of the registered user;
and storing the transaction details in the preset storage period according to the registered user index, and summarizing the transaction details outside the preset storage period.
In an embodiment of the present invention, the saving module 702 is specifically configured to save the transaction details in the preset storage period by taking a day as a unit.
In an embodiment of the present invention, the saving module 702 is specifically configured to establish a total transaction detail of one or more summary periods according to the transaction details outside the preset storage period;
and storing the transaction detail, and establishing a corresponding relation between the transaction detail and the transaction detail outside the preset storage period so as to fetch the transaction detail outside the preset storage period.
In one embodiment of the invention, the transaction summary includes one or more of a total expenditure amount, a total income amount and a start-stop period of the aggregation period.
In an embodiment of the present invention, the storage module 703 is specifically configured to store the saved transaction details and the summarized transaction details as the completion clearing data according to the institution to which the registered user belongs.
In an embodiment of the present invention, the storage module 703 is specifically configured to store the stored transaction details and the summarized transaction details as the completion clearing data according to a preset processing date.
In one embodiment of the invention, the preset processing date comprises one or more of a detail production peak date, a rest date, a stress test date, a non-working date and a rest date.
In an embodiment of the present invention, the deleting module 701 is further configured to clear the historical transaction details of the user according to the affiliated institution of the user and the number of the transaction details.
In an embodiment of the present invention, the deleting module 701 is specifically configured to clear the historical transaction details of the user according to a preset maximum number of transaction details.
In an embodiment of the present invention, the deleting module 701 is specifically configured to clear the historical transaction details of the user according to a preset date for clearing the historical transaction details and the number of the institutions to which the user belongs and the number of the transaction details.
Fig. 8 shows an exemplary system architecture 800 of a method of cleaning data or an apparatus for cleaning data to which embodiments of the present invention may be applied.
As shown in fig. 8, the system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. The network 804 serves to provide a medium for communication links between the terminal devices 801, 802, 803 and the server 805. Network 804 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 801, 802, 803 to interact with a server 805 over a network 804 to receive or send messages or the like. The terminal devices 801, 802, 803 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 801, 802, 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 805 may be a server that provides various services, such as a back-office management server (for example only) that supports shopping-like websites browsed by users using the terminal devices 801, 802, 803. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the method for cleaning data provided by the embodiment of the present invention is generally performed by the server 805, and accordingly, the apparatus for cleaning data is generally disposed in the server 805.
It should be understood that the number of terminal devices, networks, and servers in fig. 8 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 9, shown is a block diagram of a computer system 900 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 9, the computer system 900 includes a Central Processing Unit (CPU)901 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the system 900 are also stored. The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.
The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The above-described functions defined in the system of the present invention are executed when the computer program is executed by a Central Processing Unit (CPU) 901.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a deletion module, a save module, and a storage module. Where the names of these modules do not in some cases constitute a limitation of the modules themselves, for example, a delete module may also be described as "for deleting a transaction detail of a logged-out user in a database to a transaction detail of a registered user based on a preset clearing date, a transaction detail amount, and a type of current account".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:
deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account;
storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period;
and taking the stored transaction detail and the summarized transaction detail as the completion cleaning data for storage.
According to the technical scheme of the embodiment of the invention, based on the preset cleaning date, the transaction detail quantity and the type of the current account, the transaction detail of the cancelled user in the database is deleted to obtain the transaction detail of the registered user; storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period; and taking the stored transaction detail and the summarized transaction detail as the completion cleaning data for storage. The method can automatically clear the transaction details of the private account based on the preset parameters so as to avoid the production risk.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (20)
1. A method of cleaning data, comprising:
deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account;
storing the transaction details of the registered user in a preset storage period, and summarizing the transaction details of the registered user outside the preset storage period;
and taking the stored transaction detail and the summarized transaction detail as the completion cleaning data for storage.
2. The method for clearing data according to claim 1, wherein deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account comprises:
screening out the logged-out user based on the transaction detail quantity and the type of the current account according to the preset cleaning date;
and deleting the transaction detail of the cancelled user in the transaction detail of the database to obtain the transaction detail of the registered user.
3. The method for clearing data according to claim 2, wherein screening out the cancelled users based on the transaction detail amount and the type of current account according to the preset clearing date comprises:
taking the current account established before the preset cleaning date as a primary logout user;
in a preset cleaning time period, the initially selected logout users with the transaction detail quantity smaller than a preset logout transaction detail quantity threshold value are used as quantity logout users;
taking the initially selected logout user of which the type of the current account belongs to the account seller as a type logout user;
the logged-off users comprise the number logged-off users and the type logged-off users.
4. A method for cleaning data according to claim 3, characterized in that the method further comprises:
and updating the preset cleaning date and/or the preset cleaning time period.
5. The method for clearing data according to claim 1, wherein the saving the transaction details of the registered user in a preset storage period and summarizing the transaction details of the registered user outside the preset storage period comprises:
dividing the transaction details of the registered user into the transaction details in the preset storage period and the transaction details outside the preset storage period according to the preset storage period and the current date;
and according to the registered user, storing the transaction details in the preset storage period, and summarizing the transaction details outside the preset storage period.
6. The method for clearing data according to claim 5, wherein said dividing the transaction details of the registered user into the transaction details within the preset storage period and the transaction details outside the preset storage period according to the preset storage period and the current date comprises:
determining a division date according to the preset storage period and the current date;
and taking the transaction detail of the registered user after the division date as the transaction detail in the preset storage period, and taking the transaction detail of the registered user before the division date as the transaction detail outside the preset storage period.
7. A method of cleansing data according to claim 1, wherein the predetermined storage period is determined based on a transaction detail amount of the registered user.
8. The method of claim 5, wherein the storing the transaction details within the preset storage period and the summarizing the transaction details outside the preset storage period according to the registered user comprises:
establishing a registered user index according to the identification of the registered user;
and storing the transaction details in the preset storage period according to the registered user index, and summarizing the transaction details outside the preset storage period.
9. The method for clearing data according to claim 5, wherein the saving transaction details in the preset storage period comprises:
and saving the transaction details in the preset storage period by taking the day as a unit.
10. The method of claim 5, wherein the aggregating transaction details outside the predetermined storage period comprises:
establishing one or more transaction general details of the summary period according to the transaction details outside the preset storage period;
and storing the transaction detail, and establishing a corresponding relation between the transaction detail and the transaction detail outside the preset storage period so as to fetch the transaction detail outside the preset storage period.
11. A method of cleaning data as claimed in claim 10, wherein the transaction summary includes one or more of a total expenditure amount, a total income amount and a start and stop time for the aggregation period.
12. The method for clearing data according to claim 1, wherein storing the saved transaction detail and the aggregated transaction detail as the clearing completion data comprises:
and according to the mechanism to which the registered user belongs, taking the stored transaction detail and the summarized transaction detail as the completed clearing data for storage.
13. The method for clearing data according to claim 1, wherein storing the saved transaction detail and the aggregated transaction detail as the clearing completion data comprises:
and according to the preset processing date, storing the stored transaction detail and the summarized transaction detail as the completed cleaning data.
14. The method of claim 13, wherein the predetermined processing dates include one or more of a peak date, a rest date, a stress test date, a non-work date, and a rest date.
15. The method of claim 1, further comprising:
and clearing the historical transaction detail of the user according to the affiliated mechanism of the user and the number of the transaction detail.
16. A method for cleansing data as claimed in claim 15, wherein said cleansing historical transaction details of said user comprises:
and clearing the historical transaction detail of the user according to the preset maximum transaction detail.
17. The method of claim 15, wherein the clearing the historical transaction details of the user based on the affiliates of the user and the number of transaction details comprises:
and according to the preset date for cleaning the historical transaction detail, cleaning the historical transaction detail of the user according to the affiliated organization and the number of the transaction detail of the user.
18. An apparatus for cleaning data, comprising:
the deleting module is used for deleting the transaction detail of the cancelled user in the database to obtain the transaction detail of the registered user based on the preset clearing date, the transaction detail quantity and the type of the current account;
the storage module is used for storing the transaction details of the registered user in a preset storage period and summarizing the transaction details of the registered user outside the preset storage period;
and the storage module is used for storing the saved transaction details and the summarized transaction details as the finished cleaning data.
19. An electronic device for cleaning data, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-17.
20. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-17.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110577795.2A CN113190545A (en) | 2021-05-26 | 2021-05-26 | Method, device, equipment and computer readable medium for cleaning data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110577795.2A CN113190545A (en) | 2021-05-26 | 2021-05-26 | Method, device, equipment and computer readable medium for cleaning data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113190545A true CN113190545A (en) | 2021-07-30 |
Family
ID=76985286
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110577795.2A Pending CN113190545A (en) | 2021-05-26 | 2021-05-26 | Method, device, equipment and computer readable medium for cleaning data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113190545A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060178807A1 (en) * | 2004-09-10 | 2006-08-10 | Xanavi Informatics Corporation | Apparatus and method for processing and displaying traffic information in an automotive navigation system |
| CN108874912A (en) * | 2018-05-29 | 2018-11-23 | 阿里巴巴集团控股有限公司 | A kind of cancellation method and server |
| CN110400211A (en) * | 2019-07-26 | 2019-11-01 | 中移电子商务有限公司 | A kind of user account fund management method, device and storage medium |
| CN110619006A (en) * | 2019-08-30 | 2019-12-27 | 上海曼斯克物联网科技有限公司 | Statistical data management method, device, platform and storage medium based on Internet of things |
| CN111681099A (en) * | 2020-06-03 | 2020-09-18 | 中国银行股份有限公司 | Product information pushing method and device, computer equipment and readable storage medium |
| CN112308565A (en) * | 2020-08-14 | 2021-02-02 | 中国工商银行股份有限公司 | Many-to-many cross-border fund wind control method and system based on knowledge graph |
-
2021
- 2021-05-26 CN CN202110577795.2A patent/CN113190545A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060178807A1 (en) * | 2004-09-10 | 2006-08-10 | Xanavi Informatics Corporation | Apparatus and method for processing and displaying traffic information in an automotive navigation system |
| CN108874912A (en) * | 2018-05-29 | 2018-11-23 | 阿里巴巴集团控股有限公司 | A kind of cancellation method and server |
| CN110400211A (en) * | 2019-07-26 | 2019-11-01 | 中移电子商务有限公司 | A kind of user account fund management method, device and storage medium |
| CN110619006A (en) * | 2019-08-30 | 2019-12-27 | 上海曼斯克物联网科技有限公司 | Statistical data management method, device, platform and storage medium based on Internet of things |
| CN111681099A (en) * | 2020-06-03 | 2020-09-18 | 中国银行股份有限公司 | Product information pushing method and device, computer equipment and readable storage medium |
| CN112308565A (en) * | 2020-08-14 | 2021-02-02 | 中国工商银行股份有限公司 | Many-to-many cross-border fund wind control method and system based on knowledge graph |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9971777B2 (en) | Smart archiving of real-time performance monitoring data | |
| CN109901918B (en) | Method and device for processing overtime task | |
| CN111275415A (en) | Resource channel switching method, device, equipment and storage medium | |
| CN111190868A (en) | File processing method and device | |
| CN111767127A (en) | Service data processing method and device | |
| CN110727563A (en) | Cloud service alarm method and device for preset customer | |
| CN111212112B (en) | Information processing method and device | |
| CN112102099A (en) | Policy data processing method and device, electronic equipment and storage medium | |
| CN111383098A (en) | Resource distribution method and device | |
| CN113190545A (en) | Method, device, equipment and computer readable medium for cleaning data | |
| CN110705981B (en) | Real-time settlement method and device | |
| CN111415071B (en) | Method and device for analyzing operation flow | |
| CN113360689A (en) | Image retrieval system, method, related device and computer program product | |
| CN111526184A (en) | Business auditing method and device | |
| CN113434754A (en) | Method and device for determining recommended API (application program interface) service, electronic equipment and storage medium | |
| CN114706826B (en) | Method, device, equipment and computer readable medium for cleaning file | |
| CN111290927A (en) | Data monitoring method and device | |
| CN103914514A (en) | Method and system for controlling business voucher output mode | |
| CN108984431A (en) | Method and apparatus for emptying expired caching | |
| CN114756540A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN110955519A (en) | Method and device for host distribution | |
| CN112148726A (en) | Method and device for deleting table | |
| CN115829701A (en) | Account association processing method and device | |
| CN115495316A (en) | Management method and device for historical page maintenance records | |
| CN115273316A (en) | Queuing and calling method and system for business place reservation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |