CN114756540A

CN114756540A - Data processing method and device, electronic equipment and computer readable medium

Info

Publication number: CN114756540A
Application number: CN202210466117.3A
Authority: CN
Inventors: 赵芳; 杨健; 何翔飞; 邱俊凌
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2022-04-29
Filing date: 2022-04-29
Publication date: 2022-07-15

Abstract

The application discloses a data processing method, a data processing device, electronic equipment and a computer readable medium, which relate to the technical field of big data processing, and the method comprises the following steps: receiving a data processing request, and acquiring a corresponding data processing type identifier, a corresponding demand identifier and data to be processed; combining the first cleaning parameters according to the data processing type identification and the requirement identification to obtain combined cleaning parameters, and further determining corresponding combined cleaning parameter values; determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value; respectively acquiring a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter; performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. Therefore, customized data backup and cleaning under the appointed date or the appointed frequency are realized, the occupation of the space of the database is reduced, and the performance of the database is improved.

Description

Data processing method and device, electronic equipment and computer readable medium

Technical Field

The present application relates to the field of big data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a computer-readable medium.

Background

With the increase of banking business, when more and more report data are generated by a banking system and the data volume of a corresponding database table is larger and larger as time is accumulated, the report data generated by the banking system needs to be backed up and cleaned in order to ensure the normal performance of the database system. In a bank system, general reports are divided into daily reports, ten-day reports, monthly reports, seasonal reports, annual reports and the like, the data of the reports are all from a database system, the data volume of the reports in the database is larger and larger along with the increase and long-time accumulation of business volume, the space of the database is seriously occupied, and the performance of the database is influenced.

In the process of implementing the present application, the inventor finds that at least the following problems exist in the prior art:

with the increase of business volume and long-time accumulation, the report data volume in the database is larger and larger, a mature backup mechanism and a cleaning strategy are not provided, the database space is seriously occupied, and the performance of the database is influenced.

Disclosure of Invention

In view of this, embodiments of the present application provide a data processing method, an apparatus, an electronic device, and a computer readable medium, which can solve the problems that the report data volume in a database is larger and larger, a mature backup mechanism and a cleaning policy are not available, the database space is seriously occupied, and the database performance is influenced as the traffic volume increases and accumulates for a long time.

To achieve the above object, according to an aspect of an embodiment of the present application, there is provided a data processing method including:

receiving a data processing request, and acquiring a corresponding data processing type identifier, a corresponding demand identifier and data to be processed;

combining the first cleaning parameters according to the data processing type identification and the requirement identification to obtain combined cleaning parameters, and further determining corresponding combined cleaning parameter values;

determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value;

respectively acquiring a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter;

and performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value.

Optionally, determining a corresponding combined cleaning parameter value comprises:

acquiring configuration information corresponding to a demand identifier;

acquiring first cleaning parameter values corresponding to each first cleaning parameter in the combined cleaning parameters from the configuration information;

determining the arrangement sequence of each first cleaning parameter in the combined cleaning parameters, and then combining the first cleaning parameter values based on the arrangement sequence to obtain the corresponding combined cleaning parameter values.

Optionally, determining the second cleaning parameter and the third cleaning parameter includes:

determining an execution time identifier from the combined cleaning parameter values, and further determining a second cleaning parameter based on the execution time identifier;

and determining a branch mark from the combined cleaning parameter values, and further determining a third cleaning parameter based on the branch mark.

Optionally, performing backup and/or scrubbing processing on the to-be-processed data based on the combined scrubbing parameter value, the second scrubbing parameter value, and the third scrubbing parameter value includes:

and responding to the retention time identifier included in the combined cleaning parameter values, and executing backup and/or cleaning processing on the data to be processed based on the retention time corresponding to the retention time identifier, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

and in response to one or more of the date field identifier, the date type identifier or the date format identifier included in the combined cleaning parameter value, performing backup and/or cleaning processing on the data to be processed according to the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value and one or more of the date field identifier, the date type identifier or the date format identifier.

and in response to one or more of the month-end reservation mechanism identifier and the year-end reservation mechanism identifier included in the combined cleaning parameter value, performing backup and/or cleaning processing on the to-be-processed data according to the one or more of the month-end reservation mechanism identifier and the year-end reservation mechanism identifier, the execution time corresponding to the second cleaning parameter value, and the branch data corresponding to the third cleaning parameter value.

acquiring a backup identifier and a cleaning identifier corresponding to data to be processed from the data processing request;

determining data to be backed up and data to be cleaned in the data to be processed according to the backup identification and the cleaning identification respectively;

and executing backup processing on the data to be backed up and executing cleaning processing on the data to be backed up based on the combined cleaning parameter value, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

In addition, the present application also provides a data processing apparatus, including:

The receiving unit is configured to receive the data processing request and acquire a corresponding data processing type identifier, a corresponding demand identifier and data to be processed;

the parameter combination unit is configured to combine the first cleaning parameters according to the data processing type identification and the requirement identification to obtain combined cleaning parameters, and then determine corresponding combined cleaning parameter values;

a parameter determination unit configured to determine a second cleaning parameter and a third cleaning parameter from the combined cleaning parameter value;

the parameter value acquisition unit is configured to respectively acquire a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter;

a data processing unit configured to perform backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value, and the third cleaning parameter value.

Optionally, the parameter combination unit is further configured to:

acquiring configuration information corresponding to the requirement identification;

Optionally, the parameter determination unit is further configured to:

and determining a branch identifier from the combined cleaning parameter values, and further determining a third cleaning parameter based on the branch identifier.

Optionally, the data processing unit is further configured to:

and in response to the combined cleaning parameter value being configured as the retention time identifier, performing backup and/or cleaning processing on the data to be processed based on the retention time corresponding to the retention time identifier, the execution time corresponding to the second cleaning parameter value, and the branch data corresponding to the third cleaning parameter value.

Optionally, the data processing unit is further configured to:

and in response to one or more of the combined cleaning parameter values configured as date field identification, date type identification or date format identification, performing backup and/or cleaning processing on the data to be processed according to the one or more of the date field identification, the date type identification or the date format identification, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

Optionally, the data processing unit is further configured to:

And in response to one or more of the combined cleaning parameter values which are configured to be the last-month retention mechanism identifier and the last-year retention mechanism identifier, performing backup and/or cleaning processing on the data to be processed according to the last-month retention mechanism identifier and one or more of the last-year retention mechanism identifier, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

Optionally, the data processing unit is further configured to:

and executing backup processing on the data to be backed up and executing cleaning processing on the data to be cleaned based on the combined cleaning parameter value, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

In addition, the present application also provides an electronic device for data processing, including: one or more processors; a storage device for storing one or more programs which, when executed by one or more processors, cause the one or more processors to implement the data processing method as described above.

In addition, the present application also provides a computer readable medium, on which a computer program is stored, which when executed by a processor implements the data processing method as described above.

To achieve the above object, according to still another aspect of embodiments of the present application, there is provided a computer program product.

A computer program product according to an embodiment of the present application includes a computer program, and when the computer program is executed by a processor, the computer program implements the data processing method according to an embodiment of the present application.

One embodiment of the above invention has the following advantages or benefits: the method comprises the steps of receiving a data processing request, and obtaining a corresponding data processing type identifier, a corresponding demand identifier and data to be processed; combining the first cleaning parameters according to the data processing type identification and the requirement identification to obtain combined cleaning parameters, and further determining corresponding combined cleaning parameter values; determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value; respectively acquiring a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter; performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. The method comprises the steps of obtaining a combined cleaning parameter by combining a first cleaning parameter according to a data processing type identifier and a demand identifier, determining a second cleaning parameter and a third cleaning parameter according to a parameter value corresponding to the combined cleaning parameter, and backing up and/or cleaning data to be processed according to the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. Therefore, customized data backup and cleaning under the appointed date or the appointed frequency are realized, the occupation of the space of the database is reduced, and the performance of the database is improved.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a further understanding of the application and are not to be construed as limiting the application. Wherein:

fig. 1 is a schematic view of a main flow of a data processing method according to a first embodiment of the present application;

fig. 2 is a schematic view of a main flow of a data processing method according to a second embodiment of the present application;

fig. 3 is a schematic view of a main flow of a data processing method according to a third embodiment of the present application;

fig. 4 is a schematic view of an application scenario of a data processing method according to a fourth embodiment of the present application;

FIG. 5 is a schematic diagram of the main elements of a data processing apparatus according to an embodiment of the present application;

FIG. 6 is an exemplary system architecture diagram to which embodiments of the present application may be applied;

fig. 7 is a schematic structural diagram of a computer system suitable for implementing the terminal device or the server according to the embodiment of the present application.

Detailed Description

The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness. According to the technical scheme, the data acquisition, storage, use, processing and the like meet relevant regulations of national laws and regulations.

Fig. 1 is a schematic diagram of a main flow of a data processing method according to a first embodiment of the present application, and as shown in fig. 1, the data processing method includes:

step S101, receiving a data processing request, and acquiring a corresponding data processing type identifier, a corresponding demand identifier and corresponding data to be processed.

In this embodiment, an execution subject (for example, a server) of the data processing method may receive the data processing request by wired connection or wireless connection. Specifically, the data processing request may be a request for data backup and/or cleaning of report data in the database. The execution subject may obtain the data processing type identifier and the requirement identifier carried in the data processing request. The data processing type identifier may be a report type corresponding to report data to be processed, where the report type may include a daily report (D), a ten-day report (P), a monthly report (M), a quarterly report (S), and a yearly report (Y). And the requirement identification is used for characterizing which parameters are backed up and/or cleaned. Specifically, the requirement identifier may be a parameter name, such as a report name, a date field name, a date type, a date format, a retention time, and the like. For example, the data processing type is daily report (D), and the requirement identifier corresponds to the data to be processed of the relevant parameters such as report name, date field name, date type, date format, retention time, etc. The execution main body can acquire the corresponding data to be processed according to the address of the data to be processed carried in the data processing request. The number of the parameters corresponding to the requirement identification and the content of the parameters are not specifically limited in the embodiment of the application.

And S102, combining the first cleaning parameters according to the data processing type identification and the requirement identification to obtain combined cleaning parameters, and further determining corresponding combined cleaning parameter values.

The first cleaning parameters may include, for example, report name, report type, date field name, date type, date format, cleaning mechanism, retention time, backup mechanism, end-of-month retention mechanism, end-of-year retention mechanism, execution time, branch processing mechanism.

For example, the requirement identifier includes BBM, RQZDM, and RQLX, which correspond to the entry name, the date field name, and the date type, respectively. When the data processing type is marked as D, the daily report is indicated to be subjected to data processing. The first cleaning parameter is combined according to the data processing type identifier and the requirement identifier to obtain a combined cleaning parameter, specifically, the first cleaning parameter corresponding to the requirement identifier is combined according to a combination mode corresponding to the data processing type identifier, for example, if the parameter arrangement sequence corresponding to the combination mode corresponding to the daily report is a report name, a date type, and a date field name, the execution main body may perform arrangement and combination on the requirement identifier according to the arrangement sequence, and finally obtain the combined cleaning parameter BBM-RQLX-RQZDM.

After the execution main body obtains the combined cleaning parameter, the execution main body can obtain a combined cleaning parameter value corresponding to the combined cleaning parameter from the data to be processed. For example, BBM-RQLX-RQZDM corresponds to the combined cleaning parameter values: the name m of the database table to be cleaned-character string type-2022.1.1 means that data to be processed with the character string type in the name m of the database table with the date 2022.1.1 is backed up and/or cleaned.

And step S103, determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value.

For example, when the combined cleaning parameter is BBM-ZXSJ-FHCLJZ, the corresponding combined cleaning parameter value is m-S-N-1. And determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value. Specifically, when the combined cleaning parameter value is m-S-N-1, the second cleaning parameter corresponding to S-N in the combined cleaning parameter value m-S-N-1 is SDAY, namely the fixed day of the Nth month of each quarter, and the third cleaning parameter corresponding to 1 in the combined cleaning parameter value m-S-N-1 is processed according to the branch, and the branch parameter needs to be obtained.

And step S104, respectively obtaining a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter.

For example, if the second cleaning parameter is SDAY, the second cleaning parameter value corresponding to the second cleaning parameter may be 15, which means that data processing is performed on the 15 th day of the nth month every quarter.

For example, if the third cleaning parameter is 1, the third cleaning parameter value corresponding to the third cleaning parameter is processed in rows, and the process of obtaining the row dividing parameter is executed.

And step S105, performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value.

Illustratively, the corresponding data M to be processed is obtained according to the database table name M corresponding to the report type corresponding to the combined cleaning parameter value M-S-N-1, and backup and/or cleaning are performed according to the corresponding data N in the data M to be processed corresponding to the branch parameter corresponding to the 15 th day of the nth month of each quarter of the second cleaning parameter value and the third cleaning parameter value.

Specifically, executing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value includes:

In an example, the processing is performed according to the retention time, the numerical value N is modeled, the calculation is performed according to the report type, the report type D represents that N days are retained, the report type P represents that N ten days are retained, the report type M represents that N months are retained, and so on. Processing according to the execution time, D, M-N, S-N, Y-N; d is daily administration; M-N is executed every N days of each month, and when N is ND, the execution is shown at the end of each month, and the configuration is carried out according to the month in ten days; S-N is performed for a fixed day of month N (SDAY) for each quarter; Y-N is performed every year on the fixed day of month N (YDAY); the report backup and the second cleaning parameter are required to be read, the second cleaning parameter values SDAY and YDAY are obtained, and data backup and/or cleaning are carried out according to the second cleaning parameter values SDAY and YDAY. And (3) processing according to a branch processing mechanism, wherein 0-processing according to the branch and 1-processing according to the branch, and if the parameter value of the branch processing mechanism is 1, namely the third cleaning parameter value is 1, the branch parameter needs to be acquired to participate in data backup and/or cleaning.

and in response to the fact that the combined cleaning parameter value comprises one or more of a date field identifier, a date type identifier or a date format identifier, executing backup and/or cleaning processing on the data to be processed according to the one or more of the date field identifier, the date type identifier or the date format identifier, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value. By way of example, a date field identification, such as a date field name; the date type identification comprises D (date type), S (character string type); date format label, YYYYMMDD for year, month and day, YYYYMM for year, and YYYY for year.

and in response to the combined cleaning parameter value including one or more of the month end reservation mechanism identifier and the year end reservation mechanism identifier, performing backup and/or cleaning processing on the data to be processed according to the month end reservation mechanism identifier, one or more of the year end reservation mechanism identifier, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value. For example, the end-of-month reservation mechanism identifier, including 0-no-reservation, 1-reservation, is valid only for the datebook, the ten-day-newspaper. The end-of-year retention mechanism identification comprises 0-no retention and 1-retention, and is only effective for the daily newspaper, the ten-day newspaper and the monthly newspaper.

In the embodiment, by receiving a data processing request, a corresponding data processing type identifier, a corresponding demand identifier and data to be processed are acquired; combining the first cleaning parameters according to the data processing type identification and the requirement identification to obtain combined cleaning parameters, and further determining corresponding combined cleaning parameter values; determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value; respectively acquiring a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter; performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. The method comprises the steps of obtaining a combined cleaning parameter by combining a first cleaning parameter according to a data processing type identifier and a demand identifier, determining a second cleaning parameter and a third cleaning parameter according to a parameter value corresponding to the combined cleaning parameter, and backing up and/or cleaning data to be processed according to the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. Therefore, customized data backup and cleaning under the appointed date or the appointed frequency are realized, the occupation of the space of the database is reduced, and the performance of the database is improved.

Fig. 2 is a schematic main flow chart of a data processing method according to a second embodiment of the present application, and as shown in fig. 2, the data processing method includes:

step S201, receiving a data processing request, and acquiring a corresponding data processing type identifier, a requirement identifier, and to-be-processed data.

Step S202, the first cleaning parameter is combined according to the data processing type identification and the requirement identification to obtain a combined cleaning parameter.

Step S203, obtaining configuration information corresponding to the requirement identifier.

Step S204, acquiring first cleaning parameter values corresponding to each first cleaning parameter in the combined cleaning parameters from the configuration information.

The execution main body may obtain each first cleaning parameter in the combined cleaning parameter from the preset configuration information corresponding to the requirement identifier, for example, a report name, a report type, an execution time, and a first cleaning parameter value corresponding to a branch processing mechanism, for example, m, D, Y-N, 1.

Step S205, determining the arrangement sequence of each first cleaning parameter in the combined cleaning parameters, such as the branch processing mechanism, the execution time, the report name and the report type, and combining the first cleaning parameters based on the arrangement sequence to obtain the corresponding combined cleaning parameter value, such as 1-Y-N-m-D. Therefore, the combined cleaning parameter values based on the importance sequence can be provided for users, priority references are provided for data backup and/or cleaning, and data processing quality is improved.

And step S206, determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value.

Step S207, respectively obtaining a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter.

And step S208, performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value.

acquiring a backup identifier (for example, 0-no backup, 1-backup) and a cleaning identifier (for example, 0-no cleaning, 1-cleaning) corresponding to the data to be processed from the data processing request; determining data to be backed up and data to be cleaned in the data to be processed according to the backup identification and the cleaning identification respectively; and executing backup processing on the data to be backed up and executing cleaning processing on the data to be backed up based on the combined cleaning parameter value, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

The execution main body can determine which data to be backed up and which data to be cleaned are to be cleaned according to the corresponding relation between the backup identification and the cleaning identification carried in the data processing request and the data to be processed, so that the data to be backed up and the data to be cleaned are backed up and cleaned, the data to be backed up and the data to be cleaned are prevented from being mixed up, and the accuracy of data processing is improved.

Fig. 3 is a schematic diagram of a main flow of a data processing method according to a third embodiment of the present application. As shown in fig. 3, the data processing method includes:

step S301, receiving a data processing request, and acquiring a corresponding data processing type identifier, a requirement identifier and data to be processed.

The data processing request may be, for example, a request for backing up and cleaning data. The data processing type identifier may be a report type corresponding to the report data to be processed, where the report type may include a daily report (D), a ten-day report (P), a monthly report (M), a quarterly report (S), and a yearly report (Y).

Step S302, combining the first cleaning parameter according to the data processing type identifier and the requirement identifier to obtain a combined cleaning parameter, and further determining a corresponding combined cleaning parameter value.

The first cleaning parameters may include, for example, report name, report type, date field name, date type, date format, cleaning mechanism, retention time, backup mechanism, end-of-month retention mechanism, end-of-year retention mechanism, execution time, branch handling mechanism.

Illustratively, the requirement identifier includes BBM, RQZDM, RQLX, corresponding to entry name, date field name, date type, respectively. When the data processing type is marked as D, the daily report is indicated to be subjected to data processing. The first cleaning parameter is combined according to the data processing type identifier and the requirement identifier to obtain a combined cleaning parameter, specifically, the first cleaning parameter corresponding to the requirement identifier may be combined according to a combination mode corresponding to the data processing type identifier, for example, if a parameter arrangement sequence corresponding to a combination mode corresponding to a daily newspaper is a report name, a date type, and a date field name, the execution main body may arrange and combine the requirement identifier according to the arrangement sequence, and finally obtain the combined cleaning parameter BBM-RQLX-RQZDM.

After the execution main body obtains the combined cleaning parameter, the combined cleaning parameter value corresponding to the combined cleaning parameter can be obtained from the data to be processed. For example, BBM-RQLX-RQZDM corresponds to a combined cleaning parameter value of: the name m of the database table to be cleaned-character string type-2022.1.1 means that data to be processed with the name m of the database table with the date 2022.1.1 is backed up and/or cleaned.

Step S303, determining an execution time identifier from the combined cleaning parameter values, and determining a second cleaning parameter based on the execution time identifier.

Illustratively, when the combined cleaning parameter is BBM-ZXSJ-FHCLJZ, the corresponding combined cleaning parameter value is m-S-N-1. And determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value. Specifically, when the combined cleaning parameter value is m-S-N-1, the second cleaning parameter corresponding to S-N in the combined cleaning parameter value m-S-N-1 is SDAY, namely the fixed day of the Nth month of each quarter, and the third cleaning parameter corresponding to 1 in the combined cleaning parameter value m-S-N-1 is processed according to the branch, the branch parameter needs to be obtained.

An execution time stamp (e.g., S-N or Y-N) is determined from the combined cleaning parameter values, and a second cleaning parameter (where S-N corresponds to the second cleaning parameter SDAY, i.e., fixed day of month N for each quarter, and Y-N corresponds to the second cleaning parameter YDAY, i.e., fixed day of month N per year) is determined based on the execution time stamp.

Step S304, determining a branch mark from the combined cleaning parameter values, and further determining a third cleaning parameter based on the branch mark.

And determining a branch identifier (for example, 0 or 1) from the combined cleaning parameter values, and further determining a third cleaning parameter based on the branch identifier (the third cleaning parameter corresponding to the branch identifier 0 represents that the cleaning is not processed according to the branch, and the third cleaning parameter corresponding to the branch identifier 1 represents that the cleaning is processed according to the branch).

Step S305, respectively obtaining a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter.

Step S306, executing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value.

According to the embodiment of the application, the first cleaning parameter is combined according to the data processing type identifier and the requirement identifier to obtain the combined cleaning parameter, the second cleaning parameter and the third cleaning parameter are determined according to the parameter value corresponding to the combined cleaning parameter, and the data to be processed is backed up and/or cleaned according to the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. Therefore, customized data backup and cleaning are realized on the appointed date or the appointed frequency according to the execution time corresponding to the second cleaning parameter value and whether the third cleaning parameter value is processed according to the branch line, the occupation of the space of the database is reduced, and the performance of the database is improved.

Fig. 4 is a schematic view of an application scenario of a data processing method according to a fourth embodiment of the present application. The data processing method is applied to a scene of backing up and cleaning long-time accumulated data. Taking bank data processing as an example, universal report data backup and cleaning are realized in a parameterized manner, and main parameters include a report name, a report type, a date field name, a date type, a date format, a cleaning mechanism, retention time, a month end retention mechanism, a year end retention mechanism, a backup mechanism, execution time, a branch processing mechanism, a time parameter associated with the execution time parameter and a branch parameter associated with the branch processing mechanism. As shown in fig. 4, an execution main body of the data processing method according to the embodiment of the present application may read a report backup and a first cleaning parameter, where the first cleaning parameter includes: report name, report type, date field name, date type, date format, cleaning mechanism, retention time, backup mechanism, month end retention mechanism, year end retention mechanism, execution time, branch processing mechanism; processing according to the report name, wherein the report name is the name of the database table to be cleaned; processing according to report types, wherein the report types comprise daily (D), ten-day (P), monthly (M), quarterly (S) and annual (Y); processing according to the date field name, and counting the date field name by a report; processing according to date types, wherein the date types comprise D (date type) and S (character string type); processing according to a date format, wherein the date format YYYYMMDD represents the year, month and day, YYYYMM represents the year and month, and YYYY represents the year; processing according to a cleaning mechanism, namely 0-no cleaning and 1-cleaning; processing according to the retention time, integer numerical value N, calculating according to the report type, wherein the report type D represents retention for N days, the report type P represents retention for N ten days, the report type M represents retention for N months, and the like; processing according to a backup mechanism, wherein 0-no backup and 1-backup are carried out; processing according to a retention mechanism at the end of a month, wherein 0-retention is not performed, 1-retention is performed, and the retention is only effective for the daily newspaper and the ten-day newspaper; processing according to a retention mechanism at the end of the year, wherein the retention mechanism is 0-no retention and 1-retention, and the retention mechanism is only effective for the daily newspaper, the ten-day newspaper and the monthly newspaper; processing according to the execution time, D, M-N, S-N, Y-N; d is daily execution; M-N is executed in N days of each month, N represents the end of each month when being ND, and the configuration is carried out according to the month in ten days; S-N is performed for a fixed day of month N (SDAY) for each quarter; Y-N is performed on a fixed day of month N per year (YDAY); reading the report backup and the second cleaning parameter to obtain SDAY and YDAY; and (3) processing according to a branch processing mechanism, wherein 0-processing according to the branch and 1-processing according to the branch, if the parameter value of the branch processing mechanism is 1, the report backup and the third cleaning parameter need to be read, and the branch parameter is obtained.

Each parameter contained in the first cleaning parameter can be flexibly configured according to the report type and the actual condition of report service data; the second cleaning parameter is used for obtaining SDAY and YDAY, wherein the SDAY represents the fixed day of the Nth month of each quarter, and the YDAY represents the fixed day of the Nth month every year; if the parameter value (i.e., the third cleaning parameter value) configured by the branch processing mechanism in the first cleaning parameter is 1, the report backup needs to be read and the branch parameters need to be obtained, and the branch parameters may include parameters of all branches, or parameters of designated branches may be configured according to actual service needs. The report backup and cleaning method has the advantages of multiple report backup and cleaning parameter configurations, flexible combination and wide coverage, can meet the requirements of various backup and cleaning mechanisms of various reports, can pertinently realize report data backup and cleaning, and can be universal.

According to the embodiment of the application, the report data backup and the configuration of the cleaning parameters can be realized, and the parameters contained in the first cleaning parameter can be flexibly combined and configured aiming at different report types and report data; the report data backup and cleaning time can be flexibly configured, and the method can be universal for different types of reports; the branch parameters can be flexibly configured, and customized branch parameter configuration can be realized aiming at the characteristic services and specific reports of different branches. According to the embodiment of the application, different report names of a daily report, a ten-day report, a monthly report, a seasonal report and an annual report and customized data backup and cleaning of different branches of the corresponding reports on a specified date or a specified frequency are realized through the configuration of the report data backup and the first cleaning parameter, the second cleaning parameter and the third cleaning parameter, and the branch parameterized report data backup and cleaning are realized for branch characteristic services and specific reports.

Fig. 5 is a schematic diagram of main units of a data processing apparatus according to an embodiment of the present application. As shown in fig. 5, the data processing apparatus 500 includes a receiving unit 501, a parameter combining unit 502, a parameter determining unit 503, a parameter value acquiring unit 504, and a data processing unit 505.

The receiving unit 501 is configured to receive a data processing request, and obtain a corresponding data processing type identifier, a requirement identifier, and to-be-processed data.

A parameter combination unit 502 configured to combine the first cleaning parameters according to the data processing type identifier and the requirement identifier to obtain combined cleaning parameters, and then determine corresponding combined cleaning parameter values.

A parameter determination unit 503 configured to determine the second cleaning parameter and the third cleaning parameter based on the combined cleaning parameter value.

A parameter value obtaining unit 504 configured to obtain a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter, respectively.

A data processing unit 505 configured to perform a backup and/or scrubbing process on the data to be processed based on the combined scrubbing parameter value, the second scrubbing parameter value and the third scrubbing parameter value.

In some embodiments, the parameter combination unit 502 is further configured to: acquiring configuration information corresponding to a demand identifier; acquiring first cleaning parameter values corresponding to each first cleaning parameter in the combined cleaning parameters from the configuration information; determining the arrangement sequence of each first cleaning parameter in the combined cleaning parameters, and then combining the first cleaning parameter values based on the arrangement sequence to obtain the corresponding combined cleaning parameter values.

In some embodiments, the parameter determination unit 503 is further configured to: determining an execution time identifier from the combined cleaning parameter values, and further determining a second cleaning parameter based on the execution time identifier; and determining a branch mark from the combined cleaning parameter values, and further determining a third cleaning parameter based on the branch mark.

In some embodiments, the data processing unit 505 is further configured to: and in response to the combined cleaning parameter value which is configured as the retention time identifier, performing backup and/or cleaning processing on the to-be-processed data based on the retention time corresponding to the retention time identifier, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

In some embodiments, the data processing unit 505 is further configured to: and in response to one or more of the combined cleaning parameter values which are configured to be date field identification, date type identification or date format identification, performing backup and/or cleaning processing on the data to be processed according to one or more of the date field identification, the date type identification or the date format identification, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

In some embodiments, the data processing unit 505 is further configured to: and in response to one or more of the combined cleaning parameter values which are configured to be the last-month retention mechanism identifier and the last-year retention mechanism identifier, performing backup and/or cleaning processing on the data to be processed according to the last-month retention mechanism identifier and one or more of the last-year retention mechanism identifier, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

In some embodiments, the data processing unit 505 is further configured to: acquiring a backup identifier and a cleaning identifier corresponding to data to be processed from the data processing request; determining data to be backed up and data to be cleaned in the data to be processed according to the backup identification and the cleaning identification respectively; and executing backup processing on the data to be backed up and executing cleaning processing on the data to be backed up based on the combined cleaning parameter value, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

It should be noted that, in the present application, the data processing method and the data processing apparatus have corresponding relation in the specific implementation contents, and therefore, the repeated contents are not described again.

Fig. 6 shows an exemplary system architecture 600 to which the data processing method or data processing apparatus of the embodiments of the present application may be applied.

As shown in fig. 6, the system architecture 600 may include

terminal devices

601, 602, 603, a network 604, and a server 605. The network 604 serves as a medium for providing communication links between the

terminal devices

601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

A user may use the

terminal devices

601, 602, 603 to interact with a server 605, via a network 604, to receive or send messages or the like. The

terminal devices

601, 602, 603 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

601, 602, 603 may be various electronic devices having data processing screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 605 may be a server providing various services, such as a background management server (for example only) providing support for data processing requests submitted by users using the

terminal devices

601, 602, 603. The background management server can receive the data processing request and acquire a corresponding data processing type identifier, a corresponding demand identifier and data to be processed; combining the first cleaning parameters according to the data processing type identification and the requirement identification to obtain combined cleaning parameters, and further determining corresponding combined cleaning parameter values; determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value; respectively acquiring a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter; performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. The method comprises the steps of obtaining a combined cleaning parameter by combining a first cleaning parameter according to a data processing type identifier and a demand identifier, determining a second cleaning parameter and a third cleaning parameter according to a parameter value corresponding to the combined cleaning parameter, and backing up and/or cleaning data to be processed according to the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. Therefore, customized data backup and cleaning under the appointed date or the appointed frequency are realized, the occupation of the space of the database is reduced, and the performance of the database is improved.

It should be noted that the data processing method provided in the embodiment of the present application is generally executed by the server 605, and accordingly, the data processing apparatus is generally disposed in the server 605.

It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use in implementing a terminal device of an embodiment of the present application. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the computer system 700 are also stored. The CPU701, the ROM702, and the RAM703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a liquid crystal credit authorization query processor (LCD), and the like, and a speaker and the like; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

In particular, according to embodiments disclosed herein, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments disclosed herein include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the system of the present application when executed by the Central Processing Unit (CPU) 701.

It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a receiving unit, a parameter combining unit. The device comprises a parameter determining unit, a parameter value acquiring unit and a data processing unit. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs, and when the one or more programs are executed by one device, the device receives a data processing request and acquires a corresponding data processing type identifier, a requirement identifier and to-be-processed data; combining the first cleaning parameters according to the data processing type identification and the requirement identification to obtain combined cleaning parameters, and further determining corresponding combined cleaning parameter values; determining a second cleaning parameter and a third cleaning parameter according to the combined cleaning parameter value; respectively acquiring a second cleaning parameter value and a third cleaning parameter value corresponding to the second cleaning parameter and the third cleaning parameter; performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value.

The computer program product of the present application comprises a computer program which, when executed by a processor, implements the data processing method of the embodiments of the present application.

According to the technical scheme of the embodiment of the application, the first cleaning parameter is combined according to the data processing type identifier and the requirement identifier to obtain the combined cleaning parameter, the second cleaning parameter and the third cleaning parameter are determined according to the parameter value corresponding to the combined cleaning parameter, and the data to be processed is backed up and/or cleaned according to the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value. Therefore, customized data backup and cleaning under the appointed date or appointed frequency are realized, the occupation of the database space is reduced, and the database performance is improved.

The above-described embodiments are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of data processing, comprising:

Combining a first cleaning parameter according to the data processing type identifier and the demand identifier to obtain a combined cleaning parameter, and further determining a corresponding combined cleaning parameter value;

performing backup and/or cleaning processing on the data to be processed based on the combined cleaning parameter value, the second cleaning parameter value and the third cleaning parameter value.

2. The method of claim 1, wherein determining the corresponding combined cleaning parameter value comprises:

acquiring first cleaning parameter values corresponding to all first cleaning parameters in the combined cleaning parameters from the configuration information;

determining the arrangement sequence of each first cleaning parameter in the combined cleaning parameters, and then combining the first cleaning parameter values based on the arrangement sequence to obtain the corresponding combined cleaning parameter value.

3. The method of claim 1, wherein determining the second cleaning parameter and the third cleaning parameter comprises:

4. The method of claim 1, wherein the performing backup and/or scrubbing processing on the data to be processed based on the combined scrubbing parameter value, the second scrubbing parameter value, and the third scrubbing parameter value comprises:

and in response to the fact that the combined cleaning parameter value comprises a retention time identifier, performing backup and/or cleaning processing on the data to be processed based on retention time corresponding to the retention time identifier, execution time corresponding to the second cleaning parameter value and branch data corresponding to the third cleaning parameter value.

5. The method of claim 1, wherein the performing backup and/or scrubbing processing on the data to be processed based on the combined scrubbing parameter value, the second scrubbing parameter value, and the third scrubbing parameter value comprises:

and in response to one or more of a date field identifier, a date type identifier or a date format identifier included in the combined cleaning parameter values, performing backup and/or cleaning processing on the data to be processed according to one or more of the date field identifier, the date type identifier or the date format identifier, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

6. The method of claim 1, wherein the performing backup and/or scrubbing processing on the data to be processed based on the combined scrubbing parameter value, the second scrubbing parameter value, and the third scrubbing parameter value comprises:

and in response to one or more of a month end reservation mechanism identifier and an end-of-year reservation mechanism identifier included in the combined cleaning parameter values, performing backup and/or cleaning processing on the to-be-processed data according to the one or more of the month end reservation mechanism identifier and the end-of-year reservation mechanism identifier, the execution time corresponding to the second cleaning parameter value, and the branch data corresponding to the third cleaning parameter value.

7. The method of claim 1, wherein the performing backup and/or scrubbing processing on the data to be processed based on the combined scrubbing parameter value, the second scrubbing parameter value, and the third scrubbing parameter value comprises:

acquiring a backup identifier and a cleaning identifier corresponding to the data to be processed from the data processing request;

And executing backup processing on the data to be backed up based on the combined cleaning parameter value, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value, and executing cleaning processing on the data to be cleaned.

8. A data processing apparatus, characterized by comprising:

the parameter combination unit is configured to combine the first cleaning parameters according to the data processing type identifier and the requirement identifier to obtain combined cleaning parameters, and then determine corresponding combined cleaning parameter values;

a parameter value obtaining unit configured to obtain the second cleaning parameter, a second cleaning parameter value corresponding to the third cleaning parameter, and a third cleaning parameter value, respectively;

a data processing unit configured to perform a backup and/or scrubbing process on the data to be processed based on the combined scrubbing parameter value, the second scrubbing parameter value, and the third scrubbing parameter value.

9. The apparatus of claim 8, wherein the parameter combining unit is further configured to:

10. The apparatus of claim 8, wherein the parameter determination unit is further configured to:

11. The apparatus of claim 8, wherein the data processing unit is further configured to:

and in response to the combined cleaning parameter value being configured as a retention time identifier, performing backup and/or cleaning processing on the data to be processed based on the retention time corresponding to the retention time identifier, the execution time corresponding to the second cleaning parameter value, and the branch data corresponding to the third cleaning parameter value.

12. The apparatus of claim 8, wherein the data processing unit is further configured to:

and in response to one or more of the combined cleaning parameter values configured as a date field identifier, a date type identifier or a date format identifier, performing backup and/or cleaning processing on the data to be processed according to one or more of the date field identifier, the date type identifier or the date format identifier, the execution time corresponding to the second cleaning parameter value and the branch data corresponding to the third cleaning parameter value.

13. The apparatus of claim 8, wherein the data processing unit is further configured to:

and in response to one or more of the combined cleaning parameter values configured as a last-month retention mechanism identifier and a last-year retention mechanism identifier, performing backup and/or cleaning processing on the data to be processed according to one or more of the last-month retention mechanism identifier and the last-year retention mechanism identifier, the execution time corresponding to the second cleaning parameter value, and the branch data corresponding to the third cleaning parameter value.

14. An electronic device for data processing, comprising:

One or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

15. A computer-readable medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method of any one of claims 1-7.

16. A computer program product comprising a computer program, characterized in that the computer program realizes the method according to any of claims 1-7 when executed by a processor.