CN117057903A - Data processing method, device, equipment and medium - Google Patents

Data processing method, device, equipment and medium Download PDF

Info

Publication number
CN117057903A
CN117057903A CN202311004301.7A CN202311004301A CN117057903A CN 117057903 A CN117057903 A CN 117057903A CN 202311004301 A CN202311004301 A CN 202311004301A CN 117057903 A CN117057903 A CN 117057903A
Authority
CN
China
Prior art keywords
data
data processing
processed
judging whether
proportion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311004301.7A
Other languages
Chinese (zh)
Inventor
蔡旻
刘佳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bosheng Haimi Technology Beijing Co ltd
Original Assignee
Bosheng Haimi Technology Beijing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bosheng Haimi Technology Beijing Co ltd filed Critical Bosheng Haimi Technology Beijing Co ltd
Priority to CN202311004301.7A priority Critical patent/CN117057903A/en
Publication of CN117057903A publication Critical patent/CN117057903A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/03Credit; Loans; Processing thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2282Tablespace storage structures; Management thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/50Indexing scheme relating to G06F9/50
    • G06F2209/5018Thread allocation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/54Indexing scheme relating to G06F9/54
    • G06F2209/548Queue

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • General Engineering & Computer Science (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Development Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Technology Law (AREA)
  • Strategic Management (AREA)
  • Marketing (AREA)
  • Economics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The application discloses a data processing method, a device, equipment and a medium, belonging to the technical field of data processing, wherein the method comprises the following steps: receiving an instruction for processing a data schedule; screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table; the method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed; configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform; and carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode. The application can improve the post-credit data expression of the business, improve the success rate of customer repayment and reduce the post-credit bad account data rate and bad account loss.

Description

Data processing method, device, equipment and medium
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, device, and medium.
Background
With the rapid development of internet finance, once the scale of finance credit business reaches a certain level, the level of bill or repayment plan data after the loan also rises in water, how to screen target data in the level of tens of millions or even hundreds of millions of data, how to use technical capability to improve overdue remittance rate, reduce credit risk and how to reduce bad losses become a common pain point problem in the industry.
Now, if the user is overdue, the transaction system initiates batch payment or the user initiates active payment, and the payment is only supported by full payment (i.e. principal+interest+service fee+penalty), but not by partial payment.
1, the payment for the payment at present can only carry out full-amount payment on the payment plan of each period according to the dimension of the overdue report, but no corresponding solution exists for the case of insufficient payment.
2, the existing substitute buckles only carry out full-amount substitute buckles aiming at each payment plan, full-amount batch buckles are carried out on users within 7 days of overdue, and the payment after 7 days needs the users to actively initiate substitute buckles.
So the whole repayment strategy is limited at present, and the service requirements cannot be met.
Disclosure of Invention
To overcome the technical defects described above, the present application aims to provide a data processing method, a device, an apparatus and a medium, wherein the method comprises: receiving an instruction for processing a data schedule; screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table; the method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed; configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform; and carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode. The application can improve the post-credit data expression of the business, improve the success rate of customer repayment and reduce the post-credit bad account data rate and bad account loss.
The specific technical scheme provided by the embodiment of the application is as follows:
in a first aspect, a data processing method is provided, the method comprising:
receiving an instruction for processing a data schedule;
screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table;
the method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed;
configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform;
and carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
Further, the screening and cleaning the data schedule according to the product type includes:
and de-duplicating the data schedule according to the product type, and reserving user ID and/or product number and/or order number and/or data processing time field information.
Further, the screening and cleaning the data schedule according to the product type further includes:
screening a data schedule with the longest overdue time from the data schedules according to the data execution time;
Dividing the data schedule with the longest overdue time into a plurality of batch schedules according to the dimension of the data processing order, and arranging the batch schedules in ascending order according to the overdue amount of the data schedule;
and merging the batch schedules with the same overdue days, and taking the data processing order with the shortest overdue days as the account age of the user level.
Further, the configuring, by the configuration platform, a multi-level data processing execution policy for each data service to be processed includes:
configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform;
wherein the multi-stage data processing execution strategy comprises setting account age and/or data processing channel and/or data processing frequency and/or data processing amount and/or data processing proportion.
Further, the data processing of the plurality of data services to be processed according to the multi-stage data processing execution policy and the multi-batch processing mode includes:
judging whether the task for carrying out data processing on a plurality of data services to be processed is a received instruction for processing a data schedule;
if yes, judging whether the data processing time of the plurality of data services to be processed spans the day or not; if not, ending the flow;
Judging whether the data processing time of the plurality of data services to be processed spans the day or not comprises the following steps:
judging whether the data processing time of a plurality of data services to be processed spans the day or not;
if yes, ending the flow; if not, judging whether the first data processing of the current service is successful.
Further, the determining whether the first data processing of the current service is successful includes:
judging whether the first data processing of the current service is successful or not;
if yes, judging whether the current service data processing proportion is the last data processing proportion; if not, judging whether the reason for the failure of processing the current service data is insufficient balance;
the judging whether the current service data processing proportion is the last data processing proportion comprises the following steps:
judging whether the current service data processing proportion is the last data processing proportion or not;
if yes, carrying out data processing on the next batch of data services to be processed, and judging whether all the data services to be processed are processed; if not, carrying out data processing on the current service according to the next data processing proportion;
judging whether all the data services to be processed are processed or not, wherein the judging comprises the steps of;
Judging whether all the data services to be processed are processed or not;
if yes, ending the flow; if not, continuing to process the data of the next batch of data service to be processed;
judging whether the reason for failure in processing the current service data is insufficient balance or not, wherein the judging comprises the steps of;
judging whether the reason of failure of processing the service data in the current period is insufficient balance;
if yes, judging whether the next data processing proportion of the current service is larger than the current data processing proportion; if not, ending the flow;
the judging whether the next data processing proportion of the current service is larger than the current data processing proportion comprises the following steps:
judging whether the next data processing proportion of the current service is larger than the current data processing proportion;
if yes, ending the flow; if not, carrying out data processing on the current service according to the next data processing proportion, and judging whether the current service has a successful record of data processing when the current data processing proportion is the last data processing proportion of the current service;
the judging whether the data processing success record exists in the current service or not comprises the following steps:
judging whether the current service has a successful record of data processing or not;
If yes, the data processing is circularly carried out on the data service to be processed of the next batch; if not, the process is ended.
Further, the data processing method further comprises:
when the data processing of the data service to be processed fails, the multilevel data processing executing strategy is adjusted and optimized through a plurality of comparison experiments.
In a second aspect, there is provided a data processing apparatus, the apparatus comprising:
the receiving module is used for receiving an instruction for processing the data schedule;
the integration module is used for screening and cleaning the data schedule according to the product type and writing the screened and cleaned data into a data preprocessing table;
the sending module is used for sending the product data preprocessing table to the message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed;
the configuration module is used for configuring a multilevel data processing execution strategy for each data service to be processed through the configuration platform;
and the processing module is used for carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
In a third aspect, there is provided a computer device, the device comprising:
A memory, a processor and a computer program stored on the memory, the processor executing the computer program to implement the steps of the data processing method according to any of the first aspects.
In a fourth aspect, there is provided a computer storage medium comprising:
on which a computer program is stored which, when being executed by a processor, implements the steps of the data processing method according to any of the first aspects.
Compared with the prior art, the method provided by the embodiment of the application comprises the following steps: receiving an instruction for processing a data schedule; screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table; the method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed; configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform; and carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode. The application can improve the post-credit data expression of the business, improve the success rate of customer repayment, reduce the post-credit bad account data rate and bad account loss, provide diversified post-credit management modes for the collection-urging business, and improve business gross profit.
The technical scheme provided by the embodiment of the application solves the multi-level deduction requirement of large data magnitude by utilizing the data warehouse and the message queue, can flexibly configure the whole solution of the multi-dimensional deduction strategy, and improves the refund rate by utilizing rich execution strategies.
The technical scheme provided by the embodiment of the application can solve the dilemma that in the financial credit business, the repayment is difficult and the post-credit performance is poor under the background of the repayment plan of the extra-large data volume sub-bank sub-table.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a first flowchart of a data processing method according to a first embodiment of the present application;
FIG. 2 is a second flowchart of a data processing method according to a first embodiment of the present application;
FIG. 3 is a third flowchart of a data processing method according to a second embodiment of the present application;
fig. 4 is a flow chart of data screening according to a second embodiment of the present application;
FIG. 5 is a flow chart of data cleansing according to a second embodiment of the present application;
FIG. 6 is a flow chart of a batch distribution provided in a second embodiment of the present application;
FIG. 7 is a flow chart of message queue message processing according to a second embodiment of the present application;
FIG. 8 is a hierarchical withholding asynchronous flow chart provided in accordance with a second embodiment of the present application;
FIG. 9 is a fourth flowchart of a data processing method according to the second embodiment of the present application;
FIG. 10 is a block diagram of a data processing apparatus according to a third embodiment of the present application;
FIG. 11 is an exemplary system that may be used to implement various embodiments described in this disclosure, as provided by embodiment five of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It is noted that, unless the context clearly requires otherwise, the words "comprise," "comprising," and the like throughout the specification and the claims should be construed in an inclusive sense rather than an exclusive or exhaustive sense; that is, it is the meaning of "including but not limited to".
Furthermore, in the description of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more.
Example 1
An embodiment of the present application provides a data processing method, as shown in fig. 1, where the method includes:
receiving an instruction for processing a data schedule;
screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table;
the method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed;
configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform;
and carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
Specifically, as shown in fig. 2, the technical scheme of the present application specifically includes: the credit core system supports partial repayment capabilities; screening and cleaning the data schedule according to the product type through an ODPS script based on a data warehouse ODPS (Open DataProcessing Service), and writing the screened and cleaned data into a data preprocessing table; batch distribution based on message middleware; configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform; and backtracking data, wherein when the data processing of the data service to be processed fails, the multilevel data processing execution strategy is adjusted and optimized through a plurality of comparison experiments.
The technical scheme provided by the embodiment of the application has the beneficial effects that:
the technical scheme provided by the embodiment of the application can improve the post-credit data expression of the business, improve the success rate of customer repayment, reduce the post-credit bad account data rate and bad account loss, provide a diversified post-credit management mode for the collection-promoting business, and improve business profits.
The technical scheme provided by the embodiment of the application solves the multi-level deduction requirement of large data magnitude by utilizing the data warehouse and the message queue, can flexibly configure the whole solution of the multi-dimensional deduction strategy, and improves the refund rate by utilizing rich execution strategies.
The technical scheme provided by the embodiment of the application can solve the dilemma that in the financial credit business, the repayment is difficult and the post-credit performance is poor under the background of the repayment plan of the extra-large data volume sub-bank sub-table.
Example two
An embodiment of the present application provides a data processing method, as shown in fig. 3, where the method includes:
in step S01, an instruction to process the data schedule is received.
Specifically, the credit core system needs to support the partial repayment function firstly, breaks through the limitation of original full payment, for example, the monthly supply amount of a user is 96 yuan (80 yuan +5 yuan interest +10 yuan premium +1 yuan fine, overdue claims), the system needs to support the function of being capable of deducting money firstly and compensating partial amount, and the system can relate to the clearing of multiple subjects (such as 10 yuan premium clearing to an insurance account, 80 yuan +5 yuan interest +1 yuan penalty clearing to a compensation account) and the downstream insurance core system supports the functions of partial premium registration and the like.
Here, the repayment plan data are returned from the repayment plan original table to the data warehouse ODPS in advance D-N days, the data needing batch running on the D-th day are sorted out in the data warehouse ODPS processing data, and since the time from the payment date to the first appointed payment date is generally one month, the date of the appointed payment date which is not the earliest one-time bill of the borrowing can be known in advance, the data needing batch running can be prepared in advance, and the data needing batch running is processed in sequence according to the amount and written into the relational database management system MySQL.
And step S02, screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table.
Specifically, the user allows multiple borrowing under one product, each borrowing has a multi-period repayment plan, along with the continuous increase of the traffic volume, the accumulated data volume to be buckled reaches tens of millions, and if complete volume buckling is initiated regularly, great pressure is caused on a server and a storage system, and great buckling cost is caused, so that the data is necessary to be pre-screened and integrated.
Step S02 further includes:
and step S021, de-duplicating the data schedule according to the product type, and reserving user ID and/or product number and/or order number and/or data processing time field information.
Specifically, as shown in fig. 4, if the original credit system repayment schedule is a structure of a relational database management system mysql+a separate database and separate table, the disadvantage of daily full schedule rotation is obvious, for example: 1, the database pressure is high, and a plurality of batches of read-write operations are carried out on a repayment schedule every day; 2, taking time for a long time, and calculating the duration once in hours, the data warehouse ODPS is considered to be adopted to process target data after preparing D days in advance, meanwhile, because the data which is related to the advanced repayment of a user and prepared in advance may be larger than the data which needs to be replaced on the actual day, parameter information such as user ID, product code, borrowing order number, execution time field and the like is reserved based on the data warehouse ODPS schedule to remove the repayment plan data according to the product type and the borrowing data, and the processed data is written into the preprocessing table.
Step S022, screening a data schedule with the longest overdue time from the data schedules according to the data execution time;
dividing the data schedule with the longest overdue time into a plurality of batch schedules according to the dimension of the data processing order, and arranging the batch schedules in ascending order according to the overdue amount of the data schedule;
And merging the batch schedules with the same overdue days, and taking the data processing order with the shortest overdue days as the account age of the user level.
Specifically, as shown in fig. 5, in the first step, the payment schedule with the earliest overdue is selected from the payment data schedules of all overdue users according to the contracted payment date.
And secondly, the default repayment card is the same under the same product by the user, if the card balance is insufficient, the repayment of all borrowing points is unsuccessful, so that the repayment batch concept is introduced, the repayment schedule is divided into a plurality of batch schedules according to the borrowing dimension, meanwhile, the batches are arranged in ascending order according to the overdue amount of the repayment schedule, the next batch point is carried out after the previous batch point is successful, and otherwise, the process is directly terminated.
Wherein, the borrowing dimension refers to: for example, a user may have three borrowings overdue, and the user may generate three lot numbers based on the borrowing dimensions.
Thirdly, the user has different paying time of a plurality of borrowing, the appointed paying days corresponding to the paying plans are different, the overdue days (account ages) are different, all batch schedules are executed within the same task time, the batch schedules with the same overdue days are required to be combined, and the borrowing list with the shortest account age is used as the account age of the user level; for example, the user borrows A for 5 days and the user borrows B for 25 days, and 5 days is taken as the account age of the user to be matched with the deduction strategy.
Here, the borrowing list with the shortest account age is used as the account age of the user level, so that the user can be conveniently deducted at higher frequency.
Step S03, a product data preprocessing table is sent to a message queue in a multi-consumption thread distribution mode, and a plurality of data services to be processed are filtered and generated.
Specifically, as shown in fig. 6, the batch processing system obtains the product of running batch at the current time point according to the configuration query, then drags the product data from the preprocessing table according to the product number, pushes the product data into the message queue to cut the peak, and after the multi-stage deduction execution task consumes the message in the message queue MQ, the data of the borrowing and repayment schedule is queried according to the separate-bank and separate-table key, the data of the combined account age and the like are processed and matched with the multi-stage deduction rule, and the data which has been successfully repaid is filtered, so that the actual repayment order is generated.
Here, batch processing data is distributed through a message queue, messages are consumed in a plurality of servers of the cluster, the cluster performance is fully utilized, and the transverse expansion of processing capacity is realized by adjusting the number of the servers in the cluster. In particular to a single server in a cluster, when the server resources are sufficient, the processing capacity (including CPU capacity, memory capacity and disk reading and writing capacity) of the server is improved or the resource utilization rate is improved by increasing the consumption thread number; when the server has performance bottleneck, the pressure on the server is reduced by reducing the number of consumption threads, and service downtime is prevented.
Step S04, configuring, by the configuration platform, a multi-level data processing execution policy for each data service to be processed.
Specifically, the configuration platform supports configuration of a multi-stage deduction execution strategy, and supports configuration of account age, deduction channels, deduction time, deduction frequency, minimum deduction amount, trial deduction proportion and the like.
Step S04 further includes:
step S041, configuring multilevel data processing execution strategies for each data service to be processed through a configuration platform;
wherein the multi-stage data processing execution strategy comprises setting account age and/or data processing channel and/or data processing frequency and/or data processing amount and/or data processing proportion.
Specifically, after entering the flow of a single transaction, each data service to be processed is firstly configured with account age (MOB), overdue days (DPD), withholding time point, withholding frequency, minimum withholding amount and withholding ratio.
The trial deduction proportion supports a plurality of configurations, such as 100% and 50%, and 100% of the amount to be returned of the user is deducted in one deduction process, and 50% of the amount to be returned of the user is deducted when the balance is insufficient.
And step S05, carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
Specifically, as shown in fig. 7, a distributed scheduling platform is utilized to trigger a multi-level deduction task, and a task system obtains a running lot at a current time point according to configuration inquiry; the first batch of data of the product is fished from the pretreatment table according to the product number, and is pushed to a message queue, wherein the message queue can set the consumption thread number so as to achieve the effect of controlling the message speed; after consuming the messages in the message queue MQ, checking the execution time, inquiring real-time deduction data according to the user ID, and inquiring the deduction execution strategy from the configuration platform.
Step S05 further includes:
step S051, judging whether the task for carrying out data processing on a plurality of data services to be processed is a received instruction for processing a data schedule;
if yes, judging whether the data processing time of the plurality of data services to be processed spans the day or not; if not, ending the flow;
judging whether the data processing time of the plurality of data services to be processed spans the day or not comprises the following steps:
judging whether the data processing time of a plurality of data services to be processed spans the day or not;
if yes, ending the flow; if not, judging whether the first data processing of the current service is successful.
Step S052, judging whether the first data processing of the current service is successful or not;
if yes, judging whether the current service data processing proportion is the last data processing proportion; if not, judging whether the reason for the failure of processing the current service data is insufficient balance;
the judging whether the current service data processing proportion is the last data processing proportion comprises the following steps:
judging whether the current service data processing proportion is the last data processing proportion or not;
if yes, carrying out data processing on the next batch of data services to be processed, and judging whether all the data services to be processed are processed; if not, carrying out data processing on the current service according to the next data processing proportion;
judging whether all the data services to be processed are processed or not, wherein the judging comprises the steps of;
judging whether all the data services to be processed are processed or not;
if yes, ending the flow; if not, continuing to process the data of the next batch of data service to be processed;
judging whether the reason for failure in processing the current service data is insufficient balance or not, wherein the judging comprises the steps of;
judging whether the reason of failure of processing the service data in the current period is insufficient balance;
If yes, judging whether the next data processing proportion of the current service is larger than the current data processing proportion; if not, ending the flow;
the judging whether the next data processing proportion of the current service is larger than the current data processing proportion comprises the following steps:
judging whether the next data processing proportion of the current service is larger than the current data processing proportion;
if yes, ending the flow; if not, carrying out data processing on the current service according to the next data processing proportion, and judging whether the current service has a successful record of data processing when the current data processing proportion is the last data processing proportion of the current service;
the judging whether the data processing success record exists in the current service or not comprises the following steps:
judging whether the current service has a successful record of data processing or not;
if yes, the data processing is circularly carried out on the data service to be processed of the next batch; if not, the process is ended.
Specifically, as shown in fig. 8, the batch processing system obtains the running batch at the current time point according to the configuration query; the first batch of data of the product is fished from the pretreatment table according to the product number, and the first batch of data is pushed to a message queue for peak clipping; after consuming the messages in the message queue MQ, checking the execution time, inquiring real-time deduction data according to the database and table dividing keys, and checking whether the deduction strategy meets the requirements or not; determining the first deduction amount of the service in the current period according to the deduction proportion of the deduction strategy, and initiating a deduction application after storing a deduction record; monitoring a deduction result, checking whether the deduction period number is completely matched, namely judging whether a task for carrying out data processing on a plurality of data services to be processed is a received instruction for processing a data schedule, and if the task is not the received instruction for processing the data schedule, ending the flow; checking whether the generation deduction time spans the days, only executing the data needed to be executed in the days, and ending the flow if the days are crossed, wherein the generation deduction execution strategy in the next day is different from the generation deduction execution strategy in the same day.
Successful replacement buckling scene: if the current service is not clear, executing the next trial deduction proportion on the current service, and if the current service is the last trial deduction proportion, executing the next batch of service; if the current period is clear, executing the next generation button, namely executing the next generation button in the next batch; if all the overdue numbers have been cleared, the process ends.
Substitution failure scenario: judging whether the reason of the replacement buckle failure is insufficient balance, if not, ending the flow; if the failure cause is insufficient balance, judging whether the next data processing proportion of the current service is larger than the current data processing proportion, if so, ending the flow, and if so, continuing executing the substitute deduction; if the ratio is the last trial button ratio, judging whether all the substitute buttons have successful records in the current period, executing the substitute buttons of the next batch if the successful records exist, and ending the flow if the successful records exist.
The next batch of button replacing repeats the above flow, judges whether to enter a third batch, and so on.
Step S06, when the data processing of the data service to be processed fails, the multi-stage data processing execution strategy is adjusted and optimized through multiple comparison experiments.
Specifically, data backtracking analysis: and (3) performing incremental backflow on the data of the multi-stage deduction repayment table to an ODPS (data warehouse), analyzing failure reasons when the data processing of the data service to be processed fails, analyzing whether the deduction proportion is optimal, and subsequently adjusting and optimizing a multi-stage data processing execution strategy through a plurality of comparison experiments to further improve the multi-stage deduction repayment effect.
FIG. 9 is a flowchart showing a data processing method according to the present application, wherein the method of the present application is implemented by receiving an instruction for processing a data schedule; screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table; the method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed; configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform; and carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode. The application can improve the post-credit data expression of the business, improve the success rate of customer repayment and reduce the post-credit bad account data rate and bad account loss.
The method for processing data provided by the embodiment of the application can also make a plurality of improvements and optimizations on the premise of not deviating from the technical scheme of the application, and the improvements and the optimizations are also considered as the protection scope of the application.
The technical scheme provided by the embodiment of the application has the beneficial effects that:
the technical scheme provided by the embodiment of the application can improve the post-credit data expression of the business, improve the success rate of customer repayment, reduce the post-credit bad account data rate and bad account loss, provide a diversified post-credit management mode for the collection-promoting business, and improve business profits.
The technical scheme provided by the embodiment of the application solves the multi-level deduction requirement of large data magnitude by utilizing the data warehouse and the message queue, can flexibly configure the whole solution of the multi-dimensional deduction strategy, and improves the refund rate by utilizing rich execution strategies.
The technical scheme provided by the embodiment of the application can solve the dilemma that in the financial credit business, the repayment is difficult and the post-credit performance is poor under the background of the repayment plan of the extra-large data volume sub-bank sub-table.
Example III
The application provides a data processing device, as shown in fig. 10, which comprises a receiving module, an integrating module, a sending module, a configuration module, a processing module and an optimizing module.
In this embodiment, the receiving module is configured to receive an instruction for processing the data schedule;
the integration module is used for screening and cleaning the data schedule according to the product type and writing the screened and cleaned data into a data preprocessing table;
the sending module is used for sending the product data preprocessing table to the message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed;
the configuration module is used for configuring a multilevel data processing execution strategy for each data service to be processed through the configuration platform;
and the processing module is used for carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
Further, the integration module is used for de-duplicating the data schedule according to the product type, and reserving user ID and/or product number and/or order number and/or data processing time field information.
Further, the integration module is further used for screening out a data schedule with the longest overdue time from the data schedules according to the data execution time;
dividing the data schedule with the longest overdue time into a plurality of batch schedules according to the dimension of the data processing order, and arranging the batch schedules in ascending order according to the overdue amount of the data schedule;
And merging the batch schedules with the same overdue days, and taking the data processing order with the shortest overdue days as the account age of the user level.
Further, the configuration module is used for configuring a multilevel data processing execution strategy for each data service to be processed through the configuration platform;
wherein the multi-stage data processing execution strategy comprises setting account age and/or data processing channel and/or data processing frequency and/or data processing amount and/or data processing proportion.
Further, the processing module is further used for judging whether the task for carrying out data processing on the plurality of data services to be processed is a received instruction for processing the data schedule;
if yes, judging whether the data processing time of the plurality of data services to be processed spans the day or not; if not, ending the flow;
judging whether the data processing time of the plurality of data services to be processed spans the day or not comprises the following steps:
judging whether the data processing time of a plurality of data services to be processed spans the day or not;
if yes, ending the flow; if not, judging whether the first data processing of the current service is successful.
Further, the processing module is also used for judging whether the first data processing of the current service is successful or not;
If yes, judging whether the current service data processing proportion is the last data processing proportion; if not, judging whether the reason for the failure of processing the current service data is insufficient balance;
the judging whether the current service data processing proportion is the last data processing proportion comprises the following steps:
judging whether the current service data processing proportion is the last data processing proportion or not;
if yes, carrying out data processing on the next batch of data services to be processed, and judging whether all the data services to be processed are processed; if not, carrying out data processing on the current service according to the next data processing proportion;
judging whether all the data services to be processed are processed or not, wherein the judging comprises the steps of;
judging whether all the data services to be processed are processed or not;
if yes, ending the flow; if not, continuing to process the data of the next batch of data service to be processed;
judging whether the reason for failure in processing the current service data is insufficient balance or not, wherein the judging comprises the steps of;
judging whether the reason of failure of processing the service data in the current period is insufficient balance;
if yes, judging whether the next data processing proportion of the current service is larger than the current data processing proportion; if not, ending the flow;
The judging whether the next data processing proportion of the current service is larger than the current data processing proportion comprises the following steps:
judging whether the next data processing proportion of the current service is larger than the current data processing proportion;
if yes, ending the flow; if not, carrying out data processing on the current service according to the next data processing proportion, and judging whether the current service has a successful record of data processing when the current data processing proportion is the last data processing proportion of the current service;
the judging whether the data processing success record exists in the current service or not comprises the following steps:
judging whether the current service has a successful record of data processing or not;
if yes, the data processing is circularly carried out on the data service to be processed of the next batch; if not, the process is ended.
And the optimizing module is used for adjusting and optimizing the multilevel data processing executing strategy through a plurality of comparison experiments when the data processing of the data service to be processed fails.
The technical scheme provided by the embodiment of the application has the beneficial effects that:
the technical scheme provided by the embodiment of the application can improve the post-credit data expression of the business, improve the success rate of customer repayment, reduce the post-credit bad account data rate and bad account loss, provide a diversified post-credit management mode for the collection-promoting business, and improve business profits.
The technical scheme provided by the embodiment of the application solves the multi-level deduction requirement of large data magnitude by utilizing the data warehouse and the message queue, can flexibly configure the whole solution of the multi-dimensional deduction strategy, and improves the refund rate by utilizing rich execution strategies.
The technical scheme provided by the embodiment of the application can solve the dilemma that in the financial credit business, the repayment is difficult and the post-credit performance is poor under the background of the repayment plan of the extra-large data volume sub-bank sub-table.
Example IV
The application provides a computer device comprising a memory, a processor and a computer program stored on the memory and operable on the processor, the processor executing the computer program to perform the following data processing method:
receiving an instruction for processing a data schedule;
screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table;
the method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed;
configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform;
And carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
The technical scheme provided by the embodiment of the application has the beneficial effects that:
the technical scheme provided by the embodiment of the application can improve the post-credit data expression of the business, improve the success rate of customer repayment, reduce the post-credit bad account data rate and bad account loss, provide a diversified post-credit management mode for the collection-promoting business, and improve business profits.
The technical scheme provided by the embodiment of the application solves the multi-level deduction requirement of large data magnitude by utilizing the data warehouse and the message queue, can flexibly configure the whole solution of the multi-dimensional deduction strategy, and improves the refund rate by utilizing rich execution strategies.
Example five
The application provides a computer storage medium comprising a memory, a processor and a computer program stored on the memory and running on the processor, the processor executing the computer program to realize the following steps:
receiving an instruction for processing a data schedule;
screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table;
The method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed;
configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform;
and carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
Further, the screening and cleaning the data schedule according to the product type includes:
and de-duplicating the data schedule according to the product type, and reserving user ID and/or product number and/or order number and/or data processing time field information.
Further, the screening and cleaning the data schedule according to the product type further includes:
screening a data schedule with the longest overdue time from the data schedules according to the data execution time;
dividing the data schedule with the longest overdue time into a plurality of batch schedules according to the dimension of the data processing order, and arranging the batch schedules in ascending order according to the overdue amount of the data schedule;
and merging the batch schedules with the same overdue days, and taking the data processing order with the shortest overdue days as the account age of the user level.
Further, the configuring, by the configuration platform, a multi-level data processing execution policy for each data service to be processed includes:
configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform;
wherein the multi-stage data processing execution strategy comprises setting account age and/or data processing channel and/or data processing frequency and/or data processing amount and/or data processing proportion.
Further, the data processing of the plurality of data services to be processed according to the multi-stage data processing execution policy and the multi-batch processing mode includes:
judging whether the task for carrying out data processing on a plurality of data services to be processed is a received instruction for processing a data schedule;
if yes, judging whether the data processing time of the plurality of data services to be processed spans the day or not; if not, ending the flow;
judging whether the data processing time of the plurality of data services to be processed spans the day or not comprises the following steps:
judging whether the data processing time of a plurality of data services to be processed spans the day or not;
if yes, ending the flow; if not, judging whether the first data processing of the current service is successful.
Further, the determining whether the first data processing of the current service is successful includes:
judging whether the first data processing of the current service is successful or not;
if yes, judging whether the current service data processing proportion is the last data processing proportion; if not, judging whether the reason for the failure of processing the current service data is insufficient balance;
the judging whether the current service data processing proportion is the last data processing proportion comprises the following steps:
judging whether the current service data processing proportion is the last data processing proportion or not;
if yes, carrying out data processing on the next batch of data services to be processed, and judging whether all the data services to be processed are processed; if not, carrying out data processing on the current service according to the next data processing proportion;
judging whether all the data services to be processed are processed or not, wherein the judging comprises the steps of;
judging whether all the data services to be processed are processed or not;
if yes, ending the flow; if not, continuing to process the data of the next batch of data service to be processed;
judging whether the reason for failure in processing the current service data is insufficient balance or not, wherein the judging comprises the steps of;
Judging whether the reason of failure of processing the service data in the current period is insufficient balance;
if yes, judging whether the next data processing proportion of the current service is larger than the current data processing proportion; if not, ending the flow;
the judging whether the next data processing proportion of the current service is larger than the current data processing proportion comprises the following steps:
judging whether the next data processing proportion of the current service is larger than the current data processing proportion;
if yes, ending the flow; if not, carrying out data processing on the current service according to the next data processing proportion, and judging whether the current service has a successful record of data processing when the current data processing proportion is the last data processing proportion of the current service;
the judging whether the data processing success record exists in the current service or not comprises the following steps:
judging whether the current service has a successful record of data processing or not;
if yes, the data processing is circularly carried out on the data service to be processed of the next batch; if not, the process is ended.
Further, the data processing method further comprises:
when the data processing of the data service to be processed fails, the multilevel data processing executing strategy is adjusted and optimized through a plurality of comparison experiments.
The application can improve the post-credit data expression of the business, improve the success rate of customer repayment and reduce the post-credit bad account data rate and bad account loss.
FIG. 11 is an exemplary system that may be used to implement various embodiments described in this disclosure, provided by embodiment five of the present application;
as shown in fig. 11, in some embodiments, the system can be implemented as any of the above-described devices for data processing in any of the described embodiments. In some embodiments, a system may include one or more computer-readable media (e.g., system memory or NVM/storage) having results and one or more processors (e.g., processor (s)) coupled with the one or more computer-readable media and configured to execute the results to implement the modules to perform the actions described in this disclosure.
Those skilled in the art will appreciate that implementing all or part of the above-described embodiment methods may be accomplished by result-dependent hardware by a computer program stored on a non-transitory computer readable storage medium, which when executed may comprise the embodiment methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims (10)

1. A method of data processing, the method comprising:
receiving an instruction for processing a data schedule;
screening and cleaning the data schedule according to the product type, and writing the screened and cleaned data into a data preprocessing table;
the method comprises the steps of sending a product data preprocessing table to a message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed;
Configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform;
and carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
2. The data processing method according to claim 1, wherein the screening and cleaning of the data schedule according to the product type includes:
and de-duplicating the data schedule according to the product type, and reserving user ID and/or product number and/or order number and/or data processing time field information.
3. The data processing method according to claim 1, wherein the screening and cleaning of the data schedule according to the product type further comprises:
screening a data schedule with the longest overdue time from the data schedules according to the data execution time;
dividing the data schedule with the longest overdue time into a plurality of batch schedules according to the dimension of the data processing order, and arranging the batch schedules in ascending order according to the overdue amount of the data schedule;
and merging the batch schedules with the same overdue days, and taking the data processing order with the shortest overdue days as the account age of the user level.
4. The data processing method according to claim 1, wherein configuring, by the configuration platform, a multi-level data processing execution policy for each data service to be processed, comprises:
configuring a multilevel data processing execution strategy for each data service to be processed through a configuration platform;
wherein the multi-stage data processing execution strategy comprises setting account age and/or data processing channel and/or data processing frequency and/or data processing amount and/or data processing proportion.
5. The data processing method according to claim 1, wherein the data processing of the plurality of data services to be processed according to the multi-level data processing execution policy and the multi-batch processing mode comprises:
judging whether the task for carrying out data processing on a plurality of data services to be processed is a received instruction for processing a data schedule;
if yes, judging whether the data processing time of the plurality of data services to be processed spans the day or not; if not, ending the flow;
judging whether the data processing time of the plurality of data services to be processed spans the day or not comprises the following steps:
judging whether the data processing time of a plurality of data services to be processed spans the day or not;
If yes, ending the flow; if not, judging whether the first data processing of the current service is successful.
6. The data processing method according to claim 5, wherein the determining whether the first data processing for the current service is successful comprises:
judging whether the first data processing of the current service is successful or not;
if yes, judging whether the current service data processing proportion is the last data processing proportion; if not, judging whether the reason for the failure of processing the current service data is insufficient balance;
the judging whether the current service data processing proportion is the last data processing proportion comprises the following steps:
judging whether the current service data processing proportion is the last data processing proportion or not;
if yes, carrying out data processing on the next batch of data services to be processed, and judging whether all the data services to be processed are processed; if not, carrying out data processing on the current service according to the next data processing proportion;
judging whether all the data services to be processed are processed or not, wherein the judging comprises the steps of;
judging whether all the data services to be processed are processed or not;
If yes, ending the flow; if not, continuing to process the data of the next batch of data service to be processed;
judging whether the reason for failure in processing the current service data is insufficient balance or not, wherein the judging comprises the steps of;
judging whether the reason of failure of processing the service data in the current period is insufficient balance;
if yes, judging whether the next data processing proportion of the current service is larger than the current data processing proportion; if not, ending the flow;
the judging whether the next data processing proportion of the current service is larger than the current data processing proportion comprises the following steps:
judging whether the next data processing proportion of the current service is larger than the current data processing proportion;
if yes, ending the flow; if not, carrying out data processing on the current service according to the next data processing proportion, and judging whether the current service has a successful record of data processing when the current data processing proportion is the last data processing proportion of the current service;
the judging whether the data processing success record exists in the current service or not comprises the following steps:
judging whether the current service has a successful record of data processing or not;
if yes, the data processing is circularly carried out on the data service to be processed of the next batch; if not, the process is ended.
7. The data processing method of claim 1, wherein the method further comprises:
when the data processing of the data service to be processed fails, the multilevel data processing executing strategy is adjusted and optimized through a plurality of comparison experiments.
8. A data processing apparatus, the apparatus comprising:
the receiving module is used for receiving an instruction for processing the data schedule;
the integration module is used for screening and cleaning the data schedule according to the product type and writing the screened and cleaned data into a data preprocessing table;
the sending module is used for sending the product data preprocessing table to the message queue in a multi-consumption thread distribution mode, filtering and generating a plurality of data services to be processed;
the configuration module is used for configuring a multilevel data processing execution strategy for each data service to be processed through the configuration platform;
and the processing module is used for carrying out data processing on a plurality of data services to be processed according to the multi-stage data processing execution strategy and the multi-batch processing mode.
9. A computer device comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to carry out the steps of the data processing method according to any one of claims 1 to 7.
10. A computer storage medium on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 7.
CN202311004301.7A 2023-08-10 2023-08-10 Data processing method, device, equipment and medium Pending CN117057903A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311004301.7A CN117057903A (en) 2023-08-10 2023-08-10 Data processing method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311004301.7A CN117057903A (en) 2023-08-10 2023-08-10 Data processing method, device, equipment and medium

Publications (1)

Publication Number Publication Date
CN117057903A true CN117057903A (en) 2023-11-14

Family

ID=88654586

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311004301.7A Pending CN117057903A (en) 2023-08-10 2023-08-10 Data processing method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN117057903A (en)

Similar Documents

Publication Publication Date Title
US20020138376A1 (en) Multi-processing financial transaction processing system
CN100375038C (en) Finally agent optimization method and system of source in reorder two-stage refering
Hashimoto et al. Asset bubbles, unemployment, and a financial crisis
CN114186989A (en) Capital allocation decision method, device, server and storage medium
CN109614263B (en) Disaster tolerance data processing method, device and system
CN117057903A (en) Data processing method, device, equipment and medium
CN116467352A (en) Transaction inquiry system
CN111429125B (en) Account management method and device, storage medium and electronic equipment
KR102148152B1 (en) Processing method and apparatus for financial instrument information
CN114971637A (en) Risk early warning method, device, equipment and medium
CN114511314A (en) Payment account management method and device, computer equipment and storage medium
CN113610518A (en) Bank card arrearage additional payment processing method and device based on online transaction triggering
CN113377823A (en) Value data processing method, device, equipment and storage medium
CN112035503A (en) Transaction data updating method and device
CN110766546A (en) Bank account management method
US20240054130A1 (en) Systems and methods for scheduling information retrieval
CN106157015A (en) The method and device of fund recovery
CN116382924B (en) Recommendation method and device for resource allocation, electronic equipment and storage medium
CN115311065A (en) Splitting and money returning method and device
CN111626871B (en) Data processing method, device, equipment and storage medium
CN107730381B (en) Method and device for backing up cross section data
CN114186267A (en) Virtual asset data processing method and device and computer readable storage medium
CN116382871A (en) Information change identification and task rerun method, device, equipment, medium and product
CN115860957A (en) Financing and purchasing method and device and electronic equipment
CN115510827A (en) Data processing method, device, equipment and medium based on line data type label

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination