CN110659999A - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN110659999A CN110659999A CN201910813409.8A CN201910813409A CN110659999A CN 110659999 A CN110659999 A CN 110659999A CN 201910813409 A CN201910813409 A CN 201910813409A CN 110659999 A CN110659999 A CN 110659999A
- Authority
- CN
- China
- Prior art keywords
- data
- loan
- processing
- target post
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/03—Credit; Loans; Processing thereof
Abstract
The embodiment of the specification provides a data processing method, a data processing device and electronic equipment, wherein the method comprises the following steps: the method comprises the steps of obtaining target post-loan data to be processed, processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset service rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different service classes, wherein the service classes are determined based on the loan time limit related information corresponding to the target post-loan data, and then outputting the processing results of the target post-loan data of different service classes. Therefore, the PB-level data can be rapidly inquired and analyzed by using an Impala memory processing mechanism in the Hadoop big data platform, the data processing efficiency is effectively improved, and the data processing efficiency is improved by more than 60% compared with the preprocessing efficiency. And further provides powerful guarantee for the development of services such as overdue collection after loan of insurance institutions, loan expiration reminding and the like.
Description
Technical Field
The present invention relates to the field of big data processing technologies, and in particular, to a data processing method and apparatus, and an electronic device.
Background
In recent years, big data processing and analysis have become global problems, and with the continuous improvement of the informatization and automation level of the economy and the society in China, big data problems are also faced in many fields such as governments, public services, scientific research and commercial application.
At present, with the rapid development of insurance strategic business of insurance loan insurance assurance, the number of banks to which insurance business processing systems of insurance companies guarantee insurance business is continuously increased, the business volume is multiplied and the data volume is sharply increased, so that the execution time of each system flow is long, the report inquiry is slow, and the generation timeliness of data cannot be guaranteed by adopting the original SQL statement and storage process to realize batch processing tasks, so that the development work of insurance institutions on services such as overdue collection after loan, loan due reminding and the like is seriously influenced.
Disclosure of Invention
The embodiment of the invention aims to provide a data processing method, a data processing device and electronic equipment, and aims to solve the problems that in the prior art, in the face of sharp increase of data volume, the generation timeliness of data cannot be guaranteed by adopting the original SQL statement and the mode of realizing batch processing tasks in the storage process, so that the development of services such as overdue collection after loan, loan expiration reminding and the like by an insurance institution is seriously influenced.
In order to solve the above technical problem, the embodiment of the present invention is implemented as follows:
in a first aspect, an embodiment of the present invention provides a data processing method, including:
acquiring target post-loan data to be processed;
processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset business rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different business categories, wherein the business categories are determined based on the related information of loan terms corresponding to the target post-loan data;
and outputting the processing result of the target credited data of different service classes.
Optionally, the service category includes: one or more of overdue hastening, due reminders, claim settlement processing, and post-loan status processing.
Optionally, the outputting the processing result of the target credited data of the different service categories includes:
and outputting the processing results of the target credited data of different service classes to a preset database for storage, and/or writing the processing results of the target credited data of different service classes into a preset display page.
Optionally, the method further comprises:
and pushing the corresponding processing result of the target post-loan data to a predetermined processing system according to the service category corresponding to the processing result of the target post-loan data, wherein the predetermined processing system comprises one or more of a claim settlement system, an underwriting system, a repayment reminding system, a credit investigation system and a loan acceptance system.
Optionally, the outputting the processing result of the target credited data of different service classes to a predetermined database for storage includes:
and connecting the Java database with a JDBC write-back mechanism, and writing the processing result of the target credited data of different service classes into a preset Informix database.
Optionally, after the obtaining of the target post-loan data to be processed, the method further includes:
and calling a preset Shell script to store the acquired target post-loan data into a Kudu-based distributed column storage system on a Hadoop big data platform.
Optionally, before the acquiring target post-loan data to be processed, the method further includes:
obtaining source credit data;
and processing the obtained source post-credit data to obtain the target post-credit data with different service types in a preset data format.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, including:
the target data acquisition module is used for acquiring target post-loan data to be processed;
the processing result generation module is used for processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset service rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different service types, wherein the service types are determined based on the related information of the loan term corresponding to the target post-loan data;
and the processing result output module is used for outputting the processing result of the target credited data of different service classes.
Optionally, the service category includes: one or more of overdue hastening, due reminders, claim settlement processing, and post-loan status processing.
Optionally, the processing result output module is configured to:
and outputting the processing results of the target credited data of different service classes to a preset database for storage, and/or writing the processing results of the target credited data of different service classes into a preset display page.
Optionally, the apparatus further includes a processing result pushing module, configured to:
and pushing the corresponding processing result of the target post-loan data to a predetermined processing system according to the service category corresponding to the processing result of the target post-loan data, wherein the predetermined processing system comprises one or more of a claim settlement system, an underwriting system, a repayment reminding system, a credit investigation system and a loan acceptance system.
Optionally, the processing result output module is configured to:
and connecting the Java database with a JDBC write-back mechanism, and writing the processing result of the target credited data of different service classes into a preset Informix database.
Optionally, the apparatus further comprises: a target data storage module to:
and calling a preset Shell script to store the acquired target post-loan data into a Kudu-based distributed column storage system on a Hadoop big data platform.
Optionally, the apparatus further comprises a target data generation module, configured to:
obtaining source credit data;
and processing the obtained source post-credit data to obtain the target post-credit data with different service types in a preset data format.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus; the processor, the communication interface and the memory complete mutual communication through a bus; the memory is used for storing a computer program; the processor is configured to execute the program stored in the memory to implement the steps of the data processing method according to the first aspect.
In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data processing method according to the first aspect are implemented.
An embodiment of the specification provides a data processing method, a data processing device and electronic equipment, wherein the method comprises the following steps: the method comprises the steps of obtaining target post-loan data to be processed, processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset service rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different service classes, wherein the service classes are determined based on the loan time limit related information corresponding to the target post-loan data, and then outputting the processing results of the target post-loan data of different service classes. Therefore, according to the target post-loan data and the preset business rules corresponding to the target post-loan data, the Impala memory processing mechanism in the Hadoop big data platform is used for processing the target post-loan data, PB-level data can be rapidly inquired and analyzed, the data processing efficiency is effectively improved, and powerful guarantee is provided for the development of services such as overdue collection after loan of insurance institutions, loan expiration reminding and the like.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a first schematic flow chart of a data processing method provided in an embodiment of the present disclosure;
fig. 2 is a second schematic flow chart of a data processing method provided in an embodiment of the present disclosure;
fig. 3 is a third schematic flow chart of a data processing method provided in an embodiment of the present disclosure;
fig. 4 is a fourth schematic flowchart of a data processing method provided in an embodiment of the present disclosure;
fig. 5 is a fifth flowchart of a data processing method provided in an embodiment of the present disclosure;
fig. 6 is a sixth flowchart of a data processing method provided in an embodiment of the present disclosure;
FIG. 7 is a system flow diagram of a data processing method provided by an embodiment of the present description;
fig. 8 is a schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 9 is a schematic structural diagram of an electronic device provided in an embodiment of this specification.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step should fall within the scope of protection of the present specification.
Currently, each insurance institution widely carries out loan insurance business, so that the volume of business data to be processed is increased sharply. The parallel data flow of the post-loan business to be processed comprises the steps of generating an overdue collection urging and expiration reminding task in the Check storage process, starting short message logic processing and sending short messages aiming at the overdue and expiration reminding task, cancelling policy keeping processing after no payment is released for 7 days, checking and approving a bank without batch quitting, reporting a credit at a central bank, prompting the storage process for expiration, clearing a service of clearing a claim, clearing the service of clearing the service in advance, carrying out penalty calculation on the monthly commission batch task, carrying out batch validation operation on the policy by bank payment and the like.
However, in the conventional technical solution, when the loan insurance business is processed, the business process is performed by calling different SQL processing logic and stored procedures through Java program codes. As each financial institution (such as a bank and the like) is provided with a corresponding storage process and a corresponding code flow, the data volume is increased rapidly with the increase of the number of the butted financial institutions, so that the data processing logic is more complex, the report inquiry is slow, and each flow needs to complete the scheduling processing in a hard coding mode. The data processing time is longer and longer to a certain extent, the requirements of quick decision making and data use of management institutions of all levels of insurance institutions cannot be met, and along with the increase of the number of the connected financial institutions and the improvement of business complexity, the defects of the development and portability of the storage process are more obvious, and the data processing efficiency and performance are low.
Hadoop is a distributed system infrastructure. The main goal of Hadoop is to process "big data" in a distributed environment in a reliable, efficient, scalable manner. The Hadoop system mainly comprises an Impala memory calculation engine, a Yarn resource scheduling platform, Hive data warehouse management, Kudu data storage, Hdfs original data storage and the like. The data processing is carried out based on an Impala memory calculation engine and Kudu data storage in Hadoop, all processes can be realized in Hadoop due to the fact that Kudu storage data support data updating operation, frequent interaction of interfaces is not needed, and frequent leading-in of data is not needed.
In addition, Hadoop is a software framework capable of performing distributed processing on a large amount of data, and is a distributed computing platform which can be easily constructed and used by users. The Hadoop supports development and running of application programs and components for processing mass data, and the Hadoop environment is a basic environment for running all components, so that calculation and processing of various tasks related to the data processing method provided by the embodiment of the specification can be performed under the Hadoop.
The embodiment of the specification applies a Hadoop big data technology to improve data processing timeliness and powerfully improve the concurrent generation efficiency of data processing, so that the progress of rapid business decision of the insurance industry or other loan industries is further guaranteed, and the business decision efficiency is further guaranteed.
As shown in fig. 1, an execution subject of the method may be a server, where the server may be an independent server, or a server cluster composed of multiple servers. The method can be used for carrying out rapid data processing on PB level data.
Fig. 1 is a first schematic flowchart of a data processing method provided in an embodiment of the present disclosure, and as shown in fig. 1, the method at least includes the following steps:
and S101, acquiring target post-loan data to be processed.
The target post-loan data may be target post-loan data of different business categories in a predetermined data format, which is obtained by uniformly processing data files generated by a plurality of financial institutions (such as banks) and acquired by an insurance agency. The target post-loan data may be: one or more of loan data, approval data, loan date supply data, loan repayment data, claim settlement application data, claim settlement processing data, premium date supply data, premium repayment data, reconciliation data, compensation withholding data, premium withholding failure data, withholding fine details data, and withholding default fund data upon advance repayment.
S102, processing the target post-loan data by using an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset business rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different business categories, wherein the business categories are determined based on the related information of the loan term corresponding to the target post-loan data.
The predetermined business rule may be a business rule preset for a certain type of loan service, for example: for the service category of the due reminding, the preset service rule can be that 5 days before the due, short messages are sent to the loan users about to expire every day to remind the loan users to pay back the loans in time. The related information of the loan term can be the information of the loan term supply and payment, the information of the prior settlement, the settlement and the like.
In implementation, because Kudu is a column-type storage system running on a Hadoop big data platform, data updating operation is supported while data is stored, all processes can be implemented by being placed on the Hadoop big data platform, frequent interaction of interfaces is not needed, frequent leading-in of data is not needed, high-performance storage equipment is used, and the advantages of cross-region real-time data backup and query are supported, so that the obtained target credited data can be led into Kudu in the Hadoop big data platform.
In addition, since Impala is used as an SQL (Structured Query Language) parsing engine, the stability and speed of Query class requests are widely verified in the industry, and Impala does not have its own storage engine, which is responsible for parsing SQL and connecting to its underlying storage engine. At the beginning of release, the Impala mainly supports Hdfs, and after Kudu release, the Impala and Kudu are deeply integrated. Based on this, in the present embodiment, a method of integrating Impala and Kudu is adopted, and Impala is used to process the target post-loan data stored in Kudu.
In implementation, when the acquired target credited data to be processed is stored in Kudu, after the Kudu environment data is prepared, SQL required by complex business processing can be processed by starting an Impala memory calculation engine.
Specifically, according to the target post-loan data and the preset business rules corresponding to the target post-loan data, data results required by each process link such as due reminding, overdue hastening and post-loan state statistics are obtained through corresponding SQL statements and data operation and processing of Impala, wherein Impala can be calculated based on a memory, interactive real-time query and analysis can be performed on PB-level data, and data processing efficiency is effectively improved.
In implementation, for example, Impala uses example code as follows:
String LoanAfterSql＝"SELECT DISTINCT"
......
+"re.comcode comcode,"
+"uwt.insuredname,"
+"com.comcname,"
+"re.proposalno proposalno,"
+"re.policyno policyno,"
+"loan.loanamt,"
+"FROM prpdloaninfo loan"
+"JOIN re_pproposal re"
+"ON trim(re.policyno)＝trim(loan.guarantyid)"
+"JOIN uwtmain uwt"
+"ON trim(re.proposalno)＝trim(uwt.proposalno)"
......
+"where loan.corpbankcode＝'"+bankcode+"'"；
ps＝impalaConn.prepareStatement(LoanAfterSql.toString())；
rs＝ps.executeQuery()；
s103, outputting the processing result of the target credited data of different service types, wherein the service types comprise: one or more of overdue hastening, due reminders, claim settlement processing, and post-loan status processing.
In implementation, the processing result of the obtained target credited data of different service classes can be output to a predetermined database (such as an Informix database) for persistent storage, so as to be used by subsequent services. The obtained target credited data of different service types can be pushed to other systems by using an interface according to a specific service rule. In addition, the processing result of the obtained target credited data of different service types can be assembled to form a report required by the corresponding service according to the service rule, so that the service staff can work efficiently.
In the embodiment of the description, target post-loan data to be processed is acquired, the target post-loan data is processed by using an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset service rule corresponding to the target post-loan data, processing results of the target post-loan data of different service classes are obtained, wherein the service classes are determined based on the loan time limit related information corresponding to the target post-loan data, and then the processing results of the target post-loan data of different service classes are output. Therefore, according to the target post-loan data and the preset business rules corresponding to the target post-loan data, the Impala memory processing mechanism in the Hadoop big data platform is used for processing the target post-loan data, PB-level data can be rapidly inquired and analyzed, the data processing efficiency is effectively improved, and powerful guarantee is provided for the development of services such as overdue collection after loan of insurance institutions, loan expiration reminding and the like.
As shown in fig. 2, the specific processing manner of S103 may be various, and an alternative processing manner is provided below, which may specifically refer to the processing of S1031 to S1032 below.
And S1031, outputting the processing results of the target credited data of different service classes to a preset database for storage, and/or S1032, writing the processing results of the target credited data of different service classes into a preset display page, wherein the preset database can be an Informix database.
As shown in fig. 3, the method further includes:
and S104, pushing the corresponding processing result of the target credited data to a preset processing system according to the service class corresponding to the processing result of the target credited data.
The preset processing system comprises one or more of a claim settlement system, an underwriting system, a repayment reminding system, a credit investigation system and a loan acceptance prompting system, wherein the repayment reminding system can be a short message reminding system, and the loan acceptance prompting system can be an intelligent voice acceptance prompting system.
In implementation, the processing result of the target credited data of different service types output in S103 may be directly pushed to a predetermined processing system according to the service type corresponding to the processing result of the target credited data. The processing result of the target credited data of different service types output by the step S103 may also be persistently stored in the Informix database, and then the corresponding processing result of the target credited data is pushed to a predetermined processing system from the Informix database according to the service type corresponding to the processing result of the target credited data.
As shown in fig. 4, the specific processing manner of S1031 may be various, and an optional processing manner is provided below, which may specifically refer to the processing of S10311 described below.
And S10311, connecting the JDBC write-back mechanism through the Java database, and writing the processing results of the data after target lending of different service classes into a preset Informix database.
The sending of the SQL statement to various relational data becomes more convenient and faster through a JDBC (Java DataBase Connectivity) write-back mechanism. By adopting the JDBC write-back mechanism, a Program does not need to be specially written for accessing a Sybase database, a Program is specially written for accessing an Oracle database, another Program is written for accessing an Informix database, and the like, and when the processing results of the target credited data of different service classes are output to a preset database for storage, only one Program needs to be written by using a JDBC API (Application Program Interface), the JDBC write-back mechanism can send SQL to the preset database and write the processing results of the target credited data of different service classes into the preset Informix database or the Oracle database and the like. In addition, through a JDBC write-back mechanism, report data can be displayed for a subsequent system page, and persistent data storage can be well performed when the system interacts with other business systems.
As shown in fig. 5, after the step S101 acquires the target post-loan data to be processed, the method further includes:
and S105, calling a preset Shell script to store the obtained target credited data into a Kudu column-type storage system on a Hadoop big data platform.
Kudu is used as bottom storage, good Scan performance is kept while high-concurrency low-delay kv query is supported, the characteristic enables the system to theoretically consider both OLTP (On-Line Transaction Processing) and OLAP (On-Line Analytical Processing) query, Kudu is a column storage manager made for Hadoop system environment, and the system has the functions of running On general hardware, horizontal expansibility, high-availability operation support and the like as other applications in general Hadoop ecological environment, and based On the functions, the obtained target post-credit data can be stored in a Kudu-based distributed storage system On a Hadoop big data platform.
In an implementation, the obtained target post-loan data stored in the Informix database can be stored in Kudu of Hadoop using Shell derivative script, for example, the import script for examining and approving the information table prpdloannifo by a financial institution (e.g., bank) is as follows:
parameter configuration for driving and checking database
-CONNECT
"jdbc:informix-sqli://10.133.201.77:10001/blazedundb:informixserver＝gdfx_hdr；
NEWLOCALE＝zh_cn,en_us；NEWCODESET＝gbk,8859-1,819；IFX_USE_STREN
C＝true"
--username ccpqry--password-alias mydb.password.alias--query"SELECT*,rowid
FROM prpdloaninfo
WHERE\$CONDITIONS and operatetime between extend(date(today),year to second)and extend(date(today)+1,year to second)-interval(1)second to second"
--fields-terminated-by'\001'--lines-terminated-by'\n'-m 1--target-dir/user/hive/warehouse/prpdloaninfo--hive-table prpdloaninfo--append；
upsert INTO prpdloaninfo
SELECT*FROM hive_prpdloaninfo；
……
Specifically, the Shell derivative script can be used for importing the acquired target post-loan data to be processed into Kudu of Hadoop, calling a preset Shell script, uploading the Shell script to a file directory corresponding to an application server, then sending an instruction to the server through a Java program code, and running the script to complete the dump of the target post-loan data from an Informix database to Kudu:
as shown in fig. 6, before the target post-loan data to be processed is acquired in S101, the following processes S099 to S100 may be executed.
S099, obtaining source credited data, where the source credited data may be a data file returned by a financial institution (e.g., a bank).
Specifically, a financial institution (such as a bank) stores a bank File to an SFTP (Secure File Transfer Protocol) server in a TXT form every morning, where the data File includes: one or more of loan data, approval data, loan date supply data, loan repayment data, claim settlement application data, claim settlement processing data, premium date supply data, premium repayment data, reconciliation data, compensation deduction data, premium deduction failure data, deduction policy detail file, and deduction default data in advance repayment.
In implementation, the source credited data can be acquired from the SFTP server through the bank security module, the acquired source credited data is analyzed through the Java application program, and then the analyzed data can be stored in a bank base table in the Informix database for persistent storage, so that consistency between the source credited data provided by the financial institution and the data stored in the Informix database is ensured, and accuracy of subsequent process processing is ensured.
And S100, processing the obtained source credited data to obtain target credited data with different service types in a preset data format.
In implementation, in consideration of the fact that each financial institution is set with a corresponding storage process, the stored data is preset with a corresponding format standard, so that after the insurance institution or the lending company acquires the source lending data provided by the multiple financial institutions, the acquired source lending data provided by the multiple financial institutions needs to be processed to obtain target lending data of different service classes in a predetermined data format.
As shown in fig. 7, fig. 7 is a system flowchart of a data processing method provided in an embodiment of this specification, where a specific process of data processing may be:
(1) source credited data is obtained, wherein the source credited data can be a data file returned by a financial institution (such as a bank).
(2) The acquired source credited data can be analyzed through the bank security service, and the analyzed source credited data is processed to obtain target credited data of different service types in a preset data format.
(3) And importing the obtained target credited data of different service types in the preset data format into a financial institution basic table in a preset database (such as an insurance company insurance service processing system Informix database).
(4) And calling a preset Shell script, and dumping the target post-loan data which is pre-stored in the fusion mechanism basic table in the Informix database of the insurance company insurance business processing system into a Kudu-based distributed column type storage system on a Hadoop big data platform.
(5) Enabling an Impala memory calculation engine to perform data processing according to the target post-loan data and a preset business rule corresponding to the target post-loan data, and obtaining a data processing result, wherein the data processing result comprises: and 7 days, auditing concrete form data required by the services such as failure tasks, expiration reminding, overdue insurance policy processing, post-loan state processing and the like or required by the process.
(6) And storing the data processing result into a predetermined basic table in a predetermined database (such as an insurance company insurance service processing system Informix database) through a JDBC write-back mechanism.
(7) And pushing the data processing result in the predetermined basic table to a predetermined processing system (such as an insurance company insurance business processing system) by using an interface according to a specific business rule, wherein the predetermined processing system comprises one or more of a claim settlement system, an underwriting system, a loan repayment reminding system, a credit investigation system and a loan acceptance system. And reporting and counting can be performed according to specific business rules (for example, a report required by corresponding business is assembled and insured), and further, business personnel can be helped to work efficiently through the report required by the business.
The data processing method in the embodiment of the present specification includes: the method comprises the steps of obtaining target post-loan data to be processed, processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset service rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different service classes, wherein the service classes are determined based on the loan time limit related information corresponding to the target post-loan data, and then outputting the processing results of the target post-loan data of different service classes. Therefore, according to the target post-loan data and the preset business rules corresponding to the target post-loan data, the Impala memory processing mechanism in the Hadoop big data platform is used for processing the target post-loan data, PB-level data can be rapidly inquired and analyzed, the data processing efficiency is effectively improved, and powerful guarantee is provided for the development of services such as overdue collection after loan of insurance institutions, loan expiration reminding and the like.
Furthermore, the data after the target is credited is processed by utilizing a Kudu storage system and an Impala memory processing mechanism in the Hadoop big data platform, and the processing result of the data after the target is credited in different service classes is obtained.
On the basis of the same technical concept, the embodiment of the present specification further provides a data processing apparatus, and fig. 8 is a schematic diagram illustrating a module composition of the data processing apparatus provided in the embodiment of the present specification, where the data processing apparatus is configured to execute the data processing method described in fig. 1 to fig. 7, and as shown in fig. 8, the data processing apparatus includes:
a target data obtaining module 801, configured to obtain target post-loan data to be processed;
a processing result generation module 802, configured to process the target post-loan data according to the target post-loan data and a predetermined service rule corresponding to the target post-loan data by using an Impala memory processing mechanism in a Hadoop big data platform, so as to obtain processing results of the target post-loan data of different service categories, where the service categories are determined based on information on loan terms corresponding to the target post-loan data;
and the processing result output module 803 is configured to output a processing result of the target credited data of the different service types.
The embodiment of the specification provides a data processing device, according to target post-loan data and a preset business rule corresponding to the target post-loan data, an Impala memory processing mechanism in a Hadoop big data platform is used for processing the target post-loan data, PB-level data can be quickly inquired and analyzed, the data processing efficiency is effectively improved, and further powerful guarantee is provided for the development of businesses such as overdue collection after insurance agency loan, loan expiration reminding and the like.
Optionally, the service category includes: one or more of overdue hastening, due reminders, claim settlement processing, and post-loan status processing.
Optionally, the processing result output module 803 is configured to:
and outputting the processing results of the target credited data of different service classes to a preset database for storage, and/or writing the processing results of the target credited data of different service classes into a preset display page.
Optionally, the apparatus further includes a processing result pushing module, configured to:
and pushing the corresponding processing result of the target post-loan data to a predetermined processing system according to the service category corresponding to the processing result of the target post-loan data, wherein the predetermined processing system comprises one or more of a claim settlement system, an underwriting system, a repayment reminding system, a credit investigation system and a loan acceptance system.
Optionally, the processing result output module 803 is configured to:
and connecting the Java database with a JDBC write-back mechanism, and writing the processing result of the target credited data of different service classes into a preset Informix database.
Optionally, the apparatus further comprises: a target data storage module to:
and calling a preset Shell script to store the acquired target post-loan data into a Kudu-based distributed columnar storage system on a Hadoop big data platform.
Optionally, the apparatus further includes a target data generation module, configured to:
obtaining source credit data;
and processing the obtained source post-credit data to obtain the target post-credit data with different service types in a preset data format.
The data processing device in the embodiment of the present specification obtains target post-loan data to be processed, and processes the target post-loan data by using an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a predetermined service rule corresponding to the target post-loan data, so as to obtain processing results of the target post-loan data of different service classes, where the service classes are determined based on information about loan time limits corresponding to the target post-loan data, and then outputs the processing results of the target post-loan data of different service classes. Therefore, according to the target post-loan data and the preset business rules corresponding to the target post-loan data, the Impala memory processing mechanism in the Hadoop big data platform is used for processing the target post-loan data, PB-level data can be rapidly inquired and analyzed, the data processing efficiency is effectively improved, and powerful guarantee is provided for the development of services such as overdue collection after loan of insurance institutions, loan expiration reminding and the like.
Furthermore, the data after the target is credited is processed by utilizing a Kudu storage system and an Impala memory processing mechanism in the Hadoop big data platform, and the processing result of the data after the target is credited in different service classes is obtained.
The data processing apparatus provided in the embodiment of the present invention can implement each process in the embodiment corresponding to the data processing method, and is not described here again to avoid repetition.
It should be noted that the mobile terminal provided in the embodiment of the present invention and the data processing method provided in the embodiment of the present invention are based on the same inventive concept, and therefore, for specific implementation of the embodiment, reference may be made to implementation of the data processing method, and repeated details are not described again.
Based on the same technical concept, the embodiment of the present invention further provides an electronic device for executing the method for identifying malicious fees, where fig. 9 is a schematic structural diagram of an electronic device implementing the embodiments of the present invention, as shown in fig. 9. Electronic devices may vary widely in configuration or performance and may include one or more processors 901 and memory 902, where the memory 902 may store one or more stored applications or data. Memory 902 may be, among other things, transient storage or persistent storage. The application program stored in memory 902 may include one or more modules (not shown), each of which may include a series of computer-executable instructions for the electronic device. Still further, the processor 901 may be configured to communicate with the memory 902 to execute a series of computer-executable instructions in the memory 902 on the electronic device. The electronic device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input-output interfaces 905, one or more keyboards 906.
Specifically, in this embodiment, the electronic device includes a processor, a communication interface, a memory, and a communication bus; the processor, the communication interface and the memory complete mutual communication through a bus; the memory is used for storing a computer program; the processor is used for executing the program stored in the memory and realizing the following method steps:
acquiring target post-loan data to be processed;
processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset business rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different business categories, wherein the business categories are determined based on the related information of loan terms corresponding to the target post-loan data;
and outputting the processing result of the target credited data of different service classes.
An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and when executed by a processor, the computer program implements the following method steps:
acquiring target post-loan data to be processed;
processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset business rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different business categories, wherein the business categories are determined based on the related information of loan terms corresponding to the target post-loan data;
and outputting the processing result of the target credited data of different service classes.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A method of data processing, the method comprising:
acquiring target post-loan data to be processed;
processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset business rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different business categories, wherein the business categories are determined based on the related information of loan terms corresponding to the target post-loan data;
and outputting the processing result of the target credited data of different service classes.
2. The method of claim 1, wherein the traffic classes comprise: one or more of overdue hastening, due reminders, claim settlement processing, and post-loan status processing.
3. The method of claim 1, wherein outputting the processing result of the target post-credit data of the different traffic classes comprises:
and outputting the processing results of the target credited data of different service classes to a preset database for storage, and/or writing the processing results of the target credited data of different service classes into a preset display page.
4. The method of claim 1, further comprising:
and pushing the corresponding processing result of the target post-loan data to a predetermined processing system according to the service category corresponding to the processing result of the target post-loan data, wherein the predetermined processing system comprises one or more of a claim settlement system, an underwriting system, a repayment reminding system, a credit investigation system and a loan acceptance system.
5. The method according to claim 3, wherein the outputting the processing result of the target post-credit data of different service classes to a predetermined database for storage comprises:
and connecting the Java database with a JDBC write-back mechanism, and writing the processing result of the target credited data of different service classes into a preset Informix database.
6. The method of claim 1, further comprising, after said obtaining target post-loan data to be processed:
and calling a preset Shell script to store the acquired target post-loan data into a Kudu-based distributed columnar storage system on a Hadoop platform.
7. The method of claim 1, further comprising, prior to said obtaining target post-loan data to be processed:
obtaining source credit data;
and processing the obtained source post-credit data to obtain the target post-credit data with different service types in a preset data format.
8. A data processing apparatus, characterized in that the apparatus comprises:
the target data acquisition module is used for acquiring target post-loan data to be processed;
the processing result generation module is used for processing the target post-loan data by utilizing an Impala memory processing mechanism in a Hadoop big data platform according to the target post-loan data and a preset service rule corresponding to the target post-loan data to obtain processing results of the target post-loan data of different service types, wherein the service types are determined based on the related information of the loan term corresponding to the target post-loan data;
and the processing result output module is used for outputting the processing result of the target credited data of different service classes.
9. An electronic device comprising a processor, a communication interface, a memory, and a communication bus; the processor, the communication interface and the memory complete mutual communication through a bus; the memory is used for storing a computer program; the processor, for executing the program stored in the memory, implements the steps of the data processing method of any of claims 1-7.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the data processing method steps of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910813409.8A CN110659999A (en) | 2019-08-30 | 2019-08-30 | Data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910813409.8A CN110659999A (en) | 2019-08-30 | 2019-08-30 | Data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110659999A true CN110659999A (en) | 2020-01-07 |
Family
ID=69036544
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910813409.8A Pending CN110659999A (en) | 2019-08-30 | 2019-08-30 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110659999A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112182106A (en) * | 2020-09-28 | 2021-01-05 | 中国平安人寿保险股份有限公司 | Insurance business data processing method, system, device and storage medium |
| CN112465656A (en) * | 2020-12-08 | 2021-03-09 | 中国人寿保险股份有限公司 | Insurance detail data sending method and device |
| CN112905323A (en) * | 2021-02-09 | 2021-06-04 | 泰康保险集团股份有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107578331A (en) * | 2017-09-19 | 2018-01-12 | 马上消费金融股份有限公司 | The method and system of risk monitoring and control after a kind of loan |
| CN108334557A (en) * | 2017-12-29 | 2018-07-27 | 东软集团（上海）有限公司 | A kind of aggregated data analysis method, device, storage medium and electronic equipment |
| CN110109905A (en) * | 2019-04-26 | 2019-08-09 | 深圳前海微众银行股份有限公司 | Risk list data generation method, device, equipment and computer storage medium |
-
2019
- 2019-08-30 CN CN201910813409.8A patent/CN110659999A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107578331A (en) * | 2017-09-19 | 2018-01-12 | 马上消费金融股份有限公司 | The method and system of risk monitoring and control after a kind of loan |
| CN108334557A (en) * | 2017-12-29 | 2018-07-27 | 东软集团（上海）有限公司 | A kind of aggregated data analysis method, device, storage medium and electronic equipment |
| CN110109905A (en) * | 2019-04-26 | 2019-08-09 | 深圳前海微众银行股份有限公司 | Risk list data generation method, device, equipment and computer storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 大话金融IT: "百亿级金融业务大数据架构与实践", 《HTTPS://ZHUANLAN.ZHIHU.COM/P/55421626》 * |
| 天云数据: "光大银行：风险一体化项目实施", 《WWW.BEAGLEDATA.COM/?PORTFOLIO=光大银行：风险一体化项目实施》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112182106A (en) * | 2020-09-28 | 2021-01-05 | 中国平安人寿保险股份有限公司 | Insurance business data processing method, system, device and storage medium |
| CN112182106B (en) * | 2020-09-28 | 2023-11-28 | 中国平安人寿保险股份有限公司 | Insurance business data processing method, system, device and storage medium |
| CN112465656A (en) * | 2020-12-08 | 2021-03-09 | 中国人寿保险股份有限公司 | Insurance detail data sending method and device |
| CN112905323A (en) * | 2021-02-09 | 2021-06-04 | 泰康保险集团股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112905323B (en) * | 2021-02-09 | 2023-10-27 | 泰康保险集团股份有限公司 | Data processing method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11144670B2 (en) | Data processing systems for identifying and modifying processes that are subject to data subject access requests | |
| US11409908B2 (en) | Data processing systems and methods for populating and maintaining a centralized database of personal data | |
| US10572686B2 (en) | Consent receipt management systems and related methods | |
| US10346638B2 (en) | Data processing systems for identifying and modifying processes that are subject to data subject access requests | |
| US10678945B2 (en) | Consent receipt management systems and related methods | |
| US11200341B2 (en) | Consent receipt management systems and related methods | |
| US10685140B2 (en) | Consent receipt management systems and related methods | |
| US10776518B2 (en) | Consent receipt management systems and related methods | |
| US10440062B2 (en) | Consent receipt management systems and related methods | |
| US10592648B2 (en) | Consent receipt management systems and related methods | |
| US10503926B2 (en) | Consent receipt management systems and related methods | |
| US20190179490A1 (en) | Consent receipt management systems and related methods | |
| CN107767070B (en) | Method and device for information popularization | |
| CN110659999A (en) | Data processing method and device and electronic equipment | |
| WO2019168599A1 (en) | Data retention handling for data object stores | |
| US10783112B2 (en) | High performance compliance mechanism for structured and unstructured objects in an enterprise | |
| US10776514B2 (en) | Data processing systems for the identification and deletion of personal data in computer systems | |
| CN110032594B (en) | Customizable data extraction method and device for multi-source database and storage medium | |
| JP2023531186A (en) | Systems and methods for implementing market data contract analysis tools | |
| US20240127379A1 (en) | Generating actionable information from documents | |
| US11625502B2 (en) | Data processing systems for identifying and modifying processes that are subject to data subject access requests | |
| US10121138B2 (en) | Correctable pre-payment for database services | |
| CN106874327B (en) | Counting method and device for business data | |
| CN114265842A (en) | Audit data processing method, device, equipment and storage medium based on ERP system | |
| US20230267557A1 (en) | Generic configuration platform for generating electronic reports |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200107 |