CN107220892B

CN107220892B - Intelligent preprocessing tool and method applied to massive P2P network loan financial data

Info

Publication number: CN107220892B
Application number: CN201710392181.0A
Authority: CN
Inventors: 马秀娟; 毛洪亮; 王秀文; 苏沐冉; 张露晨; 吴震; 李焱余; 唐积强; 徐小磊; 李传海; 苏志坚; 谢铭
Original assignee: Beijing Scistor Technologies Co ltd; National Computer Network and Information Security Management Center
Current assignee: Beijing Scistor Technologies Co ltd; National Computer Network and Information Security Management Center
Priority date: 2017-05-27
Filing date: 2017-05-27
Publication date: 2020-10-16
Anticipated expiration: 2037-05-27
Also published as: CN107220892A

Abstract

The invention discloses an intelligent preprocessing tool and method applied to massive P2P network credit financial data, which can accurately and efficiently preprocess massive P2P network credit financial data before storing the data in a standard financial service database, and ensure the accuracy and the effectiveness of the data, thereby establishing an effective monitoring mechanism of a P2P network credit platform and effectively strengthening the supervision of the P2P network credit platform. The preprocessing comprises establishing validity and integrity check rules to realize data check before warehousing; data classification, data deduplication, data correction, data conversion and state calculation are carried out, and data processing before warehousing is achieved; and finally loading the data into a standard financial service database.

Description

Intelligent preprocessing tool and method applied to massive P2P network loan financial data

Technical Field

The invention relates to an intelligent preprocessing method applied to massive P2P network loan financial data, and belongs to the field of preprocessing methods of massive data.

Background

In recent years, with the rapid development of domestic internet finance, P2P network loan type internet financial platforms are in the endlessly. According to statistics, the established P2P network loan platform reaches more than 5000 families by 2016 and 7 months nationwide, and the historical transaction amount of the P2P network loan industry exceeds 2 trillion.

However, while the development is fast, P2P network lending continuously causes high bad account rate, frequent running and other risk events, and it is estimated that approximately half of the platforms have related problems. Therefore, it is urgently needed to grasp the specific investment and loan conditions of each platform by means of a unified monitoring platform, form bidirectional monitoring of "information flow" and "fund flow", and perform big data analysis on platform transaction data, so as to establish an effective monitoring mechanism of a P2P network loan platform, and effectively strengthen the monitoring of the P2P network loan platform.

Effective monitoring and accurate analysis of real-time transaction data of a large-scale platform are uniformly realized on a single monitoring platform, enterprise data completeness verification is needed, and automatic identification of wrong, lack and other problem data is realized. Therefore, it is important to accurately and efficiently perform the preprocessing.

Disclosure of Invention

The invention aims to solve the problems and provides a tool and a method for intelligently preprocessing massive P2P network credit financial data, and the invention relates to an intelligent preprocessing method for massive P2P network credit financial data, which is a method for realizing the whole process from the intelligent preprocessing process to the storage of a service library for the accessed massive financial data.

The invention discloses a tool and a method for intelligently preprocessing massive P2P network loan financial data, wherein the intelligent preprocessing comprises data classification of various manufacturers, data deduplication, data correction, format processing, data conversion, state calculation and the like. Thereby assisting the user to establish a standard database of standard internet financial transaction data. The method mainly comprises the following steps:

1. establishing various data integrity and validity check rules to verify the data integrity and validity;

2. and (4) pre-treating before warehousing.

The integrity check rule mainly comprises: null check, format check, etc.:

1. checking a null value: checking whether the field is empty;

2. and (3) format checking: the format of the field is checked for compliance with the specification.

The validity check rule comprises value threshold check, date format check, self-definition and the like:

1. threshold check: checking whether the field value exceeds the range;

2. and (3) date format verification: checking whether the format of the date field satisfies a prescribed format;

3. self-defining: defining a new validity check rule and developing an operator.

The warehousing pretreatment comprises: data classification, data deduplication, data correction, data conversion, state calculation, data loading and the like:

1. data classification: classifying and screening the data according to enterprises, and inputting the data into corresponding channels to perform a data preprocessing flow;

2. data deduplication: filtering out repeated data;

3. and (3) data correction: modifying missing, format-wrong, abnormal and other data messages by means of removing, cutting and the like;

4. data conversion: the data types or transaction types of different platforms are not consistent with the central standard, and are converted into standard type data according to a comparison table provided by the platform, such as: the ant golden uniform 1 represents investment data; and the rule 2 in the standard library indicates investment data, in which case data conversion is required for adjustment.

5. And (3) state calculation: the product data (scattered bidding | financing) of the P2P network loan enterprise, the product investment and loan state change (full bidding, running bidding, overdue and repayment) need to be calculated through transaction flow data. Calculating the formula:

(1) full scale: the investment amount is greater than the bid amount;

(2) and (4) flow marking: no deposit record;

(3) overdue: open time +8 days + deadline < current time.

6. Loading data: and loading the data passing the pretreatment and entering a service library.

In order to ensure the high efficiency and expandability of intelligent pretreatment, the invention adopts the following technical means:

1. the data preprocessing adopts a distributed real-time processing framework, and by utilizing the batch consumption characteristic of a distributed message queue, the real-time checking and processing of data integrity and validity are realized by formulating and developing data integrity and validity check rules;

2. loose coupling among the modules is realized through the message queue, and the expandability and the robustness of the system are improved;

3. operator specification and operator extension. Operators are packages developed according to certain specifications to realize certain aspect preprocessing of data. Such as null check operator, date check operator, value range check operator, character string check operator, scrambling code check operator, etc.

In the invention, the integrity and validity check rules can be regarded as operators in the streaming task, and the integrity and validity check can be carried out on fields such as whether the transaction time meets the 'yyyy-MM-dd HH: MM: ss' format, whether the value of a hash state is in the range of 1, 2 and 3, or user information messy codes and the like. Meanwhile, the validity and integrity check of the data can be expanded by developing other check rule operators, and only certain development and packaging specifications need to be followed. And uploading the newly developed operator to a system through an uploading interface, and then using the newly developed operator. In the aspect of operators, the system has the following functions:

(1) presetting common operators, such as null value checking operators, date checking operators and the like;

(2) issuing operator development specifications and packaging specifications;

(3) and the uploading operator adds and expands data preprocessing capacity for the system.

The invention has the advantages that:

1. the tool has the distributed processing characteristic, can perform parallel computation and efficiently process; meanwhile, the system has the system characteristics of fault tolerance, expandability, durability and the like;

2. the tool has strong pertinence, can efficiently carry out intelligent preprocessing on massive P2P network credit financial data, and has wide application prospect.

Drawings

FIG. 1 is a flow of Web loan financial data preprocessing;

FIG. 2 is a real-time processing framework structure of SAMZA + KAFKA;

FIG. 3 is a YARN-based SAMZA resource scheduling flow;

FIG. 4 is a streaming application framework design diagram;

fig. 5 is an application example of the present invention.

Detailed Description

The present invention will be described in further detail below with reference to the accompanying drawings.

The intelligent preprocessing flow of the mass P2P network loan financial data is shown in figure 1. The data is firstly accessed to the message queue system, and during preprocessing, the data is consumed from the message queue in the role of a consumer, and integrity and validity verification is firstly carried out. If the verification fails, transferring to an abnormal data processing sub-process, and returning an error code to the network credit platform; and if the verification is passed, switching to a processing sub-process, carrying out operations such as data classification, data deduplication, data correction, data conversion, state calculation and the like, and finally loading the network credit financial data into an internet financial service library.

The data preprocessing adopts a distributed real-time processing framework. The real-time processing framework using SAMZA + KAFKA was selected by analytical studies analyzing SAMZA + KAFKA, STORM and SPARKSTREAMING. KAFKA, as a distributed message queuing system, has implemented many core infrastructures at the bottom of a streaming framework; while SAMZA, as a distributed streaming data processing framework, naturally integrates KAFKA distributed message queues, the default implementation of which is based on KAFKA.

As shown in fig. 2, the data integrity and validity check rules in the present invention are all used as a JOB in the SAMZA + KAFKA real-time processing framework. SAMZA's JOB, a basic process flow, is that a user task reads data from one or more input streams, after certain processing, outputs to one or more output streams, specifically mapping to KAFKA reads data from one or more TOPICs/PARTITIONs and writes it out to another one or more TOPICs/PARTITIONs; the streaming data processing flow is completed by connecting a plurality of JOBs in series.

This real-time processing mode of SAMZA + KAFKA is actually somewhat like a MapReduce process, where the STREAM input part determines the number of PARTITIONs and tasks (tasks) from TOPIC/PARTITION of KAFKA, similar to a Map process, and the user Task specifies TOPIC and PARTITIONs at the time of output (or the frame automatically determines PARTITIONs from Key), which is equivalent to a Shuffle process, and the next JOB reads a new STREAM, which can be considered as a Reduce or the start of the next Map process. The difference is that the concatenation between the JOBs does not have to wait for the end of the last JOB, and the real-time message distribution mechanism determines that the entire concatenated JOB is uninterrupted, i.e. streamed.

SAMZA uses YARN for resource allocation scheduling (the scheduling module is optional, YARN is used by default) as shown in fig. 3. SAMZA AM is responsible for JOB scheduling, and Task runner is responsible for the operation of user TASK, and by means of the help of KAFKA and YARN, SAMZA can realize the characteristics of distribution/fault tolerance/scalability/persistence and the like.

The tool of the invention enables a user to construct a preprocessing task through a graphical interface. One streaming task is formed by adding one or more preprocessing operators and configuring the operation parameters of each operator. FIG. 4 is a layout diagram of a streaming application framework, the operation flow of which is described as follows:

(1) operator 1 calls: the process data is sent as a producer to topic 1;

(2) operator 2 calls: send operator 3 as data after the consumer gets data processing from topic 1. The other operators and so on.

(3) And (3) abnormal data discovery: and sending the abnormal data to an abnormal topic, processing by an abnormal processing operator, and returning an error code to the network credit platform.

Basic flow of network loan platform data preprocessing:

(1) classifying the basic data according to platforms, and formulating a conversion rule according to the characteristics of each platform;

(2) data deduplication processing is carried out, and the phenomenon that an enterprise repeatedly reports data is avoided;

(3) calculating the data state of the network loan label scattering data, and calculating the actual state of the label scattering according to the dimensions of transaction flow, label base attributes and time;

(4) calculating and extracting final states of the scattered labels;

(5) classifying the network loan label-dispersing industry, setting rules according to the amount range and the field content, and marking an industry use label on the label;

(6) normalizing the transaction types, performing normalized calculation on the transaction states of different enterprises according to a set of established transaction state specifications (see table 1 in detail) with unified standards, and synchronizing the transaction states to a service library;

(7) filtering invalid data, namely deleting and filtering historical test data and data which is overdue and has no complete transaction process;

(8) dictionary data extraction, data reporting batch codes, country area codes, telephone numbers and the like.

1. The tool of the invention establishes validity and integrity check rules aiming at massive P2P network credit financial data, and realizes data check before warehousing; and data classification, data deduplication, data correction, data conversion and state calculation are carried out, and data processing before warehousing is realized.

2. The tool of the invention realizes real-time check and processing of data integrity and validity by formulating and developing data integrity and validity check rules through a distributed real-time processing framework and simultaneously utilizing the batch consumption characteristics of a distributed message queue, thereby ensuring the high efficiency, robustness and expandability of preprocessing.

3. The tool, operator definition and implementation of the invention are a big element for ensuring flexibility and expansibility. Formulating and releasing operator development and packaging specifications, presetting a default operator, and developing and expanding according to the operator specifications; in the invention, the integrity and validity check rules can be regarded as operators in the streaming task, and the distributed task is operated.

4. The tool of the present invention provides a visual modeling tool. The user may construct the pre-processing task through a graphical interface. By adding one or more preprocessing operators and configuring the operating parameters of each operator, each operator completes different preprocessing functions to form a streaming task. In the invention, operators can be dragged to the working area from the operator list in a dragging mode, the operator parameter configuration area is opened at the same time, and the operators are stored as a preprocessing task after being selected and configured, so that better user experience is provided.

5. According to the tool, basic data can be classified according to network credit platforms, and corresponding conversion rules are formulated according to the characteristics of different network credit platforms, so that data from different network credit platforms can be correctly converted into standard data. Meanwhile, repeated data can be screened, the phenomenon that the network credit platform data are repeatedly reported is prevented, and therefore accuracy of later-period data statistics and analysis results is guaranteed.

6. The tool of the invention can complete the calculation of the state of the loan and scatter data: calculating the actual state of the scattered label according to the dimensions of transaction flow, label bottom attributes and time; can complete the classification of the network loan label-dispersing industry: setting rules according to the amount range and the field content, and marking industrial purpose labels on the scattered labels;

application example:

as shown in fig. 5, each P2P web loan platform, such as ant dress, clap loan, yixin, etc., enters the KAFKA message queue through the data access layer. The data is preprocessed through massive P2P network credit financial data provided by the invention, and enters a standard financial service database through data loading.

Table 1-transaction state specification table:

Claims

1. an intelligent preprocessing method applied to massive P2P network credit financial data is characterized in that data are accessed into a message queue system to be preprocessed, data are consumed from a message queue in the role of a consumer, integrity and validity verification is carried out, if the verification fails, an abnormal data processing sub-process is carried out, an error code is returned to a network credit platform, if the verification passes, the abnormal data processing sub-process is carried out, preprocessing is carried out, data classification, data deduplication, data correction, data conversion, state calculation and data loading are carried out, and finally the network credit financial data are loaded into an internet financial service library;

the integrity verification function comprises null value check and format check, wherein the null value check refers to checking whether a field is null, and the format check refers to checking whether the format of the field is in accordance with the specification;

the validity verification function comprises threshold value check, date format check and user-defined validity check, wherein the threshold value check is used for checking whether the value of a field exceeds a range, the date format check is used for checking whether the format of the date field meets a specified format, and the user-defined validity check is a user-defined validity check rule;

the pre-treatment before warehousing specifically comprises the following key steps:

(1) data classification: classifying and screening the data according to enterprises, and inputting the data into corresponding channels to perform a data preprocessing flow;

(2) data deduplication: filtering out repeated data;

(3) and (3) data correction: correcting missing, format-incorrect and abnormal data messages;

(4) data conversion: the data types or transaction types of different platforms are not consistent with the central standard, and are converted into standard type data according to a comparison table provided by the platform;

(5) and (3) state calculation: product data of a P2P network loan enterprise, namely, loose bid and financing, and product investment and loan state changes, namely, full bid, tape flow, overdue and repayment, need to be calculated through transaction flow data, and a calculation formula is as follows:

<1> full mark: the investment amount is greater than the bid amount;

<2> stream mark: no deposit record;

<3> overdue: open time +8 days + deadline < current time;

(6) loading data: loading the data passing the pretreatment and entering a service library;

data pre-processing employs a real-time processing framework of SAMZA + KAFKA.

2. An intelligent preprocessing tool applied to massive P2P network credit fused data constructs a task through a graphical interface, a plurality of operators are added, operation parameters of each operator are configured to form a streaming task, data preprocessing is realized through the operators, integrity and validity verification are carried out on the data, if the verification fails, abnormal data processing is carried out, an error code is returned to a network credit platform, if the verification passes, data classification, data deduplication, data correction, data conversion, state calculation and data loading are realized through an algorithm, and finally the network credit fused data are loaded into an Internet financial service library;

the operators comprise an empty check operator, a date check operator, a value range check operator, a character string check operator and a messy code check operator; meanwhile, a user can develop and apply a new operator according to the operator specification;

the integrity verification function comprises null value check and format check, wherein the null value check refers to checking whether a field is null or not; the format check refers to checking whether the format of the field is in a specification;

the validity verification function comprises threshold value check, date format check and user-defined validity check, wherein the threshold value check is to check whether the value of a field exceeds a range; the date format check refers to checking whether the format of the date field meets the specified format; the user-defined validity check refers to a user-defined validity check rule;

the data classification, data deduplication, data correction, data conversion, state calculation and data loading refer to the following steps:

(1) and data classification: classifying and screening the data according to enterprises, and inputting the data into corresponding channels to perform a data preprocessing flow;

(2) data deduplication: filtering out repeated data;

(3) and data correction: correcting missing, format-incorrect and abnormal data messages;

(4) and data conversion: the data types or transaction types of different platforms are not consistent with the central standard, and are converted into standard type data according to a comparison table provided by the platform;

(5) and state calculation: product data of a P2P network loan enterprise, namely, loose bid and financing, and product investment and loan state changes, namely, full bid, tape flow, overdue and repayment, need to be calculated through transaction flow data, and a calculation formula is as follows:

<1> full mark: the investment amount is greater than the bid amount;

<2> stream mark: no deposit record;

<3> overdue: open time +8 days + deadline < current time;

(6) and data loading: loading the data passing the pretreatment and entering a service library;

data pre-processing employs a real-time processing framework of SAMZA + KAFKA.