CN110597816A

CN110597816A - Data processing method, data processing device, computer equipment and computer readable storage medium

Info

Publication number: CN110597816A
Application number: CN201910876367.2A
Authority: CN
Inventors: 林仕锋; 侯皓龄
Original assignee: Shenzhen Chase Technology Co Ltd
Current assignee: Shenzhen Chase Technology Co Ltd; Shenzhen Zhuiyi Technology Co Ltd
Priority date: 2019-09-17
Filing date: 2019-09-17
Publication date: 2019-12-20

Abstract

The application relates to a data processing method, a data processing device, computer equipment and a computer readable storage medium. The data processing method comprises the following steps: classifying the service data of a plurality of client samples according to preset classification categories to obtain classification data; extracting target data matched with the business fields from the classified data according to a plurality of preset business fields; the classification category and the service field are determined according to the training requirement of a target model; generating a standard data table corresponding to each classification type according to the service field and the target data; the standard data table is used for training the target model. By adopting the method, the data processing amount can be reduced and the data processing efficiency can be improved in the processes of feature extraction and model training according to financial data.

Description

Data processing method, data processing device, computer equipment and computer readable storage medium

Technical Field

The present invention relates to the field of financial technology, and in particular, to a data processing method, apparatus, computer device, and computer-readable storage medium.

Background

With the rapid development of computer technology in various industries, more and more traditional industries are gradually developing to science and technology, and the traditional financial industry is also gradually changing to financial science and technology.

In actual business operation, different enterprises have different data standard formats to support normal operation of business lines, and the data formats of the enterprises are often different greatly.

In the financial technology, there are a lot of financial data processing scenarios, however, if feature engineering, modeling, and the like are directly performed according to the raw data of the samples provided by each enterprise, the problems of large data processing amount and low data processing efficiency are caused.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data processing method, an apparatus, a computer device and a computer readable storage medium, which can reduce the data processing amount and improve the data processing efficiency in a financial data processing scenario.

In a first aspect, an embodiment of the present application provides a data processing method, where the data processing method includes:

classifying the service data of a plurality of client samples according to preset classification categories to obtain classification data;

extracting target data matched with the business fields from the classified data according to a plurality of preset business fields; the classification category and the service field are determined according to the training requirement of a target model;

generating a standard data table corresponding to each classification type according to the service field and the target data; the standard data table is used for training the target model.

In one embodiment, the generating a standard data table corresponding to each classification category according to the service field and the target data includes:

detecting whether the data type of the target data is an enumeration type;

if the data type of the target data is an enumeration type, finding out a digital code corresponding to the target data in a preset digital dictionary; the digital dictionary comprises a mapping relation between the target data and the digital codes;

and importing the digital code into the service field corresponding to the target data to generate the standard data table.

In one embodiment, the method further comprises:

and if the data type of the target data is not an enumeration type, importing the target data into the service field corresponding to the target data to generate the standard data table.

In one embodiment, the extracting, from the classification data according to a preset plurality of service fields, target data matched with the service fields includes:

generating a plurality of SQL tasks by adopting SQL sentences according to the plurality of business fields;

and extracting target data matched with the business field from the classification data based on the plurality of SQL tasks.

In one embodiment, the extracting target data matching the business field from the classification data based on the plurality of SQL tasks includes:

detecting whether a dependency relationship exists among the plurality of SQL tasks;

and if the dependency relationship does not exist among the plurality of SQL tasks, executing the plurality of SQL tasks in parallel to extract the target data matched with the business field from the classified data.

In one embodiment, the classification categories include a transaction category, a non-transaction category, and a transaction assistance category;

the standard data table corresponding to the transaction class comprises an internal transaction flow table and/or an external transaction flow table; the standard data table corresponding to the non-transaction type comprises at least one of an account information table, a private customer information table and a public customer information table; the standard data table corresponding to the transaction auxiliary class comprises at least one of a customer login information table, an account balance table, a reported suspicious customer information table and an area code table.

In one embodiment, before classifying the service data of the plurality of customer samples according to the preset classification category to obtain the classification data, the method further includes:

determining a database corresponding to each service data according to a service scene;

business data for the plurality of customer samples is obtained from a respective database.

In a second aspect, an embodiment of the present application provides a data processing apparatus, where the apparatus includes:

the classification module is used for classifying the original data of the plurality of client samples according to preset classification categories to obtain classified data;

the extraction module is used for extracting target data matched with the business fields from the classified data according to a plurality of preset business fields; the classification category and the service field are determined according to the training requirement of a target model;

a generating module, configured to generate a standard data table corresponding to each classification type according to the service field and the target data; the standard data table is used for training the target model.

In one embodiment, the generating module 30 includes:

the detection unit is used for detecting whether the data type of the target data is an enumeration type;

the searching unit is used for searching a digital code corresponding to the target data in a preset digital dictionary if the data type of the target data is an enumeration type; the digital dictionary comprises a mapping relation between the target data and the digital codes;

and the replacing unit is used for leading the digital code into the service field corresponding to the target data so as to generate the standard data table.

In one embodiment, the generating module 30 is further configured to, if the data type of the target data is not an enumeration type, import the target data into the service field corresponding to the target data to generate the standard data table.

In one embodiment, the extraction module 20 includes:

the generating unit is used for generating a plurality of SQL tasks by adopting SQL sentences according to the plurality of business fields;

and the extracting unit is used for extracting target data matched with the business field from the classification data based on the plurality of SQL tasks.

In one embodiment, the extraction unit includes:

the detecting subunit is used for detecting whether a dependency relationship exists among the plurality of SQL tasks;

and the execution subunit is used for executing the plurality of SQL tasks in parallel if the dependency relationship does not exist among the plurality of SQL tasks so as to extract the target data matched with the business field from the classified data.

In one embodiment, the classification classes include a transaction class, a non-transaction class, and a transaction assistance class;

In one embodiment, the apparatus further comprises:

the determining module is used for determining a database corresponding to each service data according to a service scene;

and the acquisition module is used for acquiring the business data of the plurality of customer samples from the corresponding databases.

In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the method according to the first aspect when executing the computer program.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the method according to the first aspect.

The beneficial effects brought by the technical scheme provided by the embodiment of the application at least comprise:

classifying the service data of a plurality of client samples according to preset classification categories to obtain classification data; extracting target data matched with the business fields from the classified data according to a plurality of preset business fields; the classification category and the service field are determined according to the training requirement of a target model; generating a standard data table corresponding to each classification type according to the service field and the target data; the standard data table is used for training the target model; therefore, classification categories and business fields are determined according to the training requirements of the target model, a standard data table is respectively constructed for the business data under each classification category according to the business fields, and the conversion from the business data with different enterprise formats to the standard data table is completed, so that the feature extraction and the model training can be further performed according to the standard data table; the problems of large data processing capacity and low data processing efficiency caused by large data format difference among enterprises in the traditional technology when the characteristic engineering and modeling are directly carried out according to the sample original data provided by each enterprise are solved; the characteristic extraction and model training process is only carried out based on the standard data sheet, the data processing amount in the characteristic extraction and model training process is reduced, and the data processing efficiency is improved.

Drawings

FIG. 1 is a diagram of an application environment of a data processing method according to an embodiment;

FIG. 2 is a flow diagram illustrating a data processing method, according to an embodiment;

FIG. 3 is a flow diagram illustrating a data processing method, according to an embodiment;

FIG. 4 is a flowchart of a data processing method according to an embodiment;

FIG. 5 is a schematic diagram of a detailed step of step S220 in FIG. 4;

fig. 6 is a block diagram of a data processing apparatus according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The data processing method provided by the application can be applied to the computer device shown in fig. 1, the computer device can be a server, and the internal structure diagram can be shown in fig. 1. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data of the data processing method.

The data processing method, the data processing device, the computer equipment and the computer readable storage medium provided by the embodiment of the application aim to solve the technical problems of large data processing capacity and low data processing efficiency caused by performing feature engineering, modeling and the like according to original data of samples provided by various enterprises in the traditional technology. The following describes in detail the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems by embodiments and with reference to the drawings. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.

It should be noted that, in the data processing method provided in the embodiments of the present application, the execution main body may be a data processing apparatus, and the data processing apparatus may be implemented as part of or all of a computer device by software, hardware, or a combination of software and hardware. In the following method embodiments, the execution subject is a computer device as an example.

Referring to fig. 2, which shows a flowchart of a data processing method provided in an embodiment of the present application, as shown in fig. 2, the data processing method of the present embodiment may include the following steps:

and S100, classifying the service data of the plurality of client samples according to preset classification categories to obtain classified data.

With the continuous improvement of the informatization and intellectualization degrees of each enterprise, data becomes an important asset of an enterprise unit; under the background of a big data era, data items such as feature extraction, combined modeling and the like are often needed to be carried out by combining data of each enterprise unit so as to better provide service for users; in actual business, different enterprises generally have respective data standard formats, and the standard formats between the enterprises are often different greatly. Taking an anti-money laundering scene in financial science and technology as an example, whether a customer is suspected to be money laundering needs to be judged according to the transaction behavior of the customer, so that the customer transaction flow is greatly dependent. However, at present, the recording modes of the transaction behaviors of the customers by various banks are different, for example, the records are respectively recorded according to different channels, international and international, public and private customers and even payment modes; even if the data tables of each enterprise are in the same field, different recording modes are provided, such as transaction directions, some banks are used for distinguishing the data tables from one another by using the data tables, and some banks are used for distinguishing the data tables from one another by using the data tables; if the original data of each bank is used for feature engineering and modeling, a large amount of time is consumed in data unification, research and development resources are seriously wasted, the efficiency of the project is low, and the method is not beneficial to large-scale popularization of data projects.

In this embodiment, the computer device classifies the service data of the plurality of client samples according to preset classification categories to obtain classification data; specifically, taking a business scenario as an anti-money laundering scenario as an example, after the computer device obtains business data of a plurality of customer samples from each bank, the business data is customer original operation data recorded by each bank in a respective data standard format, the computer device classifies the obtained business data, and the classification category is determined according to training requirements of a target model; as an implementation mode, according to the characteristics and the business characteristics of a modeling project which are extremely dependent on the customer transaction flow in an anti-money laundering scene, classification categories are set into a transaction category, a non-transaction category and a transaction auxiliary information category, and the computer equipment classifies the business data of a plurality of customer samples according to the three classification categories to obtain classification data; it will be appreciated that in other traffic scenarios, classification categories may be implemented in other ways.

In other embodiments, before step S100, the computer device further determines a database corresponding to each service data according to the service scenario, and obtains the service data of the plurality of client samples from the corresponding databases; namely, the data acquisition source is different for different service scenes, and a specific database is determined according to the service scenes to acquire service data.

And S200, extracting target data matched with the business fields from the classified data according to a plurality of preset business fields.

Wherein, the business field is determined according to the training requirement of the target model; specifically, for each type of classification data, determining a business field according to the training requirement of a target model, for example, a target model for identifying whether a customer is suspected of money laundering needs to be trained, and setting business fields related to transaction flow for the classification data of the transaction type, including business fields such as transaction flow number, transaction date, transaction time, transaction location, transaction code, customer number, account number, transaction counter account number, transaction amount, transaction channel, transaction direction, transaction currency, exchange rate settlement date and the like; for classification data of non-transaction types, setting account information and business fields related to customer information, wherein the business fields comprise account numbers, account number types, customer numbers, account opening dates, account opening mechanism numbers, account states, account passwords and the like, setting fields of customer numbers, registration information, industry categories, legal information and the like for enterprise customers, and setting fields of customer numbers, certificate numbers, ages, occupation, income and the like for personal customers; and setting business fields related to the transaction auxiliary information for the classification data of the transaction auxiliary information, wherein the business fields comprise fields such as an electronic bank customer number, a client IP (Internet protocol), a login channel and the like.

And the computer equipment extracts target data matched with the business field from the classification data corresponding to each classification category according to the plurality of business fields corresponding to each classification category.

Step S300, generating a standard data table corresponding to each classification type according to the service field and the target data.

The generated standard data table is used for training a target model; specifically, the extracted target data is imported into corresponding service fields in a table, and a standard data table corresponding to each classification category is generated; for example, for classification data of which the classification category is a transaction category, target data extracted from the classification data are respectively imported into business fields such as a transaction flow number, a transaction date, a transaction time, a transaction place, a transaction code, a client number, an account number, a transaction counter account number, a transaction amount, a transaction channel, a transaction direction and the like in a table to obtain an domestic transaction flow meter, wherein the domestic transaction flow meter is a standard data table under the transaction classification category; furthermore, after adding target data of business fields such as transaction currency, exchange rate settlement date and the like on the basis of the domestic transaction flow meter, the obtained overseas transaction flow meter is used as another standard data table under the classification category of transaction types, and the like.

The embodiment classifies the service data of a plurality of client samples according to preset classification categories to obtain classification data; extracting target data matched with the business fields from the classified data according to a plurality of preset business fields; the classification category and the service field are determined according to the training requirement of the target model; generating a standard data table corresponding to each classification type according to the service field and the target data; the standard data table is used for training a target model; therefore, classification categories and business fields are determined according to the training requirements of the target model, a standard data table is respectively constructed for the business data under each classification category according to the business fields, and the conversion from the business data with different enterprise formats to the standard data table is completed, so that the feature extraction and the model training can be further performed according to the standard data table; the problems of large data processing capacity and low data processing efficiency caused by large data format difference among enterprises in the traditional technology when the characteristic engineering and modeling are directly carried out according to the sample original data provided by each enterprise are solved; the feature extraction and model training process of the embodiment is only carried out based on the standard data table, so that the data processing amount in the feature extraction and model training process is reduced, and the data processing efficiency is improved.

Fig. 3 is a schematic flowchart of a data processing method according to another embodiment. On the basis of the embodiment shown in fig. 2, in the data processing method of this embodiment, step S300 specifically includes:

step S310, detecting whether the data type of the target data is an enumeration type.

Wherein, enumeration refers to listing all members of a finite sequence set of a set; for example, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday, which represent weeks, is an enumeration. In this embodiment, whether the data type of the target data is an enumeration type is detected, for example, whether the target data is data representing an enumeration type such as a certificate type, a job, a payment channel, and the like.

Step S320, if the data type of the target data is an enumeration type, finding out a digital code corresponding to the target data in a preset digital dictionary; the digital dictionary includes a mapping of target data to digital codes.

Specifically, the target data may be a character string representing a transaction serial number, a character string representing a customer name, and may also be a character string representing a customer certificate type, a scholarly, a profession, an industry, and the like; for part of target data, the same data characteristics exist, for example, the data characteristics are that only a few to dozens of categories exist generally, the content is in a text format, and the data characteristics may be expanded along with business requirements; for example, the business field is a job, the corresponding target data includes students, teachers, doctors, officers, and the like, and when some emerging industries occur, new jobs, such as take-out riders, and the like, may occur. For target data which is only of a few to dozens of categories, has a text format and may be expanded along with business requirements, the embodiment introduces a digital dictionary, the digital dictionary comprises a mapping relation between the target data and digital codes, the enumerated target data is replaced by corresponding digital codes, i.e. a character string of original target data is replaced by a few digits, for example, one hundred different target data can be represented by two digits (00-99), the representation is simple, the database does not need to completely store all the target data, and the storage space of the database table is greatly saved.

Step S330, the digital code is imported under the service field corresponding to the target data to generate a standard data table.

And searching a digital code corresponding to the target data of the enumeration type in the digital dictionary, and importing the searched digital code into a service field corresponding to the target data to generate a standard data table.

For example, a digital dictionary is provided with a target data "identity card" corresponding digital code of "01", "a house entrance book" corresponding digital code of "02", "a military officer license" corresponding digital code of "03", "a homeward license" corresponding digital code of "04", "a passport" corresponding digital code of "05", "a harbor and australian passerby passage" corresponding digital code of "06", "a taiwan resident passage to continental passerby passage" corresponding digital code of "07", "a temporary identity card" corresponding digital code of "08", "a warfare" corresponding digital code of "09", and so on; taking target data as an identity card as an example, the present embodiment imports the digital code "01" corresponding to the target data "identity card" into the service field of "certificate type" corresponding to the target data "identity card"; similarly, the digital code "02" corresponding to the target data "family notebook" is imported into the certificate type service field corresponding to the target data "family notebook", so that the generation of the standard data table by replacing the target data with the digital code is realized.

In other embodiments, as an implementation manner, if the data type of the target data is not an enumeration type, the target data is imported into a service field corresponding to the target data to generate a standard data table; it can be understood that the types of the target data, such as the client name, the client address, and the like, do not have the enumeration characteristic, and for the target data which is not the enumeration type, the target data is directly imported into the service field corresponding to the target data to generate the standard data table, and the subsequent characteristics and modeling engineering are performed based on the generated standard data table, so that the data processing amount is reduced, and the data processing efficiency is improved.

In this embodiment, by detecting whether the data type of the target data is an enumeration type, if the data type of the target data is the enumeration type, finding out a digital code corresponding to the target data in a preset digital dictionary; the digital dictionary comprises a mapping relation between target data and digital codes; importing the digital code into a service field corresponding to the target data to generate a standard data table; therefore, the form of a digital dictionary is adopted, and the original service logic words are replaced by a plurality of digits, so that the storage space of a standard data table in a database is greatly saved, and the subsequent feature engineering coding is facilitated; when the target data in the enumeration type is expanded, the mapping relation between the newly added target data and the digital codes is only needed to be correspondingly set in the digital dictionary, and then the standard data table is correspondingly increased, so that the service expansion is facilitated.

Fig. 4 is a schematic flowchart of a data processing method according to another embodiment. On the basis of the embodiment shown in fig. 2, step S200 includes:

and step S210, generating a plurality of SQL tasks by adopting SQL sentences according to the plurality of business fields.

In this embodiment, the conversion from the classified data to the target data in the standard data table is implemented by using an SQL (Structured Query Language) statement, and a plurality of SQL tasks are generated by using the SQL statement according to a plurality of service fields, so as to extract the target data from the classified data.

Step S220, based on a plurality of SQL tasks, extracting target data matched with the business fields from the classified data.

In this embodiment, as an implementation manner, for target data of the same type, SQL statements are obtained from classified data in batch; for example, for a business field "telephone number", in this embodiment, first, a plurality of customer names for which the telephone number needs to be obtained are determined, the plurality of customer names form a source table, an SQL statement is used to generate an SQL task, and when the SQL task runs, the source table is matched with classification data in a database, and target data of the plurality of telephone numbers corresponding to the plurality of customer names in the source table is obtained in batch. In the embodiment, the SQL sentences are optimized in operation efficiency.

Further, as an implementation manner, referring to fig. 5, fig. 5 is a schematic diagram of a refinement step of step S220 in this embodiment, and step S220 in this embodiment includes:

step S221, detecting whether a dependency relationship exists between multiple SQL tasks.

In this embodiment, according to a plurality of service fields, an SQL statement is used to generate a plurality of SQL tasks, and whether a dependency relationship exists among the plurality of SQL tasks is detected, if the dependency relationship exists, the dependency relationship is a condition for executing the SQL task, only the SQL task satisfying the dependency relationship can be executed by the computer device, and if the dependency relationship does not exist, an execution sequence of the plurality of SQL tasks is not limited.

Step S222, if there is no dependency relationship among the plurality of SQL tasks, executing the plurality of SQL tasks in parallel to extract the target data matched with the service field from the classification data.

Specifically, if there is no dependency relationship among the plurality of SQL tasks, the present embodiment executes the plurality of SQL tasks in parallel to extract the target data matched with the service field from the classified data, thereby speeding up the conversion efficiency of the standard data table.

In other embodiments, if there is a dependency relationship between the plurality of SQL tasks, the computer device executes the plurality of SQL tasks in sequence according to the dependency relationship.

According to the method, the SQL sentences are adopted to generate the plurality of SQL tasks according to the plurality of business fields, the target data matched with the business fields are extracted from the classified data based on the plurality of SQL tasks, whether the dependency relationship exists among the plurality of SQL tasks is detected when the plurality of SQL tasks are executed, and if the dependency relationship does not exist among the plurality of SQL tasks, the plurality of SQL tasks are executed in parallel, so that the conversion efficiency from the classified data to the target data in the standard data table is improved, the processing process from the business data to the standard data table is compressed, and the data processing efficiency is improved.

On the basis of the embodiment shown in fig. 3, another embodiment of the data processing method is provided, in this embodiment, specifically, the classification category includes a transaction category, a non-transaction category, and a transaction assistance category.

The standard data table corresponding to the transaction class comprises an internal transaction flow table and/or an external transaction flow table; the standard data sheet corresponding to the non-transaction type comprises at least one of an account information sheet, a private customer information sheet and a public customer information sheet; the standard data table corresponding to the transaction auxiliary class comprises at least one of a customer login information table, an account balance table, a reported suspicious customer information table and an area code table.

In this embodiment, taking an anti-money laundering scenario in the financial technology as an example, the classification categories are set to be transaction categories, non-transaction categories and transaction auxiliary information categories according to characteristics and business characteristics of modeling engineering that depend on the transaction flow of customers in the anti-money laundering scenario. Specifically, the standard data table corresponding to the transaction class comprises an internal transaction flow table and an external transaction flow table; the standard data tables corresponding to the non-transaction types comprise an account information table, a private client information table and a public client information table; the standard data table corresponding to the transaction auxiliary class comprises a customer login information table, an account balance table, a reported suspicious customer information table and an area code table; it is understood that in other application scenarios, the standard data table is not limited to the implementation combination of the embodiment.

In the transaction systems of the banks, each transaction flow contains a plurality of business logics, for example, a plurality of records exist in the same flow number, both the entry and the exit exist in the same flow, and the same flow initiator and receiver can be one-to-many, many-to-one, many-to-many and the like, which causes great inconvenience to data items for carrying out feature and modeling according to the business data of the banks; the embodiment designs a standard data table and corresponding business fields thereof, specifically, determines classification categories and business fields according to training requirements of a target model, and classifies business data acquired from various banks according to the classification categories to obtain classification data; and extracting target data matched with the business fields from the classified data according to the business fields, and generating a standard data table corresponding to each classification type based on the business fields and the target data.

In this embodiment, as an embodiment, the standard data tables corresponding to the transaction class, the non-transaction class, and the transaction assistance class are specifically shown in tables 1 to 9.

Referring to table 1, table 1 is a schematic form of a domestic transaction flow meter in the classification category of transaction classes in this embodiment. The domestic transaction flow chart comprises business fields such as a transaction flow number, a transaction date, a transaction time, a transaction place, a transaction occurrence mechanism number, a transaction code, a client number, an account number, a transaction counter-party account number, a counter-party mechanism number, a local account number, a transaction amount, a transaction channel, a transaction direction of coming and going, a transfer mark, a transaction remark, a transaction occurrence place, a transaction destination place and the like.

Inside TRANSACTION flow meter (T _ TRANSACTION)

TABLE 1

Referring to table 2, table 2 is a schematic form of the oversea transaction flow meter in the classification category of the transaction class in this embodiment. The overseas transaction flow sheet also includes the transaction currency, exchange rate settlement date and other foreign currency related business fields on the basis of the above table 1.

Overseas TRANSACTION flow meter (T _ TRANSACTION _ JW)

TABLE 2

Further, in the transaction flow meter, each transaction is initiated or received by taking an account as a minimum unit of transaction initiation or reception, and the embodiment further constructs an account information table, a private-to-public customer information table and a public-to-public customer information table for classification data of non-transaction classification categories.

Specifically, referring to table 3, table 3 is a schematic form of the account information table in the non-transaction classification category of the present embodiment. The account information table comprises service fields such as account numbers, account number types, client numbers, account opening dates, account opening mechanism numbers, account states, account passwords and the like.

Account information table (T _ ACCOUNT)

TABLE 3

Because the information of the enterprise client is different from that of the individual client, for example, the enterprise client has registration information, industry category, legal information and the like, and the individual client has age, occupation, income and the like, the embodiment respectively constructs a public-to-public client information table and a private-to-private client information table for the acquired client information, and introduces a digital dictionary for fields in the client information such as gender, occupation, industry and the like, so that the subsequent feature engineering and model training are facilitated.

Specifically, referring to table 4, table 4 is a schematic form of the information table for the public clients in the non-transaction classification category of the present embodiment. The public customer information table includes business fields such as a customer number, a certificate number, an enterprise name, a contact address, a business license number, a registered asset, registration time, a registration place, an industry category, an enterprise property, account opening information, legal representative information and the like, wherein a digital dictionary is introduced for a plurality of target data under the business fields such as a certificate type, an industry category code and the like, for example, the digital code corresponding to the target data 'identification card' is '01', and the like, and the table 4 is specifically referred to.

To public client information table (T _ UNITCUSTOMER)

TABLE 4

Referring to table 5, table 5 is a schematic form of the private customer information table in the non-transaction classification category in this embodiment. The private client information table comprises business fields such as client numbers, certificate numbers, client Chinese names, contact addresses, contact telephones, genders, nationalities, birth dates, client levels, monthly income, jobs, occupation, marital conditions, culture degrees and the like, and similarly, a digital dictionary is introduced for the business fields such as certificate types, client levels, jobs and the like so as to reduce the storage space occupation of the private client information table.

Private client information table (T _ PERSONCUSTOMER)

TABLE 5

Referring to table 6, table 6 is a schematic form of a customer registration information table in the classification category of the transaction auxiliary category according to this embodiment, and the customer registration information table facilitates construction of transaction characteristics of the customer through the customer registration information table in the next characteristic engineering. The customer login information table comprises business fields such as an electronic bank customer number, a client IP, a login channel, a transaction date, a transaction time, a Mac address and the like.

Client login information table (T _ EBLOGIN)

TABLE 6

Referring to table 7, table 7 is a schematic form of the account balance table in the transaction assistant classification category of the present embodiment. The account balance table is used for facilitating the construction of account balance related characteristics in the next step of characteristic engineering. The account balance table comprises service fields such as the latest transaction serial number, the transaction date, the transaction time, the account number, the transaction amount, the loan direction, the balance and the like.

Account BALANCE list (T _ BALANCE)

Name (R)	Name of field	Note	Data type	Remarks for note
					Trade serial number	SERSTAN	VARchar2(8)
Date of transaction	TRANDATE		VARchar2(8)
					Transaction time	TRANTIME	VARchar2(14)
Account number	ACCOUNTID		VARchar2(18)
					Customer number	CUSTOMERID	VARchar2(10)
Amount of transaction	TRANAMT		NUMBER(16,2)
					Directions of loan	DCFLAG	VARchar2(1)
Balance of money	BALANCE		NUMBER(16,2)

TABLE 7

Referring to table 8, table 8 is a schematic form of the reported suspicious client information table in the classification category of the transaction auxiliary class according to this embodiment, and the reported suspicious client information table records clients that have been confirmed by the service provider as suspicious of money laundering in the past. The reported suspicious customer information table comprises business fields such as a customer number, an account number, the last transaction time, the reporting time and the like.

Reported suspicious client information Table (T _ TARGET)

Name (R)	Name of field	Note	Data type	Remarks for note
					Date reporting	CREATEDATE	VARchar2(8)
Date of last transaction	LASTTRANDATE		VARchar2(8)
					Customer number	CUSTOMERID	VARchar2(10)
Client name	CUSTOMERNAME		VARchar2(100)
					Account number	ACCOUNTID	VARchar2(18)

TABLE 8

Referring to table 9, table 9 is a schematic form of an area code table in the classification category of the transaction auxiliary class according to this embodiment, and the area code table is used to query the information of the transaction institution side of each transaction record. The regional code table includes business fields such as bank codes, city codes, cities, provinces, branch names, and the like.

Region code table (T _ BANK _ CITY)

Name (R)	Name of field	Note	Data type	Remarks for note
					Bank code	BANK_CODE	VARchar2(14)
City code	CITY_CODE		VARchar2(10)
					City	CITY_NAME	VARchar2(50)
Province of labor	PROVINCE		VARchar2(20)
					Branch line code number	TRANBRC	VARchar2(3)
Branch line name	NAME		VARchar2(60)

TABLE 9

In the embodiment, standard data tables under various classification categories and specific corresponding business fields are designed, and computer equipment of the embodiment effectively splits, classifies and combines all business data acquired from various banks, has clear hierarchical structure and eliminates redundant information; simplifying complex transaction logic in original business data, integrating data of different fields of each table, and unifying expression of different values of the same field among the tables; the occupation of data on the storage space is reduced, a foundation is laid for subsequent characteristic engineering, the processing efficiency is improved, the subsequent data flow can be standardized, the labor input is reduced, and large-scale popularization of projects becomes possible.

It should be understood that although the various steps in the flow charts of fig. 2-5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-5 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.

In one embodiment, as shown in fig. 6, there is provided a data processing apparatus including:

the classification module 10 is configured to classify the service data of the plurality of client samples according to preset classification categories to obtain classification data;

an extracting module 20, configured to extract, according to a plurality of preset service fields, target data matched with the service fields from the classification data; the classification category and the service field are determined according to the training requirement of a target model;

a generating module 30, configured to generate a standard data table corresponding to each classification type according to the service field and the target data; the standard data table is used for training the target model.

Optionally, the generating module 30 includes:

Optionally, the generating module 30 is further configured to, if the data type of the target data is not an enumeration type, import the target data into the service field corresponding to the target data to generate the standard data table.

Optionally, the extraction module 20 comprises:

Optionally, the extraction unit comprises:

Optionally, the classification category includes a transaction category, a non-transaction category, and a transaction assistance category;

Optionally, the apparatus further comprises:

The data processing apparatus provided in this embodiment may execute the data processing method embodiment, and the implementation principle and the technical effect are similar, which are not described herein again.

For specific limitations of the data processing apparatus, reference may be made to the above limitations of the data processing method, which are not described herein again. The various modules in the data processing apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, there is also provided a computer device as shown in fig. 1, which may be a server. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data processing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method.

Those skilled in the art will appreciate that the architecture shown in fig. 1 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:

classifying the service data of a plurality of client samples according to preset classification categories to obtain classification data; extracting target data matched with the business fields from the classified data according to a plurality of preset business fields; the classification category and the service field are determined according to the training requirement of the target model; generating a standard data table corresponding to each classification type according to the service field and the target data; the standard data table is used to train the target model.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Ramb microsecond direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above examples only represent some embodiments of the present invention, and the description thereof is more specific, but not to be construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method of data processing, the method comprising:

2. The method of claim 1, wherein generating a standard data table corresponding to each of the classification categories according to the service field and the target data comprises:

detecting whether the data type of the target data is an enumeration type;

3. The method of claim 2, further comprising:

4. The method according to any one of claims 1-3, wherein the extracting target data matched with the service field from the classification data according to a preset plurality of service fields comprises:

5. The method according to claim 4, wherein the extracting target data matching the business field from the classified data based on the plurality of SQL tasks comprises:

6. The method of claim 1, wherein the classification categories include a transaction category, a non-transaction category, and a transaction assistance category;

7. The method of claim 1, wherein before classifying the business data of the plurality of customer samples according to the predetermined classification categories to obtain the classification data, the method further comprises:

8. A data processing apparatus, characterized in that the apparatus comprises:

9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.