CN117171174A

CN117171174A - Data processing method and device and electronic equipment

Info

Publication number: CN117171174A
Application number: CN202311291346.7A
Authority: CN
Inventors: 曹海傧
Original assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Current assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Priority date: 2023-10-08
Filing date: 2023-10-08
Publication date: 2023-12-05

Abstract

The application discloses a data processing method, a device and electronic equipment, which relate to the technical field of combination of big data and data analysis and mining and data preprocessing, and the method comprises the following steps: obtaining a change log of service data, processing an isomorphic table and an isomorphic pull chain table according to the change log, generating a time point index detail wide table according to the dimensionality and the first index of each data in the isomorphic table and the isomorphic pull chain table, and generating a section index detail wide table according to the dimensionality and the second index of each data in the isomorphic table and the isomorphic pull chain table. By the method, timeliness of data processing can be improved, and data processing efficiency can be improved when data processing is performed based on the time index detail list and the interval index detail list, so that business decision efficiency can be improved.

Description

Data processing method and device and electronic equipment

Technical Field

The present application relates to the field of technologies for combining big data, data mining, analysis and data preprocessing, and in particular, to a data processing method, apparatus and electronic device.

Background

The offline data warehouse technology (ETL) may be used to describe the process of extracting (extracting), converting (transforming), and loading (Load) data from a source end to a destination end, so as to integrate large volumes of scattered, and non-uniform data in an enterprise together, and provide an analysis basis for decision making of the enterprise. The source end may be a service system database, a distributed file system, or other data bins, and the destination end may be a target system database, a target file system, or other target data bins.

With the rapid development of communication technology, the enterprise data processed by offline ETL increases, which results in low timeliness of data processing and low data processing efficiency when processing integrated mass data.

Disclosure of Invention

The application provides a data processing method, a data processing device and electronic equipment, which can solve the problems that the time efficiency of data processing is low and the data processing efficiency is low when integrated mass data is processed due to the fact that enterprise data processed by off-line ETL is increased.

In a first aspect, the present application provides a data processing method, the method comprising:

obtaining a change log of service data, and processing an isomorphic table and an isomorphic pull linked list according to the change log, wherein the isomorphic table comprises data information of a current time point of the service, and the isomorphic pull linked list comprises data information of the current time point and a historical time point of the service;

generating a time point index detail list according to the dimensionality and the first index of each data in the isomorphic list and the isomorphic pull linked list, wherein the time point index detail list is used for representing index detail data corresponding to a time point of a service;

generating an interval index detail list according to the dimension of each data and the second index, wherein the interval index detail list is used for representing index detail data corresponding to the service in a time interval.

By the method, the isomorphic table and the isomorphic pull chain table are processed according to the change log of the service data, the time index detail list and the interval index detail list are generated based on the isomorphic table and the isomorphic pull chain table, and the integration of the service data is completed, so that the timeliness of data processing can be improved, the data processing efficiency can be improved when the data processing is performed based on the time index detail list and the interval index detail list, and further the service decision efficiency can be improved.

In some possible designs, the obtaining the change log includes:

collecting the change log from a database and sending the change log to a message center;

subscribing the message center and obtaining the change log.

By the method, service data of each service system can be obtained in real time, and data transmission efficiency can be improved.

In some possible designs, the processing the isomorphic table and the isomorphic pull linked list according to the change log includes:

updating an initial isomorphic table according to the change log to obtain the isomorphic table;

and storing the change log to an initial isomorphic pull linked list to obtain the isomorphic pull linked list.

By the method, the change logs are respectively stored in the isomorphic table and the isomorphic pull chain table, and then the time index detail list and the interval index detail list can be processed based on the isomorphic table and the isomorphic pull chain table.

In some possible designs, the dimensions of the respective data include at least customer identification, product major class, product minor class; the first index at least comprises a nominal balance and a credit risk opening; the second index includes at least transaction amount, revenue.

In some possible designs, after the generating the interval index specification table, the method further includes:

and responding to the operation of triggering the target option by the user, screening in the time point index detail list and/or the interval index detail list, and displaying the screened target data information on a user interface.

In some possible designs, the target options include at least a target object value option, a target object identification, a first index option, a second index option, a product category option, and a time option.

By the method, a user can conveniently and rapidly and accurately search the data information.

In a second aspect, the present application provides a data processing apparatus, the apparatus comprising:

the processing module is used for acquiring a change log of service data, and processing an isomorphic table and an isomorphic pull chain table according to the change log, wherein the isomorphic table comprises data information of a current time point of the service, and the isomorphic pull chain table comprises data information of the current time point and a historical time point of the service;

The first generation module is used for generating a time point index detail list according to the dimensionality and the first index of each data in the isomorphic table and the isomorphic pull chain table, wherein the time point index detail list is used for representing index detail data corresponding to a time point of a service;

and the second generation module is used for generating an interval index detail list according to the dimension of each data and the second index, wherein the interval index detail list is used for representing index detail data corresponding to the service in the time interval.

In some possible designs, the processing module is specifically configured to:

subscribing the message center and obtaining the change log.

In some possible designs, the processing module is further to:

In some possible designs, the apparatus further comprises:

and the display module is used for responding to the operation of triggering the target option by the user, screening the time point index detail list and/or the interval index detail list, and displaying the screened target data information on a user interface.

In a third aspect, the present application provides an electronic device, comprising:

a memory for storing a computer program;

and a processor for implementing the steps of the data processing method of the first aspect when executing the computer program stored on the memory.

In a fourth aspect, the present application provides a computer-readable storage medium having stored therein a computer program which, when executed by a processor, implements the data processing method steps of the first aspect described above.

In a fifth aspect, the present application provides a computer program product comprising: computer program code for causing a computer to carry out the steps of the data processing method of the first aspect described above when said computer program code is run on a computer.

Based on the data processing method, the isomorphic table and the isomorphic pull chain table are processed according to the change log of the service data, the time index detail list and the interval index detail list are generated based on the isomorphic table and the isomorphic pull chain table, and the integration of the service data is completed, so that the timeliness of the data processing can be improved, the data processing efficiency can be improved when the data processing is performed based on the time index detail list and the interval index detail list, and the service decision efficiency can be further improved.

The technical effects of each of the second to fifth aspects and the technical effects that may be achieved by each aspect are described above with reference to the first aspect or the technical effects that may be achieved by each possible aspect in the first aspect, and the detailed description is not repeated here.

Drawings

FIG. 1 is a schematic view of a scene to which embodiments of the present application are applicable;

FIG. 2 is a schematic diagram of another scenario in which an embodiment of the present application is applicable;

FIG. 3 is a flowchart of a data processing method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

FIG. 5 is a schematic diagram of another data processing apparatus according to an embodiment of the present application;

Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application. Embodiments of the application and features of the embodiments may be combined with one another arbitrarily without conflict. Also, while a logical order of illustration is depicted in the flowchart, in some cases the steps shown or described may be performed in a different order than presented.

The terms first and second in the description and claims of the application and in the above-mentioned figures are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the term "include" and any variations thereof is intended to cover non-exclusive protection. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus. The term "plurality" in the present application may mean at least two, for example, two, three or more, and embodiments of the present application are not limited.

In the technical scheme of the application, the data is collected, transmitted, used and the like, and all meet the requirements of national relevant laws and regulations.

For the convenience of understanding by those skilled in the art, technical terms related to the embodiments of the present application will be explained first.

(1) Risk exposure refers to the risk of unprotected.

(2) Change data retrieval (Change Data Capture, CDC) is a common technique in the database art that can send changes to data in database tables to downstream consumers in standard form, where the changes to data can include Insert, delete, update.

(3) Kafka is a high throughput distributed publish-subscribe messaging system.

(4) Streaming data refers to an ever-increasing, unbounded, dynamic set of data over time.

(5) Stream computation refers to processing of stream data.

(6) A flow batch integrated calculation engine: a compute engine that supports both stream computing and batch computing.

(7) A Flink is a distributed data processing engine that executes arbitrary streaming data programs in a data parallel and pipelined fashion, and that can perform batch and streaming processes.

(8) The data preparation area (Operation Data Store, ODS) is used for storing isomorphic data with the same data structure in the database of the application system, adding a time stamp field, and retaining a history zipper according to the need, which is also called a source layer and an isomorphic layer.

(9) Hybrid transaction and analytics processing (Hybrid Transaction and Analytical Processing, HTAP) storage engine is a converged database that can support both On-line transaction (On-line Transaction Pro cessing, OLTP) and On-line data analytics (On-Line Analytical Processing, OLAP) scenarios.

(10) The data detail layer (Data Warehouse Details, DWD) is the core hierarchy of the data center, and mainly cleans and integrates ODS layer data into corresponding fact tables.

(11) A broad table refers to a database table that associates business-related metrics, dimensions, attributes together.

(12) Data service: and a unified data access interface is provided, underlying storage and technical differences are shielded, and the capability of real-time query analysis of data and rapid conversion of a data table into an application program interface service is provided.

(13) Business intelligence (Business Intelligence, BI) service: refers to a service for interactively and visually analyzing data and generating flexible report forms by utilizing a data warehouse technology, an online analysis technology and a visualization technology.

(14) A Primary Key (PK), a Primary constraint Key in a database, enables a unique data record to be determined in a table by the PK.

(15) Event time event_tms: the time of occurrence of the event/service is defined, and once the event/service is determined, the sequence of occurrence of the event/service can be restored through the event time.

(16) Version number version is event time.

The following description is made for some simple descriptions of application scenarios applicable to the technical solution of the embodiment of the present application, and it should be noted that the application scenarios described below are only used for illustrating the embodiment of the present application, but not limiting. In the specific implementation, the technical scheme provided by the embodiment of the application can be flexibly applied according to actual needs.

Fig. 1 is a schematic diagram of an application scenario suitable for the embodiment of the present application. The scenario mainly includes a database 101, a data collection module 102, a message center 103, a batch integrated calculation engine 104, and a storage engine 105.

The database 101 is used for storing business data, such as basic information of clients and trade partners, risk information, transaction amount, risk exposure, and business income, for example. Wherein the service data may be stored in the form of a log.

The data collection module 102 is configured to collect a change log of service data from the database 101 by using CDC technology, and send the change log to the message center 103, so that a great deal of modification of a service system of the database 101 can be avoided.

Alternatively, the number of databases and data acquisition modules in the embodiment of the present application is not limited, and as shown in fig. 1, only the database 101 and the data acquisition module 102 are described as an example.

Illustratively, the message center 103 may be Kafka, which is used to store a change log of service data, so that the batch integrated calculation engine 104 may timely obtain the change log by subscribing to the message center 103.

The batch integrated calculation engine 104 may be a Flink, and is configured to process, when the change log is obtained, ODS data according to the change log, including: the system comprises an isomorphic table and an isomorphic pull linked list, wherein the isomorphic table comprises data information of a current time point of a service, and the isomorphic pull linked list comprises data information of the current time point and a historical time point of the service. At the same time, the isomorphic table and isomorphic pull chain table are saved to a storage engine 105, such as an HTAP storage engine.

In addition, the flow batch integrated calculation engine 104 is further configured to scan the isomorphic table and the isomorphic pull chain table in the storage engine 105 by using flow calculation or batch calculation according to different timeliness requirements of the indexes, and generate DWDs according to the isomorphic table and the isomorphic pull chain table. Specifically, when the real-time index is selected, current data in the isomorphic table and the isomorphic pull chain table are respectively processed by adopting stream calculation, and a time point index detail list is generated according to the dimension of the current data and the first index, wherein the time point index detail list is used for representing index detail data corresponding to a time point of a service. Meanwhile, a section index detail list is generated according to the dimension of the current data and the second index, wherein the section index detail list is used for representing index detail data corresponding to the service in the time section.

When the non-real-time index is selected, carrying out pre-association integration on each data in the isomorphic table and the isomorphic pull chain table respectively by adopting batch calculation, and generating a time point index detail wide table according to the dimensionality and the first index of each data in the isomorphic table and the isomorphic pull chain table. Meanwhile, a section index detail wide table is generated according to the dimensionality and the second index of each data in the isomorphic table and the isomorphic pull chain table.

In the process, the wide table structure is adopted to avoid large table association of downstream data application, and data processing efficiency is improved. Meanwhile, the time point index detail wide table and the interval index detail wide table provide data support for developing comprehensive marketing and differentiated customer service.

In some possible designs, a scene to which the embodiment of the present application is applied may further include a unified view front end application, as shown in fig. 2, which is another schematic view of the scene to which the embodiment of the present application is applied. The scenario mainly includes a database 101, a data collection module 102, a message center 103, a batch all-in-one calculation engine 104, a storage engine 105, and a unified view front end application 201.

Illustratively, the unified view front end application 201 is configured to provide data services and BI services for users, so that service managers and traders can count various service data of the dimensions of clients and traders, thereby improving the efficiency of service decision-making and trade judgment.

Based on the application scenario, the data processing method provided by the embodiment of the application processes the isomorphic table and the isomorphic pull chain table according to the change log of the service data, generates the time index detail width table and the interval index detail width table based on the isomorphic table and the isomorphic pull chain table, completes the integration of the service data, not only can improve the timeliness of the data processing, but also can improve the data processing efficiency when the data processing is performed based on the time index detail width table and the interval index detail width table, and further can improve the service decision efficiency. The method and the device according to the embodiments of the present application are based on the same technical concept, and because the principles of the problems solved by the method and the device are similar, the embodiments of the device and the method can be referred to each other, and the repetition is not repeated.

In order to further explain the technical solution provided by the embodiments of the present application, the following details are described with reference to the accompanying drawings and the detailed description. Although embodiments of the present application provide the method operational steps shown in the following embodiments or figures, more or fewer operational steps may be included in the method, either on a routine or non-inventive basis. In steps where there is logically no necessary causal relationship, the execution order of the steps is not limited to the execution order provided by the embodiments of the present application. The method may be performed sequentially or and in accordance with the method shown in the embodiments or drawings when the actual process or apparatus is performed.

Fig. 3 is a flowchart of a data processing method according to an embodiment of the present application, where the flowchart may be executed by a data processing apparatus, and the apparatus may be implemented by software, or may be implemented by hardware, or may be implemented by a combination of software and hardware, so as to ensure normal operation of a system. As shown in fig. 3, the process includes the steps of:

s301, obtaining a change log of service data, and processing an isomorphic table and an isomorphic pull chain table according to the change log;

optionally, the method for obtaining the change log includes: the data collection module 102 shown in fig. 1 or 2 collects a change log of service data from the database 101 through CDC technology, and sends the change log to the message center 103, so that the batch integrated calculation engine 104 can obtain the change log through subscribing to the message center 103.

Optionally, the method for processing the isomorphic table and the isomorphic pull chain table comprises the following steps: the flow batch integrated calculation engine 104 updates the initial isomorphism table according to the change log to obtain the isomorphism table, wherein the structure of the initial isomorphism table is as follows: PK, field 1, field 2, … …, event_tms, the isomorphic table includes data information of the current time point of the service. Meanwhile, the flow batch integrated calculation engine 104 stores the change log to an initial isomorphism pull linked list to obtain the isomorphism pull linked list, wherein the structure of the initial isomorphism pull linked list is as follows: the PK_version, the field 1, the field 2, the … …, the event_tms and the isomorphic pull linked list comprise data information of the current time point and the historical time point of the service.

For example, the database 101 has a data record of pk=1 in Table 1: pk=1, field 1, field 2, … …, event_tms=tms1, there is one pk=1 data record in the initial isomorphic table: pk=1, field 1, field 2, … …, event_tms=tms1; the initial isomorphic pull chain list has one data record of pk=1_tms1: pk=1_tms1, field 1, field 2, … …, event_tms=tms1.

At tms2 (tms 2> tms 1), the user performs an update operation on the data record of pk=1_tms1 in the database 101, and the updated data record is: pk=1_tms2, field 1_new, field 2_new, … …, event_tms=tms2, corresponding update log is log1. The flow batch integrated calculation engine 104 obtains log1 through the method and processes it into the initial isomorphism table to obtain the isomorphism table as follows: pk=1_tms2, field 1_new, field 2_new, … …, event_tms=tms2. The isomorphic pull chain table is as follows: pk=1_tms1, field 1, field 2, … …, event_tms=tms1; pk=1_tms2, field 1_new, field 2_new, … …, event_tms=tms2.

From the above example, the isomorphic Table may be considered as a copy Table of Table1, and the latest state of the change log obtained by the integral flow calculation engine 104 is kept, but there is a small time interval from Table1, which is the time for network transmission and flow processing of the change log. The isomorphic pull chain Table uses a zipper mode to store the record states of each moment of Table 1.

The flow batch integrated calculation engine 104 processes the isomorphic table and the isomorphic pull chain table by the method, and then stores the isomorphic table and the isomorphic pull chain table in the storage engine 105.

S302, generating a time point index detail width table according to the dimensionality and the first index of each data in the isomorphic table and the isomorphic pull chain table;

optionally, the method for generating the time point index detail width table may be: the flow batch integrated calculation engine 104 respectively uses isomorphic tables and isomorphic pull chain tables in the flow calculation or batch calculation scanning storage engine 105 according to different timeliness requirements of indexes, and generates a time point index detail width table according to dimensions and a first index of each data in the isomorphic tables and the isomorphic pull chain tables, wherein the dimensions at least comprise customer identifications, major classes of products and minor classes of products, the first index at least comprises nominal balances and credit risk openings, the time point index detail width table is used for representing index detail data corresponding to a service at a time point, and the structure of the time point index detail width table is as follows: PK, start_tms, end_tms, dimension 1, dimension 2, … …, index 1, index 2, … …, for example, the time point index detail table is: pk=1, start_tms=t, end_tms=t+1, dimension 1=customer a, dimension 2=product major class=exchange rate class, dimension 3=product minor class=distant foreign exchange transaction, index 1=nominal balance=100, index 2=credit risk exposure=100, representing that the transaction record with primary key 1 corresponds to the time point index detail data: customer a has a nominal balance of 100 and a credit risk exposure of 100 for the long term foreign exchange transactions in [ T, t+1).

Further, the integrated flow batch calculation engine 104 stores the generated time point index detail list into the storage engine 105, so that layered storage of data information is realized, and in the display process, the data information stored in the storage engine 105 in a layered manner is displayed according to a user instruction, so that traceability display of the data information is realized, and a more visual and logical quantized service data display mode is provided.

S303, generating a section index detail wide table according to the dimensions and the second index of each data in the isomorphic table and the isomorphic pull chain table.

Optionally, the method for generating the section index detail width table may be: the flow batch integrated calculation engine 104 respectively uses an isomorphic table and an isomorphic pull chain table in the flow calculation or batch calculation scanning storage engine 105 according to different timeliness requirements of indexes, and generates an interval index detail width table according to dimensions of each data in the isomorphic table and the isomorphic pull chain table and a second index, wherein the dimensions at least comprise customer identifications, major classes of products and minor classes of products, the second index at least comprises transaction amount and income, the interval index detail width table is used for representing index detail data corresponding to a service in a time interval, and the structure of the interval index detail width table is as follows: PK, event_tms, dimension 1, dimension 2, … …, index 1, index 2, … …. For example, the section index detail table is: pk=1, event_tms=t, dimension 1=customer B, dimension 2=product major class=exchange rate class, dimension 3=product minor class=long-term foreign exchange transaction, index 1=transaction amount=100, index 2=income=2, representing that the section index detail data corresponding to the transaction record with primary key 1 is: customer B has a long-term foreign exchange transaction at time T, with a transaction amount of 100 and a revenue of 2.

Further, the flow batch integrated calculation engine 104 stores the generated section index detail list into the storage engine 105, so that layered storage of the data information is realized, and in the display process, the data information stored in the storage engine 105 in a layered manner is displayed according to a user instruction, so that traceability display of the data information is realized, and a more visual and logical quantized service data display mode is provided.

In some possible embodiments, the unified view front end application 201 as shown in FIG. 2 provides the user with a data service that provides real-time customer image, index metering services and a BI service that provides flexible data statistics, custom report output services. Specific: and responding to the operation of triggering the target options by the user, screening in a time point index detail list and/or an interval index detail list, and displaying screened target data information in a user interface, wherein the target options at least comprise a target object numerical value option, a target object identifier, a first index option, a second index option, a product category option and a time option.

In short, when the user wants to query the first index corresponding to the time point of the query_tms, the user only needs to screen out the time point index detail record of the query_tms in [ start_tms, end_tms) in the time point index detail list, and then summarize the corresponding result in real time according to the first preset service query rule. For example: the first preset business query rule is indexes (nominal balance and credit risk opening) displayed at the time point of query_tms according to the major classes (exchange rate classes), and the first-level drill-down (display of product minor class indexes) and the second-level drill-down are performed to the traffic detail information. The service index is displayed according to the product major class, namely, the time point index detail of the query_tms in the [ start_tms, end_tms) is summarized and calculated according to the product major class; the first-stage drill-down is to filter the time point index details of the query_tms in the [ start_tms, end_tms ] according to the product major class, and then to perform grouping summary calculation according to the product minor class; the transaction detail information is obtained by filtering the time index details of the query_tms in the [ start_tms, end_tms ] according to the product major class and the product minor class.

When the user wants to query the second index in the [ query_start_tms, query_end_tms) time interval, the user only needs to screen the interval index detail record of the event_tms in the [ query_start_tms, query_end_tms) in the interval index detail list, and then summarize the corresponding result in real time according to the second preset service query rule. For example: the second preset business rule is an index (transaction amount and income) for displaying target data according to the product major class (exchange rate class), and the first-level drill-down (display of product minor class index) and the second-level drill-down are carried out to the transaction detail information. Wherein, the service index is displayed according to the product major class, namely, the section index detail of the event_tms in the [ query_start_tms, query_end_tms) is summarized and calculated according to the product major class; the first-level drill-down is to filter the section index details of the event_tms in the [ query_start_tms, query_end_tms) according to the major product classes, and then to perform grouping summary calculation according to the minor product classes; the transaction detail information is that the interval index detail of the event_tms in the [ query_start_tms, query_end_tms) is filtered according to the major class and the minor class of the product.

Example one: the unified view front end application 201 includes a target object numerical option in a first interface, and in response to a user selecting a single target object from the first interface, jumps to a second interface including a target object identifier, and in response to a user selecting a target object from the second interface as a client a, jumps to a third interface including a first index option, a second index option, a product category option, and a time option.

Responding to the first index option selected by the user on the third interface, wherein the time option is 2023, 1 month and 20 days, the product category option is product category=exchange rate category- & product category=long-term foreign exchange transaction, screening is performed in the time point index detail list, and the screened target data information is displayed on the third interface as follows: customer a had a nominal balance=200 and a credit risk exposure=200 for a long term foreign exchange transaction at day 1 and 20 of 2023.

Example two: the unified view front end application 201 includes a target object numerical option in a first interface, and in response to a user selecting a single target object from the first interface, jumps to a second interface including a target object identifier, and in response to a user selecting a target object from the second interface as a client a, jumps to a third interface including a first index option, a second index option, a product category option, and a time option.

Responding to the user selecting the second index option on the third interface, wherein the time option is from the beginning of the last year to the end of the last year, the product category option is the product category=exchange rate category, screening is performed in the interval index detail list, and the screened target data information is displayed on the third interface as follows: customer a trade volume of exchange rate class from the beginning of the last year to the end of the last year=100, income=2, product coverage=7.

Example three: the first interface of the unified view front end application 201 includes a target object numerical option, and in response to the user jumping to a fourth interface when the target object numerical option selected by the first interface is a non-single target object, the fourth interface includes a target object identifier, a first index option, a second index option, a product category option, and a time option.

In response to the user selecting the time option from the beginning of the last year to the end of the last year in the fourth interface, and the product category option being the product category=exchange rate category, screening in the time point index detail list and the interval index detail list, and displaying the screened target data information in the fourth interface as follows: the nominal balance, credit risk of the exchange rate class at the end of the last year of all customers are open, the transaction amount, income and product coverage of all customers from the end of the last year to the end of the last year.

By the data processing method, the isomorphic table and the isomorphic pull chain table are processed according to the change log of the service data, the time index detail list and the interval index detail list are generated based on the isomorphic table and the isomorphic pull chain table, and the integration of the service data is completed, so that the timeliness of the data processing can be improved, the data processing efficiency can be improved when the data processing is performed based on the time index detail list and the interval index detail list, and the service decision efficiency can be further improved.

Based on the same inventive concept, an embodiment of the present application further provides a data processing apparatus, as shown in fig. 4, which is a schematic structural diagram of the data processing apparatus provided in the embodiment of the present application, where the apparatus includes:

the processing module 401 is configured to obtain a change log of service data, and process an isomorphic table and an isomorphic pull linked list according to the change log, where the isomorphic table includes data information of a current time point of the service, and the isomorphic pull linked list includes data information of the current time point and a historical time point of the service;

a first generation module 402, configured to generate a time point index detail table according to dimensions and a first index of each data in the isomorphic table and the isomorphic pull chain table, where the time point index detail table is used to characterize index detail data corresponding to a time point of a service;

a second generating module 403, configured to generate an interval index detail table according to the dimensions and the second index of each data, where the interval index detail table is used to characterize index detail data corresponding to the service in the time interval.

In some possible designs, the processing module 401 is specifically configured to:

Subscribing the message center and obtaining the change log.

In some possible designs, the process module 401 is also to:

In other embodiments, in addition to the modules shown in fig. 4, a display module may be further included, as shown in fig. 5, which schematically illustrates a structural schematic diagram of another data processing apparatus according to an embodiment of the present application. The device comprises a processing module 401, a first generating module 402, a second generating module 403 and a display module 501.

By means of the data processing device, the isomorphic table and the isomorphic pull chain table are processed according to the change log of the service data, the time index detail list and the interval index detail list are generated based on the isomorphic table and the isomorphic pull chain table, and the integration of the service data is completed, so that timeliness of data processing can be improved, data processing efficiency can be improved when the data processing is carried out based on the time index detail list and the interval index detail list, and further service decision efficiency can be improved.

Based on the same inventive concept, an embodiment of the present application further provides an electronic device, where the electronic device may implement the functions of the foregoing data processing apparatus, and referring to fig. 6, the electronic device includes:

at least one processor 601, and a memory 602 connected to the at least one processor 601, a specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application, and in fig. 6, the processor 601 and the memory 602 are connected through a bus 600 as an example. Bus 600 is shown in bold lines in fig. 6, and the manner in which the other components are connected is illustrated schematically and not by way of limitation. The bus 600 may be divided into an address bus, a data bus, a control bus, etc., and is represented by only one thick line in fig. 6 for convenience of representation, but does not represent only one bus or one type of bus. Alternatively, the processor 601 may be referred to as a controller, and the names are not limited.

In an embodiment of the present application, the memory 602 stores instructions executable by the at least one processor 601, and the at least one processor 601 may perform the data processing method described above by executing the instructions stored in the memory 602. The processor 601 may implement the functions of the respective modules in the apparatus shown in fig. 4 or 5.

The processor 601 is a control center of the device, and various interfaces and lines can be used to connect various parts of the whole control device, and through running or executing instructions stored in the memory 602 and calling data stored in the memory 602, various functions of the device and processing data can be performed, so that the device can be monitored as a whole.

In one possible design, processor 601 may include one or more processing units, and processor 601 may integrate an application processor and a modem processor, wherein the application processor primarily processes operating systems, user interfaces, application programs, and the like, and the modem processor primarily processes wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 601. In some embodiments, processor 601 and memory 602 may be implemented on the same chip, or they may be implemented separately on separate chips in some embodiments.

The processor 601 may be a general purpose processor such as a Central Processing Unit (CPU), digital signal processor, application specific integrated circuit, field programmable gate array or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, which may implement or perform the methods, steps and logic blocks disclosed in embodiments of the application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the data processing method disclosed in connection with the embodiments of the present application may be directly embodied as a hardware processor executing, or may be executed by a combination of hardware and software modules in the processor.

The memory 602 is a non-volatile computer readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. The Memory 602 may include at least one type of storage medium, which may include, for example, flash Memory, hard disk, multimedia card, card Memory, random access Memory (Random Access Memory, RAM), static random access Memory (Static Random Access Memory, SRAM), programmable Read-Only Memory (Programmable Read Only Memory, PROM), read-Only Memory (ROM), charged erasable programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory), magnetic Memory, magnetic disk, optical disk, and the like. Memory 602 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 602 in embodiments of the present application may also be circuitry or any other device capable of performing storage functions for storing program instructions and/or data.

By programming the processor 601, the code corresponding to the data processing method described in the foregoing embodiment can be solidified into a chip, so that the chip can execute the steps of the data processing method of the embodiment shown in fig. 3 at the time of operation. How to design and program the processor 601 is a well-known technique for those skilled in the art, and will not be described in detail herein.

Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, the computer program product comprising: computer program code which, when run on a computer, causes the computer to perform the data processing method as any of the preceding discussion. Since the principle of the above-mentioned computer readable storage medium for solving the problem is similar to that of the data processing method, the implementation of the above-mentioned computer readable storage medium may refer to the implementation of the method, and the repetition is omitted.

Based on the same inventive concept, embodiments of the present application also provide a computer program product comprising: computer program code which, when run on a computer, causes the computer to perform the data processing method as any of the preceding discussion. Since the principle of the solution of the problem of the computer program product is similar to that of the data processing method, the implementation of the computer program product can refer to the implementation of the method, and the repetition is omitted.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

Claims

1. A method of data processing, the method comprising:

2. The method of claim 1, wherein the obtaining a change log comprises:

subscribing the message center and obtaining the change log.

3. The method of claim 1, wherein the processing the isomorphic table and the isomorphic pull linked list from the change log comprises:

4. The method of claim 1, wherein the dimensions of each data include at least customer identification, product major class, product minor class; the first index at least comprises a nominal balance and a credit risk opening; the second index includes at least transaction amount, revenue.

5. The method of claim 1, further comprising, after the generating the interval indicator specification table:

6. The method of claim 5, wherein the target options include at least a target object value option, a target object identification, a first index option, a second index option, a product category option, and a time option.

7. A data processing apparatus, the apparatus comprising:

8. The apparatus of claim 7, wherein the processing module is specifically configured to:

subscribing the message center and obtaining the change log.

9. The apparatus of claim 7, wherein the processing module is further to:

10. The apparatus of claim 7, wherein the dimensions of each data include at least customer identification, product major class, product minor class; the first index at least comprises a nominal balance and a credit risk opening; the second index includes at least transaction amount, revenue.

11. The apparatus of claim 7, wherein the apparatus further comprises:

12. The apparatus of claim 11, wherein the target options comprise at least a target object value option, a target object identification, a first index option, a second index option, a product category option, and a time option.

13. An electronic device, comprising:

a memory for storing a computer program;

a processor for carrying out the method steps of any one of claims 1-6 when executing a computer program stored on said memory.

14. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored therein a computer program which, when executed by a processor, implements the method steps of any of claims 1-6.

15. A computer program product, the computer program product comprising: computer program code which, when run on a computer, causes the computer to perform the method of any of the preceding claims 1-6.